Redis集群 Cluster搭建
方式一 docker-compose 部署
适用场景 : 开发 , 测试等环境 ;
注 : cluster最少需要六个服务 , 组成最小分片集群;
1 . 创建目录和文件
├── docker-compose.yml
├── c_redis_6379
│ ├── conf
│ │ └── redis.conf
│ └── data
├── c_redis_6380
│ ├── conf
│ │ └── redis.conf
│ └── data
├── c_redis_6381
│ ├── conf
│ │ └── redis.conf
│ └── data
├── c_redis_6382
│ ├── conf
│ │ └── redis.conf
│ └── data
├── c_redis_6383
│ ├── conf
│ │ └── redis.conf
│ └── data
└── c_redis_6384
├── conf
│ └── redis.conf
└── data
2 . Redis配置文件
每个节点只需要修改配置中相关端口
# 端口
port 6379
cluster-enabled yes
cluster-config-file nodes-6371.conf
cluster-node-timeout 5000
appendonly yes
protected-mode no
# 密码认证
requirepass xxxxx
masterauth xxxxx
# 这里是宿主机IP
cluster-announce-ip 192.168.2.13
cluster-announce-port 6379
cluster-announce-bus-port 16379
3 . docker-compose 配置文件
编写完成后使用docker-compose up -d启动容器 ;
version: '3.3'
services:
c_redis_6379: # 服务名称
image: redis:7.0.13 # 创建容器时所需的镜像
container_name: c_redis_6379 # 容器名称
restart: always # 容器总是重新启动
volumes: # 数据卷,目录挂载
- ./c_redis_6379/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./c_redis_6379/data:/data
ports:
- 6379:6379
- 16379:16379
command:
redis-server /usr/local/etc/redis/redis.conf
c_redis_6380: # 服务名称
image: redis:7.0.13 # 创建容器时所需的镜像
container_name: c_redis_6380 # 容器名称
restart: always # 容器总是重新启动
volumes: # 数据卷,目录挂载
- ./c_redis_6380/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./c_redis_6380/data:/data
ports:
- 6380:6380
- 16380:16380
command:
redis-server /usr/local/etc/redis/redis.conf
c_redis_6381: # 服务名称
image: redis:7.0.13 # 创建容器时所需的镜像
container_name: c_redis_6381 # 容器名称
restart: always # 容器总是重新启动
volumes: # 数据卷,目录挂载
- ./c_redis_6381/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./c_redis_6381/data:/data
ports:
- 6381:6381
- 16381:16381
command:
redis-server /usr/local/etc/redis/redis.conf
c_redis_6382: # 服务名称
image: redis:7.0.13 # 创建容器时所需的镜像
container_name: c_redis_6382 # 容器名称
restart: always # 容器总是重新启动
volumes: # 数据卷,目录挂载
- ./c_redis_6382/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./c_redis_6382/data:/data
ports:
- 6382:6382
- 16382:16382
command:
redis-server /usr/local/etc/redis/redis.conf
c_redis_6383: # 服务名称
image: redis:7.0.13 # 创建容器时所需的镜像
container_name: c_redis_6383 # 容器名称
restart: always # 容器总是重新启动
volumes: # 数据卷,目录挂载
- ./c_redis_6383/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./c_redis_6383/data:/data
ports:
- 6383:6383
- 16383:16383
command:
redis-server /usr/local/etc/redis/redis.conf
c_redis_6384: # 服务名称
image: redis:7.0.13 # 创建容器时所需的镜像
container_name: c_redis_6384 # 容器名称
restart: always # 容器总是重新启动
volumes: # 数据卷,目录挂载
- ./c_redis_6384/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./c_redis_6384/data:/data
ports:
- 6384:6384
- 16384:16384
command:
redis-server /usr/local/etc/redis/redis.conf
4 . 进入容器 , 搭建集群
上面只是启动了 6 个 Redis 实例,此时并没有构建成 Cluster 集群。执行以下命令 , 进入容器; (6选1)
docker exec -it c_redis_6379 bash
继续执行以下命令创建集群:
# 命令
redis-cli -a xxxxx --cluster create 192.168.2.13:6379 192.168.2.13:6380 192.168.2.13:6381 192.168.2.13:6382 192.168.2.13:6383 192.168.2.13:6384 --cluster-replicas 1
# 执行后输出
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.2.13:6383 to 192.168.2.13:6379
Adding replica 192.168.2.13:6384 to 192.168.2.13:6380
Adding replica 192.168.2.13:6382 to 192.168.2.13:6381
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: bc28058dd468b7b162c8e791ba834e330382c2cc 192.168.2.13:6379
slots:[0-5460] (5461 slots) master
M: a3aae6b212e6eda15abc099c68ef65d7861c6c15 192.168.2.13:6380
slots:[5461-10922] (5462 slots) master
M: ca6bcd96010ca4d23e4ededb391b63618428da2d 192.168.2.13:6381
slots:[10923-16383] (5461 slots) master
S: 043396fb0cc65259fa38a497d615648d1e1f58ca 192.168.2.13:6382
replicates ca6bcd96010ca4d23e4ededb391b63618428da2d
S: 35881b8f9797edde8bbd6f1b7c8fbd3ccd0e9788 192.168.2.13:6383
replicates bc28058dd468b7b162c8e791ba834e330382c2cc
S: e8b592b54829b8aae8ebf58156642cf067a43493 192.168.2.13:6384
replicates a3aae6b212e6eda15abc099c68ef65d7861c6c15
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.2.13:6379)
M: bc28058dd468b7b162c8e791ba834e330382c2cc 192.168.2.13:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: e8b592b54829b8aae8ebf58156642cf067a43493 192.168.2.13:6384
slots: (0 slots) slave
replicates a3aae6b212e6eda15abc099c68ef65d7861c6c15
S: 35881b8f9797edde8bbd6f1b7c8fbd3ccd0e9788 192.168.2.13:6383
slots: (0 slots) slave
replicates bc28058dd468b7b162c8e791ba834e330382c2cc
M: ca6bcd96010ca4d23e4ededb391b63618428da2d 192.168.2.13:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 043396fb0cc65259fa38a497d615648d1e1f58ca 192.168.2.13:6382
slots: (0 slots) slave
replicates ca6bcd96010ca4d23e4ededb391b63618428da2d
M: a3aae6b212e6eda15abc099c68ef65d7861c6c15 192.168.2.13:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
看到上面的输出即为 Cluster 集群配置完成。
方式二 直装版
适用场景 : 生产环境
1 . 环境配置
三台服务器,六个节点,每台服务器两个节点,组成一个redis分片集群。六个节点分别为:
10.0.41.11:13789 10.0.41.11:14789
10.0.41.12:13789 10.0.41.12:14789
10.0.41.13:13789 10.0.41.13:14789
2 . 集群搭建
1 . 在三台机器上,每台上创建两个redis.conf文件
bind 0.0.0.0
protected-mode yes
port ****
tcp-backlog 511
timeout 600
tcp-keepalive 0
daemonize yes
supervised no
pidfile "/data/redis/data_****/redis_****.pid"
loglevel notice
logfile "/data/redis/data_****/redis_****.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/data/redis/data_****/"
requirepass "**************"
io-threads-do-reads yes
io-threads 6
slave-serve-stale-data yes
slave-read-only no
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-timeout 600
repl-disable-tcp-nodelay no
repl-backlog-size 60mb
slave-priority 10
maxclients 30000
maxmemory 18gb
maxmemory-policy allkeys-lru
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
cluster-enabled yes
cluster-config-file node-*****.conf
cluster-node-timeout 15000
cluster-require-full-coverage no
cluster-allow-reads-when-down yes
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes
masterauth "***********"
最后一行的“masterauth”一定要加,因为搭建的redis集群是带权限的,不加这一条,分配主从的时候,从库连接主库会失败,日志会一直输出提示。
其中cluster开头的三条参数是指当前redis是开启的集群模式。
2 . 目录
# mkdir -p /data/redis/data_{13789,14789}
3 . 启动服务
/usr/local/redis/bin/redis-server /usr/local/redis/redis_****.conf
这个时候虽然所有节点都启动了,但是目前每个服务之间都是独立的,没有任何关系,还不能称之为集群。
4 . 创建集群
redis5.0之前的版本是通过redis集群命令工具redis-trib实现,需要安装ruby环境。redis5.0之后redis的源码src文件夹里面已经包含了redis-trib文件.
# /usr/src/redis/src/redis-trib.rb create --replicas 1 10.0.41.11:13789 10.0.41.11:14789 10.0.41.12:13789 10.0.41.12:14789 10.0.41.13:13789 10.0.41.13:14789 -a "***********"
或者也可以直接使用redis-cli将集群关联起来。
# /usr/local/redis/bin/redis-cli --cluster create --cluster-replicas 1 10.0.41.11:13789 10.0.41.11:14789 10.0.41.12:13789 10.0.41.12:14789 10.0.41.13:13789 10.0.41.13:14789 -a "**********"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.41.12:14789 to 10.0.41.11:13789
Adding replica 10.0.41.13:14789 to 10.0.41.12:13789
Adding replica 10.0.41.11:14789 to 10.0.41.13:13789
M: a452b32de46c6f03bd9261ac8e09544a88baa30b 10.0.41.11:13789
slots:[0-5460] (5461 slots) master
S: c4eb2e57e5acb01ff744844d1eb837b798457d2c 10.0.41.11:14789
replicates 33d48932656a7ade70d83010bad469271188311d
M: bf8c020329dfcbbcf4e7c89dc62ff7fbf678c316 10.0.41.12:13789
slots:[5461-10922] (5462 slots) master
S: 26b1f03fa8403dd820c5f0972c2498b64d1b7b80 10.0.41.12:14789
replicates 72868a5bfbbd823c2725b7a9b3e7975a3d60dff0
M: d826ef84bea246b26963697a2a6a17b6355b4120 10.0.41.13:13789
slots:[10923-16383] (5461 slots) master
S: 99a17fe29b7e36207d384a143eb4491fcffe68b2 10.0.41.13:14789
replicates 7554756718848314ea97401ff97f49576a3011b6
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 10.0.41.41:13789)
M: a452b32de46c6f03bd9261ac8e09544a88baa30b 10.0.41.11:13789
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: c4eb2e57e5acb01ff744844d1eb837b798457d2c 10.0.41.11:14789
slots: (0 slots) slave
replicates 33d48932656a7ade70d83010bad469271188311d
M: bf8c020329dfcbbcf4e7c89dc62ff7fbf678c316 10.0.41.12:13789
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 99a17fe29b7e36207d384a143eb4491fcffe68b2 10.0.41.13:14789
slots: (0 slots) slave
replicates 7554756718848314ea97401ff97f49576a3011b6
S: 26b1f03fa8403dd820c5f0972c2498b64d1b7b80 10.0.41.12:14789
slots: (0 slots) slave
replicates 72868a5bfbbd823c2725b7a9b3e7975a3d60dff0
M: d826ef84bea246b26963697a2a6a17b6355b4120 10.0.41.13:13789
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
备注:
- redis-cli —cluster 或者 redis=trib.rb:代表集群操作命令;
- create:代表创建集群;
- —replicas 1 或者 —cluster-replicas 1 指定集群中每个主节点的副本个数为1。计算公式:主节点的数量=节点总数÷(replicas + 1)。可以通过 redis-cli —cluster help 命令进行集群命令的查看。
集群相关命令
- 登录集群,正常登录redis命令,后面跟-c参数代表连接集群。
/usr/local/redis/bin/redis-cli -c -p 13789 -a "****************"
[root@bogon ~]# /usr/local/redis/bin/redis-cli -c -p 13789 -a "****************"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:13789> cluster nodes
c4eb2e57e5acb01ff744844d1eb837b798457d2c 10.0.41.11:14789@24789 slave d826ef84bea246b26963697a2a6a17b6355b4120 0 1677635874000 5 connected
26b1f03fa8403dd820c5f0972c2498b64d1b7b80 10.0.41.12:14789@24789 slave a452b32de46c6f03bd9261ac8e09544a88baa30b 0 1677635873000 1 connected
d826ef84bea246b26963697a2a6a17b6355b4120 10.0.41.13:13789@23789 master - 0 1677635873000 5 connected 10923-16383
99a17fe29b7e36207d384a143eb4491fcffe68b2 10.0.41.13:14789@24789 slave bf8c020329dfcbbcf4e7c89dc62ff7fbf678c316 0 1677635874981 3 connected
a452b32de46c6f03bd9261ac8e09544a88baa30b 10.0.41.11:13789@23789 myself,master - 0 1677635871000 1 connected 0-5460
bf8c020329dfcbbcf4e7c89dc62ff7fbf678c316 10.0.41.12:13789@23789 master - 0 1677635873977 3 connected 5461-10922
127.0.0.1:13789>
- cluster info:显示集群基本信息(节点数量,故障转移等)
- cluster meet:将一个节点加入到集群
- cluster addslots:将一个或多个槽位分配给一个节点。
- cluster delslots:将一个或多个槽位从一个节点中删除。
- cluster replicate:将一个节点设置为另一个节点的从节点。
- cluster failover:手动触发一次故障转移操作。
- cluster setslot:设置一个槽位的状态。
- cluster countkeysinslot:统计一个槽位中的键值对数量。
- cluster getkeysinslot: 列出一个槽位中的所有键。
- cluster flushslot:删除一个槽位中的所有键值对。
- cluster reset:重置集群状态。
- cluster saveconfig:将集群配置保存到磁盘。
- cluster loadconfig:从磁盘加载集群配置。