因单个程序访问redis流量超出980Mb,单个redis不能满足业务需要,需要做redis主从,让单个程序从redis从只读,主的数据为1.7G,做完从后,发现数据没有同步过来,查看从库日志
18546:S 27 Dec 11:31:01.738 * Connecting to MASTER 192.170.5.88:6379
18546:S 27 Dec 11:31:01.738 * MASTER <-> SLAVE sync started
18546:S 27 Dec 11:31:01.738 * Non blocking connect for SYNC fired the event.
18546:S 27 Dec 11:31:01.740 * Master replied to PING, replication can continue...
18546:S 27 Dec 11:31:01.742 * Partial resynchronization not possible (no cached master)
18546:S 27 Dec 11:31:01.924 * Full resync from master: 8a8b019dfa1192f81f404c68075d4c5dbc0467fd:473627650
18546:S 27 Dec 11:32:34.235 * MASTER <-> SLAVE sync: receiving 1824088898 bytes from master
18546:S 27 Dec 11:32:47.160 # I/O error trying to sync with MASTER: connection lost
18546:S 27 Dec 11:32:48.154 * Connecting to MASTER 192.170.5.88:6379
18546:S 27 Dec 11:32:48.154 * MASTER <-> SLAVE sync started
18546:S 27 Dec 11:32:48.155 * Non blocking connect for SYNC fired the event.
18546:S 27 Dec 11:32:48.155 * Master replied to PING, replication can continue...
18546:S 27 Dec 11:32:48.167 * Partial resynchronization not possible (no cached master)
18546:S 27 Dec 11:32:48.428 * Full resync from master: 8a8b019dfa1192f81f404c68075d4c5dbc0467fd:614945359
18546:S 27 Dec 11:34:21.233 * MASTER <-> SLAVE sync: receiving 1823925331 bytes from master
18546:S 27 Dec 11:34:27.113 # I/O error trying to sync with MASTER: connection lost
##错误I/O error trying to sync with MASTER: connection lost
再来查看主库日志,发现在redis主库在接到从库要求重新同步数据的时候先生成一个rdb文件,再通过psync来做部分同步,可以看到问题就出在部分同步这一块,由于client-output-buffer-limit值设置太小,导致导数发送失败。失败后,从库继续发起数据同步的请求,但是每次都失败,redis主库反复地生成rdb文件,虽然是有fork一个子进程,但是对redis主库的吞吐量是有一定影响的。
解决问题:
在主库操作
127.0.0.1:6379> config get client-output-buffer-limit
1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
可以看到,目前的限制是最大256M和60s内不超过64M,从我们上面日志信息可以看到psync的数据明显是大于256M的。我们进行如下设置把很限制调大:
config set client-output-buffer-limit 'slave 1073741824 268435456 60'
调完后再观察,发现从库的的复制状态很快就变成了up。
我们上面的调整是加大复制输出缓冲区,还有一个办法就是关闭复制输出缓冲区的限制:
config set client-output-buffer-limit 'slave 0 0 0'