记一次 Ziplist 导致 Redis Core Dump 分析

虽然Redis release notes里经常有提到，由于某bug会造成Redis core了。也许很幸运，使用多年官方的Redis，我确实没有遇到过core(ps：目前管理实例数是六位数的)。讲真，写这么一篇文章，我瑟瑟发抖，一直以来虽然我一直做 Redis 相关的工作，但处理这种问题不太多，所以哪里写的不对，欢迎指正。

清明节，突然收到报警某个Redis slave节点down了，迅速排查下，发现进程不在了，一般这种很大可能是因为机器OOM了，查了一下机器内存空闲70%，于是看了一下竟然core了。

Redis的版本是3.2.12

一、Redis的crash log

Redis的crash log还是非常给力的，里面包含了info、client list、当时堆栈信息、当前客户端的一些信息，下面可以看下。

1. 概览

可以看到t zset.c的1129行(t zset.c:1129 'dictAdd(zs->dict,ele,&node->score) == DICT_OK' is not true)造成crash，当时Object的信息也给出来了：sorted类型、encoding=5代表ziplist类型。

=== REDIS BUG REPORT START: Cut & paste starting from here ===
23368:S 05 Apr 18:18:46.621 # === ASSERTION FAILED OBJECT CONTEXT ===
23368:S 05 Apr 18:18:46.621 # Object type: 3
23368:S 05 Apr 18:18:46.621 # Object encoding: 5
23368:S 05 Apr 18:18:46.621 # Object refcount: 1
23368:S 05 Apr 18:18:46.621 # Sorted set size: 257
23368:S 05 Apr 18:18:46.621 # === ASSERTION FAILED ===
23368:S 05 Apr 18:18:46.621 # ==> t_zset.c:1129 'dictAdd(zs->dict,ele,&node->score) == DICT_OK' is not true
23368:S 05 Apr 18:18:46.621 # (forcing SIGSEGV to print the bug report.)
23368:S 05 Apr 18:18:46.621 # Redis 3.2.12 crashed by signal: 11
23368:S 05 Apr 18:18:46.621 # Crashed running the instuction at: 0x45c25f
23368:S 05 Apr 18:18:46.621 # Accessing address: 0xffffffffffffffff
23368:S 05 Apr 18:18:46.621 # Failed assertion: dictAdd(zs->dict,ele,&node->score) == DICT_OK (t_zset.c:1129)

1129行代码如下：

2.堆栈

从堆栈可以看出来，Redis在处理zadd命令后发生了core，最后一个方法栈就是zsetConvert，其中1129行就在这里面。

------ STACK TRACE ------
EIP:
bin/redis-server *:13676(_serverAssert+0x6f)[0x45c25f]

Backtrace:
bin/redis-server *:13676(logStackTrace+0x3c)[0x45c78c]
bin/redis-server *:13676(sigsegvHandler+0xa3)[0x45d673]
/lib64/libpthread.so.0(+0xf5e0)[0x7fbbe6b605e0]
bin/redis-server *:13676(_serverAssert+0x6f)[0x45c25f]
bin/redis-server *:13676(zsetConvert+0x27d)[0x44de5d]
bin/redis-server *:13676(zaddGenericCommand+0x707)[0x44e5f7]
bin/redis-server *:13676(call+0x72)[0x424652]
bin/redis-server *:13676(processCommand+0x365)[0x42a5f5]
bin/redis-server *:13676(processInputBuffer+0x109)[0x435349]
bin/redis-server *:13676(aeProcessEvents+0x1ab)[0x41fc7b]
bin/redis-server *:13676(aeMain+0x2b)[0x41ff8b]
bin/redis-server *:13676(main+0x370)[0x4277d0]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fbbe67afc05]
bin/redis-server *:13676[0x41d3d9]

3.info all

crash log还包含了info all信息，这个对于排查问题也很有帮助，不过从下面的info并没有发现明显异常。

------ INFO OUTPUT ------
# Server
redis_version:3.2.12
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:66352db37a13e34e
redis_mode:standalone
os:Linux 3.10.0-693.5.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:23368
run_id:08d8b04f78f076ed6cab61f385e97133a829c6c3
tcp_port:13676
uptime_in_seconds:11604878
uptime_in_days:134
hz:10
lru_clock:10955654
executable:/xxxxxxxxxx/bin/redis-server
config_file:/xxxxxxxxxx/conf/redis.conf

# Clients
connected_clients:8
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:1130091432
used_memory_human:1.05G
used_memory_rss:1165987840
used_memory_rss_human:1.09G
used_memory_peak:1130145776
used_memory_peak_human:1.05G
total_system_memory:540622544896
total_system_memory_human:503.49G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:5368709120
maxmemory_human:5.00G
maxmemory_policy:allkeys-lru
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-4.0.3

# Persistence
loading:0
rdb_changes_since_last_save:250957118
rdb_bgsave_in_progress:0
rdb_last_save_time:1542854648
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:1342911
total_commands_processed:324231805
instantaneous_ops_per_sec:109
total_net_input_bytes:19959176009
total_net_output_bytes:28617243060
instantaneous_input_kbps:6.28
instantaneous_output_kbps:1.25
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0

# Replication
role:slave
master_host:10.xx.xx.11
master_port:12676
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:16637646872
slave_priority:100
slave_read_only:0
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:67108864
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:3061.99
used_cpu_user:4084.48
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats
cmdstat_set:calls=2580969,usec=5211206,usec_per_call=2.02
cmdstat_del:calls=852409,usec=1577987,usec_per_call=1.85
cmdstat_zadd:calls=8108213,usec=185116242,usec_per_call=22.83
cmdstat_hincrby:calls=215200524,usec=1314563137,usec_per_call=6.11
cmdstat_hdel:calls=13525817,usec=40043633,usec_per_call=2.96
cmdstat_select:calls=1,usec=1,usec_per_call=1.00
cmdstat_expire:calls=10689186,usec=12850505,usec_per_call=1.20
cmdstat_ping:calls=34863107,usec=15125982,usec_per_call=0.43
cmdstat_info:calls=4209908,usec=157930292,usec_per_call=37.51
cmdstat_slaveof:calls=1,usec=44,usec_per_call=44.00
cmdstat_config:calls=417328,usec=4405040,usec_per_call=10.56
cmdstat_subscribe:calls=3,usec=7,usec_per_call=2.33
cmdstat_publish:calls=33774669,usec=85809478,usec_per_call=2.54
cmdstat_client:calls=6,usec=3,usec_per_call=0.50
cmdstat_slowlog:calls=9664,usec=33127,usec_per_call=3.43

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1502434,expires=251409,avg_ttl=0
hash_init_value: 1543499382

4. client list

client list包含当前所有连接Redis的客户端，可以看到包含了一些info、publish、ping。这些都是sentinel的定时操作，其中发现zadd命令在其中。

------ CLIENT LIST OUTPUT ------
id=5 addr=10.xx.131.11:12676 fd=6 name= age=11604877 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=zadd
id=6 addr=10.xx.73.30:58144 fd=7 name=sentinel-5c446d86-cmd age=11604872 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=7 addr=10.xx.73.30:58146 fd=8 name=sentinel-5c446d86-pubsub age=11604872 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=8 addr=10.xx.73.28:42216 fd=9 name=sentinel-0c5acb8f-cmd age=11604872 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=publish
id=9 addr=10.xx.73.28:42218 fd=10 name=sentinel-0c5acb8f-pubsub age=11604872 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=10 addr=10.xx.73.29:39148 fd=11 name=sentinel-8257b663-pubsub age=11604872 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=11 addr=10.xx.xx.29:39146 fd=12 name=sentinel-8257b663-cmd age=11604872 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping
id=1260384 addr=10.xx.xx.38:49728 fd=13 name= age=692578 idle=179 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info

5.当前客户端

从下面可以看出当前客户端正在执行如下操作

zadd fpgn_622094516 1.554459526619E9 9909414677

并且给出key的一些信息，譬如它是有序集合，ziplist(encoding=5), 很重要的一个信息就是它的长度是257，这个数字非常敏感，因为我们知道Redis提供了ziplist的相关优化，针对有序集合有一个配置是zset-max-ziplist-entries。

------ CURRENT CLIENT INFO ------
id=5 addr=10.54.131.11:12676 fd=6 name= age=11604877 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=zadd
argv[0]: 'ZADD'
argv[1]: 'fpgn_622094516'
argv[2]: '1.554459526619E9'
argv[3]: '9909414677'
23368:S 05 Apr 18:18:46.628 # key 'fpgn_622094516' found in DB containing the following object:
23368:S 05 Apr 18:18:46.628 # Object type: 3
23368:S 05 Apr 18:18:46.628 # Object encoding: 5
23368:S 05 Apr 18:18:46.628 # Object refcount: 1
23368:S 05 Apr 18:18:46.628 # Sorted set size: 257

6. 寄存器相关信息

------ REGISTERS ------
23368:S 05 Apr 18:18:46.628 # 
RAX:0000000000000000 RBX:0000000000000469
RCX:0000000001cc0700 RDX:00007fbbe6b4a7b8
RDI:00007fbbe6b4a760 RSI:0000000000000000
RBP:00000000004e5770 RSP:00007ffd9f14e210
R8 :0000000000000001 R9 :0000000000000000
R10:000000000000002a R11:0000000000000206
R12:00000000004e5a40 R13:00007ffd9f14e27c
R14:00007ffd9f14e258 R15:00007ffd9f14e268
RIP:000000000045c25f EFL:0000000000010202
CSGSFS:7562000000000033
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e21f) -> 00007fbb990aafa0
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e21e) -> 00007fbb990aafa0
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e21d) -> 000000000044db74
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e21c) -> 00007fbba991664b
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e21b) -> 00007fbba9916651
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e21a) -> 0000000000000000
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e219) -> 000000005bbaee1b
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e218) -> 00007fbba991735b
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e217) -> 00007fbbd5570ce0
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e216) -> 41d729ca59250e56
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e215) -> 00007fbb990d8720
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e214) -> 000000000000000a
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e213) -> 000000000044de5d
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e212) -> 00007ffd9f14e260
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e211) -> 00007fbba9915800
23368:S 05 Apr 18:18:46.628 # (00007ffd9f14e210) -> 00007fbbddc78c60

7. 内存测试

Redis提供了--test-memory功能用来测试内存分配，从下面的测试结果看没有出现内存分配不足（譬如OOM）这类问题。

------ FAST MEMORY TEST ------
23368:S 05 Apr 18:18:46.629 # Bio thread for job type #0 terminated
23368:S 05 Apr 18:18:46.629 # Bio thread for job type #1 terminated
*** Preparing to test memory region 722000 (94208 bytes)
*** Preparing to test memory region 1cb3000 (135168 bytes)
*** Preparing to test memory region 7fbb99000000 (1163919360 bytes)
*** Preparing to test memory region 7fbbde7ff000 (10485760 bytes)
*** Preparing to test memory region 7fbbdf200000 (10485760 bytes)
*** Preparing to test memory region 7fbbdfc00000 (2097152 bytes)
*** Preparing to test memory region 7fbbe6400000 (2097152 bytes)
*** Preparing to test memory region 7fbbe6b4c000 (20480 bytes)
*** Preparing to test memory region 7fbbe6d69000 (16384 bytes)
*** Preparing to test memory region 7fbbe7480000 (16384 bytes)
*** Preparing to test memory region 7fbbe7492000 (4096 bytes)
*** Preparing to test memory region 7fbbe7493000 (4096 bytes)
*** Preparing to test memory region 7fbbe7496000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: _serverAssert (base: 0x45c1f0)
Module: bin/redis-server *:13676 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x45c1f0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
23368:S 05 Apr 18:18:50.244 # dump of function (hexdump of 239 bytes):
48895c24e848896c24f089d34c896424f84883ec184989fc4889f5e810ffffffbee5824e00bf0300000031c0e83f9dfcff4d89e089d94889eabefe824e00bf0300000031c0e8269dfcffbe38894e00bf0300000031c04c8925bbb42d0048892dbcb42d00891dbeb42d00e8019dfcffc60425ffffffff78488b1c24488b6c24084c8b6424104883c418c3660f1f44000048895c24d048896c24d84889fb4c896424e04c896c24e84c897424f04c897c24f84881ec680800004883fe040f86ae0000004889f54883ed040f84a1000000488d7c2410b90001000031c04531e44c8d7b054c8db42410080000f348abeb09

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/antirez/redis/issues

  Suspect RAM error? Use redis-server --test-memory to verify it.

二、源码分析

初步分析是：zadd后当有序集合需要从ziplist转成skiplist这个期间造成了core，其中最奇怪的是dictAdd插入失败，造成core：

serverAssertWithInfo(NULL,zobj,dictAdd(zs->dict,ele,&node->score) == DICT_OK);

#define serverAssertWithInfo(_c,_o,_e) ((_e)?(void)0 : (_serverAssertWithInfo(_c,_o,#_e,__FILE__,__LINE__),_exit(1)))

看下什么原因会造成dictAdd会造成失败，很明显如果哈希表包含了元素，则插入失败。

三、内存分析

所以最好的方法就是分析下当前ziplist里都是什么就比较清楚了，于是在朋友的帮助下使用GDB完成了ziplist的分析，具体分析如下：

1. gdb redis-server core.pid

2. bt查看当前堆栈

3. 跳到zsetConvert函数

$ gdb redis-server core.23368
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/redis_server/gdb/src/redis-server...done.
[New LWP 23368]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `bin/redis-server *:13676        '.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fbbe67c34c7 in kill () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 libgcc-4.8.5-16.el7.x86_64
(gdb) bt
#0  0x00007fbbe67c34c7 in kill () from /lib64/libc.so.6
#1  0x000000000045d7a6 in sigsegvHandler (sig=11, info=<optimized out>, secret=<optimized out>) at debug.c:1107
#2  <signal handler called>
#3  _serverAssert (estr=0x4e5a40 "dictAdd(zs->dict,ele,&node->score) == DICT_OK", file=0x4e5770 "t_zset.c", line=1129) at debug.c:567
#4  0x000000000044de5d in zsetConvert (zobj=0x7fbbd5570ce0, encoding=<optimized out>) at t_zset.c:1117
#5  0x000000000044e5f7 in zaddGenericCommand (c=0x7fbbdfd2ac00, flags=0) at t_zset.c:1328
#6  0x0000000000424652 in call (c=0x7fbbdfd2ac00, flags=15) at server.c:2265
#7  0x000000000042a5f5 in processCommand (c=0x7fbbdfd2ac00) at server.c:2544
#8  0x0000000000435349 in processInputBuffer (c=0x7fbbdfd2ac00) at networking.c:1311
#9  0x000000000041fc7b in aeProcessEvents (eventLoop=0x7fbbdfc520a0, flags=3) at ae.c:431
#10 0x000000000041ff8b in aeMain (eventLoop=0x7fbbdfc520a0) at ae.c:489
#11 0x00000000004277d0 in main (argc=<optimized out>, argv=0x7ffd9f14e5a8) at server.c:4133
(gdb) up 4
#4  0x000000000044de5d in zsetConvert (zobj=0x7fbbd5570ce0, encoding=<optimized out>) at t_zset.c:1117
1117            serverAssertWithInfo(NULL,zobj,sptr != NULL);

4. print zl可以看到ziplist的起始地址0x7fbba9915800

5. print *(uint32_t *)zl 可以看到ziplist的整个内存长度

6. 算一下内存的结束地址就是0x7fbba9915800 + 7046，把它dump出来，文件名是zl.bin

(gdb) print zl
$1 = (unsigned char *) 0x7fbba9915800 "\206\033"
(gdb) print *(uint32_t *)zl
$2 = 7046
(gdb) dump binary memory zl.bin 0x7fbba9915800 0x7fbba9917386
(gdb)

7.搞出ziplist的元素

感谢Redis的代码做的非常好，可以直接抽出ziplist.h和ziplist.c的代码，然后直接分析即可。

#include <stdio.h>
#include <string.h>
#include "ziplist.h"

int main(int argc, char **argv, char **env)
{
    FILE *fp = fopen("zl.bin", "r");
    if (fp == NULL) {
    return -1;
    }

    unsigned char *buff;
    const int buff_len = 1024 * 1024;
    int n;

    buff = (unsigned char *)malloc(sizeof(unsigned char) * buff_len);
    memset(buff, 0, buff_len);
    n = fread(buff, 1, buff_len, fp);
    printf("read %d bytes from file\n", n);

    unsigned char *zl = buff;
    printf("ziplist ziplistBlobLen: %d\n", ziplistBlobLen(zl));
    printf("ziplist length: %d\n", ziplistLen(zl));

    unsigned char *vstr;
    unsigned int vlen;
    long long vlong;
    int count = 0;

    unsigned char *eptr = ziplistIndex(zl, 0);
    while (eptr != NULL) {
        if (!ziplistGet(eptr, &vstr, &vlen, &vlong)) {
            printf("ziplistGet error\n");
            return -1;
        }
        if (vstr != NULL) {
            printf("%.*s", vlen, vstr);
        } else {
            printf("%ld", vlong);
        }

        count++;
        if (count % 2 == 0) {
            printf("\n");
        } else {
            printf("\t");
        }
        eptr = ziplistNext(zl, eptr);
    }

    return 1;
}

8.打印结果

打印结果如下：

9214265314  1553754978.737
9219488144  1553870247.78
9231561063  1553870251.464
9242373948  1553997540.007
9259890295  1554028774.51
9257100883  1554028775.9579999
1538977307  1554028801.122
1092128962  1554028836.783
11876497062 1554431862.875
11851454532 1554431882.921
891110276   1554434117.8499999
894962802   1554434119.3150001
897740518   1554434128.2160001
907153595   1554434129.7079999
907181453   1554434131.076
911054502   1554434132.5280001
914890585   1554434142.335
920964455   1554434145.152
925969447   1554434146.684
944318152   1554434148.187
11941804660 1554436129.1800001
11777171286 1554458806.0120001
11742093620 1554458807.6889999
11715534010 1554458809.21
11672716772 1554458810.5769999
11613432078 1554458811.905
11550414593 1554458813.118
11517996744 1554458814.3759999
11500440617 1554458815.6140001
11478161354 1554458816.8369999
11414301904 1554458818.0739999
11346206035 1554458819.346
11291115246 1554458820.677
11222756856 1554458821.925
11150265728 1554458823.283
10970608813 1554458824.562
10938234745 1554458825.924
10899399365 1554458827.2939999
10767275989 1554458828.6630001
10636374981 1554458829.97
10570534567 1554458831.2850001
10413570162 1554458832.5929999
10322754271 1554458833.9170001
10288894258 1554458835.2820001
10193805582 1554458836.668
10076352932 1554458838.0050001
10029661763 1554458839.3329999
9949953433  1554458840.605
9921804300  1554458841.905
9876247658  1554458843.214
9838633213  1554458844.45
9827369980  1554458845.6900001
9834835967  1554458846.911
9863721076  1554458848.1240001
9913807532  1554458849.405
9925439612  1554458850.7290001
9965771927  1554458851.9749999
10052127085 1554458853.3410001
10166124816 1554458854.6240001
10272503804 1554458855.911
10299486881 1554458857.175
10338316514 1554458858.552
10581422345 1554458859.948
10715267394 1554458861.3800001
10803628815 1554458862.618
10929235600 1554458863.9159999
10969664947 1554458865.342
11066226539 1554458866.737
11175727068 1554458868.1530001
11263315139 1554458869.378
11304479114 1554458870.6300001
11370286250 1554458871.9319999
11389491458 1554458873.3340001
11427628908 1554458874.6600001
11456536043 1554458876.2149999
11516371481 1554458877.7160001
11540708208 1554458879.375
11582620855 1554458880.8429999
11655652844 1554458882.3169999
11698754987 1554458883.7149999
9803405207  1554458885.1140001
9781060124  1554458886.5120001
9720915125  1554458888.105
9680646105  1554458889.4679999
9662998980  1554458890.8139999
9652810907  1554458892.348
9628039023  1554458896.3169999
9557079656  1554458897.5999999
9536221322  1554458898.914
9516506403  1554458900.1849999
9525740535  1554458901.5139999
9544370106  1554458902.8239999
9580825848  1554458904.171
9644207950  1554458905.5610001
9661134686  1554458906.8570001
9667805988  1554458908.1689999
9707690522  1554458909.7880001
9739840217  1554458911.3169999
9793667195  1554458913.155
9814930331  1554458915.027
9467104725  1554458917.8110001
9460402940  1554458919.299
9443012513  1554458920.7320001
9422448568  1554458922.013
9390005464  1554458923.352
9350505954  1554458924.721
9316738871  1554458926.098
9304287892  1554458927.517
7665813588  1554458929.5829999
5132031042  1554458930.9319999
11789573057 1554458933.431
11770218038 1554458937.118
9473751026  1554458942.5869999
9464969176  1554458946.0139999
9451812057  1554458948.2290001
9430611330  1554458949.994
9401012549  1554458951.635
9380107339  1554458953.2149999
9337256629  1554458954.8299999
5131960833  1554458957.3659999
4517860072  1554458959.191
2844765977  1554458962.0469999
2831001627  1554458963.858
2545262460  1554458965.427
1597238347  1554458967.0150001
1275903665  1554458968.5280001
1182075767  1554458969.951
1151554587  1554458971.7420001
1121506654  1554458973.187
1119751110  1554458974.8789999
1112271696  1554458976.345
1151500542  1554458977.733
1115899417  1554458979.1300001
1538977307  1554458980.579
1274523566  1554458982.382
1680459122  1554458983.9990001
2829501050  1554458985.4449999
2836117692  1554458986.868
3259875873  1554458988.329
1104057891  1554458989.888
1092128962  1554458991.392
1029490033  1554458992.8529999
1022869312  1554458994.3380001
1016040233  1554458995.862
989853532   1554458997.431
972561774   1554458998.9519999
965710674   1554459000.4519999
957049365   1554459001.8410001
939739488   1554459003.4809999
949177731   1554459005.115
960613220   1554459006.6300001
966557783   1554459008.1340001
965679520   1554459009.5680001
969338958   1554459011.0840001
1003210949  1554459012.8150001
1018888308  1554459014.5320001
1016121625  1554459016.0810001
1029148729  1554459017.7579999
1091660523  1554459019.53
9312712062  1554459021.4820001
9268466953  1554459023.6029999
7627407505  1554459026.73
11927650707 1554459031.632
11899902967 1554459033.2449999
11829370281 1554459248.7030001
11799126653 1554459252.5150001
11855731105 1554459256.5120001
11876840880 1554459258.8510001
11769638514 1554459262.5810001
11742438742 1554459264.26
11722142361 1554459266.052
11698302198 1554459269.819
11674927600 1554459271.349
11620531706 1554459275.6110001
11593826867 1554459277.1470001
11652331020 1554459280.7019999
11544287484 1554459282.3510001
11566492789 1554459285.733
11523335182 1554459289.5369999
11499632665 1554459291.3340001
11479496181 1554459294.1949999
11446109398 1554459297.1530001
11375611642 1554459301.8959999
11419583096 1554459303.8929999
11395153806 1554459305.506
11334667261 1554459308.204
11313762930 1554459309.743
11355248256 1554459311.503
11284309708 1554459315.5680001
11255776440 1554459317.1129999
11230762164 1554459318.6400001
11180118692 1554459319.957
11173103151 1554459321.404
11153679337 1554459322.869
11122412124 1554459324.336
11098304804 1554459325.898
11194229625 1554459330.7
11210997991 1554459332.1949999
11021285607 1554459334.155
10951395470 1554459338.9330001
10933033189 1554459340.45
10909504057 1554459342.1159999
11077766158 1554459345.513
11059689566 1554459347.3859999
11040152098 1554459349.0369999
10829099231 1554459353.237
10802780063 1554459354.72
10793874021 1554459356.339
10978316673 1554459360.506
10879528496 1554459361.9300001
10852033097 1554459363.365
10779040475 1554459368.175
10750780775 1554459369.651
10723523678 1554459371.0710001
10695037719 1554459372.4879999
10643170157 1554459373.9400001
10616854605 1554459375.3510001
10669969745 1554459377.388
10324013139 1554459406.0409999
10300254143 1554459407.8570001
10273675648 1554459411.684
10249228453 1554459413.414
10225537311 1554459416.954
10180347038 1554459419.21
10136770975 1554459420.7939999
10159712300 1554459422.1559999
10110131870 1554459425.727
10067720184 1554459427.404
10026495289 1554459430.0209999
9985320927  1554459431.858
9944381217  1554459436.247
10004339597 1554459439.868
9967176641  1554459441.5999999
9927499934  1554459446.0929999
9460008595  1554459465.4100001
9476346126  1554459470.974
9497113499  1554459472.845
9519284473  1554459476.529
9534525191  1554459478.26
9551711207  1554459481.348
9583121160  1554459484.4000001
9622885843  1554459486.0420001
9645088236  1554459489.53
9668057612  1554459493.8139999
9694780480  1554459495.608
9711090967  1554459498.7190001
9725196402  1554459500.4100001
9740875788  1554459503.3859999
9761084768  1554459507.277
9780834435  1554459508.8789999
9796914997  1554459512.2090001
9812416676  1554459514.0469999
9828607002  1554459517.165
9850282068  1554459520.7179999
9866725779  1554459522.5090001
9887380747  1554459525.063
9909414677  1554459526.619

从结果可以看出，这个ziplist的数据，第134个和第7个key一样了。至此问题发现了，但是就没有但是了，为什么会出现这种情况就不知道了，我也提交了issue(#5995)暂时没有回复，看看哪位大佬知道怎么进一步分析。还有一些信息：

当时core的是slave节点，master节点正常，slave重启后就恢复了。

期待大佬...........

欢迎订阅我的公众号(Redis开发运维实战)：关注Redis开发运维实战相关问题，干掉所有的坑。

由于本公众号是新号，没有留言功能，欢迎加我然后进群讨论，同是也欢迎投稿。