内容简介：记一次 Redis 内存诡异增长，由于 一次 Redis Rehash 造成的内存暴增。
记一次 Redis 内存诡异增长，由于 一次 Redis Rehash 造成的内存暴增。
一、现象
- 实例名：r-bp1cxxxxxxxxxd04(主从)
- 时间：2017-11-16 12:26~12:27
- 问题：一分钟内存上涨了2G，如下图所示：
- 键值规模：6000万左右
二、Redis内存分析
1. 内存组成
上图中的内存统计的是Redis的info memory命令中的used_memory属性，例如：
redis> info memory# Memoryused_memory:9195978072used_memory_human:8.56Gused_memory_rss:9358786560used_memory_peak:10190212744used_memory_peak_human:9.49Gused_memory_lua:38912mem_fragmentation_ratio:1.02mem_allocator:jemalloc-3.6.0
每个属性的详细说明
计算公式如下：
used_memory = 自身内存+对象内存+缓冲内存+lua内存used_rss = used_memory + 内存碎片
2. 内存分析
(1) 自身内存：一个空的Redis占用很小，可以忽略不计
(2) kv内存：key对象 + value对象
(3) 缓冲区：客户端缓冲区(普通 + slave伪装 + pubsub)以及aof缓冲区(比较固定，一般没问题)
(4) Lua：Lua引擎所消耗的内存
3. 内存突增常见问题
(1) kv内存：bigkey、大量写入
(2) 客户端缓冲区：一般常见的有普通客户端缓冲区(例如monitor命令)或者pubsub客户端缓冲区
三、问题排查
(1) bigkey ? 经扫描未发现bigkey
Sampled67234427keysinthe keyspace! Totalkey length inbytesis1574032382(avg len 23.41) Biggeststringfound'CCARD_DEVICE_CARD_REF_MAP_KEY_016817000004209'has20862bytes Biggest list found 'CCARD_VALID_DEVICE_TRAIN_QUEUE_KEY'has51items Biggest hash found'CCARD_VALID_DEVICE_TRAIN_MAP_KEY'has51fields67234359 stringswith71767890bytes(100.00%of keys,avg size1.07)67listswith151items(00.00%of keys,avg size2.25)0setswith0members(00.00%of keys,avg size0.00)1hashswith51fields(00.00%of keys,avg size51.00)0zsetswith0members(00.00%of keys,avg size0.00)
(2) 键值个数增加？未发现键值有明显变化
(3) 客户端缓冲区
由于内存增上去后，长时间没下落，如果是因为缓冲区问题，会从info clients找到明显问题，执行后发现：
id=80207 addr=10.xx.0.4:63920 fd=46 name= age=624 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80215 addr=10.xx.0.23:43489 fd=36 name= age=591 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80366 addr=10.xx.0.8:59785 fd=18 name= age=84 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=del read=0 write=0 type=user id=80356 addr=10.xx.0.33:32117 fd=13 name= age=114 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80064 addr=10.xx.59.4:53446 fd=38 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin id=80276 addr=10.xx.0.23:48511 fd=8 name= age=387 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80188 addr=10.xx.0.33:16265 fd=42 name= age=681 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80326 addr=10.xx.0.32:59779 fd=16 name= age=209 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80065 addr=10.xx.59.4:53447 fd=45 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin id=79936 addr=10.xx.0.22:10607 fd=30 name= age=1480 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80174 addr=10.xx.0.5:60914 fd=6 name= age=722 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80300 addr=10.xx.0.22:22757 fd=48 name= age=298 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80037 addr=10.xx.0.5:55189 fd=15 name= age=1143 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80330 addr=10.xx.0.8:48533 fd=17 name= age=199 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=79896 addr=10.xx.0.30:26814 fd=11 name= age=1616 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80299 addr=10.xx.0.24:11227 fd=44 name= age=303 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80086 addr=10.xx.0.32:52526 fd=40 name= age=1002 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80202 addr=10.xx.0.33:16658 fd=26 name= age=636 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80256 addr=10.xx.0.24:60496 fd=19 name= age=448 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=79908 addr=10.xx.0.29:18975 fd=12 name= age=1583 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80365 addr=10.xx.0.29:46429 fd=14 name= age=85 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=79869 addr=10.xx.27.4:48455 fd=35 name= age=1700 idle=1700 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin id=80334 addr=10.xx.0.23:50012 fd=39 name= age=189 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80041 addr=10.xx.0.32:51107 fd=33 name= age=1132 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=79992 addr=10.xx.0.22:12068 fd=28 name= age=1289 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80251 addr=10.xx.0.30:44213 fd=23 name= age=468 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80006 addr=10.xx.0.2:45895 fd=31 name= age=1242 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80321 addr=10.xx.0.30:48048 fd=5 name= age=224 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80381 addr=10.xx.0.8:13360 fd=22 name= age=24 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=del read=0 write=0 type=user id=80200 addr=10.xx.0.24:59183 fd=24 name= age=640 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80113 addr=10.xx.0.2:52492 fd=21 name= age=915 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=174 addr=11.216.117.242:53027 fd=9 name= age=281390 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=replconf read=0 write=0 type=admin id=79991 addr=10.xx.0.4:48412 fd=25 name= age=1296 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80301 addr=127.0.0.1:47869 fd=49 name= age=291 idle=261 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=strlen read=0 write=0 type=admin id=80047 addr=10.xx.59.4:53184 fd=41 name= age=1114 idle=1114 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin id=80236 addr=10.xx.0.5:62546 fd=47 name= age=516 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80364 addr=10.xx.0.4:18794 fd=7 name= age=85 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80175 addr=10.xx.0.4:62245 fd=29 name= age=718 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80336 addr=10.xx.0.29:45701 fd=50 name= age=180 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80050 addr=10.xx.59.4:53188 fd=43 name= age=1114 idle=1114 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL read=0 write=0 type=admin id=79765 addr=10.xx.0.2:33832 fd=37 name= age=2027 idle=177 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info read=0 write=0 type=user id=80170 addr=10.xx.0.2:57853 fd=20 name= age=728 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping read=0 write=0 type=user id=80390 addr=127.0.0.1:49449 fd=27 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client read=0 write=0 type=admin
四、揪出元凶
常用的几招都用了，还是不行，同事@径远帮忙一起分析，怀疑是不是因为Redis的kv哈希表做了 rehash。
1. Redis的kv存储结构
如下图所示，Redis的所有kv保存在dict中，其中ht对应两个哈希表ht[0]和ht[1]，平时一个空闲，一个用于存储数据，只有当需要rehash时，ht[1]才会用到。
2. Redis的字典rehash
为了保证哈希表的负载，当哈希表的元素个数等于哈希表槽数时候，会进行rehash扩容。扩容后h[1]的容量等于第一个大于等于ht[0].size*2的2n，例如hash表的初始化容量是4，那么下一次扩容就是8，以此类推。
3. 测试
(1) 测试方法
先批量写入到rehash阈值附近，然后在逐条去写，观察内存变化
// 为每个键设置1天过期时间int expireTime = 60 * 60 * 24;// rehash阈值 - 50为了方便观察rehash内存变化int rehashThreshold = (int) Math.pow(2, 25) - 50;// 1.批量写入：pipeline批量写入，由于是本机测试，这里用10000，实际生产不要这么用Pipeline pipeline = jedis.pipelined();
pipeline = jedis.pipelined();for (int i = 0; i < rehashThreshold; i++) {
pipeline.setex(String.valueOf(i), expireTime, String.valueOf(i)); if (i % 10000 == 0) {
pipeline.sync();
}
}
pipeline.sync();// 2.等待写增量TimeUnit.SECONDS.sleep(5);for (int i = rehashThreshold; i < rehashThreshold + 200; i++) {
jedis.setex(String.valueOf(i), expireTime, String.valueOf(i));
TimeUnit.SECONDS.sleep(1);
}
(2) 开始测试
(a) 当阈值=215=32768，从下面可以看出到key的个数为32769时，内存涨了一些，但是还不明显。
keys mem clients blocked requests connections32766 4.69M 3 0 32797 (+2) 4 32767 4.69M 3 0 32799 (+2) 4 32768 4.69M 3 0 32801 (+2) 4 32769 5.44M 3 0 32803 (+2) 4
(b) 当阈值=220=1048576，从下面可以看出到key的个数为1048577时，内存涨了32M。因为rehash会扩容，所以新的哈希表中的槽位变为了221 * 2（因为每个key都设置了过期时间，expires表），指针为8个字节，221 ? 2 ? 8 = 225 = 32MB。
keys mem clients blocked requests connections1048574 128.69M 3 0 3364129 (+2) 16 1048575 128.69M 3 0 3364131 (+2) 16 1048576 128.69M 3 0 3364133 (+2) 16 1048577 160.69M 3 0 3364135 (+2) 16 1048578 160.69M 3 0 3364137 (+2) 16
(c) 当阈值=226=67108864，从下面可以看出到key的个数为67108865时，内存涨了2GB。因为rehash会扩容，所以新的哈希表中的槽位变为了227 * 2（因为每个key都设置了过期时间，expires表），指针为8个字节，227 ? 2 ? 8 = 231 = 2GB。
keys mem clients blocked requests connections67108862 9.70G 3 0 70473683 (+2) 18 67108863 9.70G 3 0 70473685 (+2) 18 67108864 9.70G 3 0 70473687 (+2) 18 67108865 11.70G 3 0 70473689 (+2) 18 67108866 11.70G 3 0 70473691 (+2) 18 67108867 11.70G 3 0 70473693 (+2) 18
回过来看r-bp1c15fd9b142d04的key和内存变化图，可以发现上面的规则是正确的：
4. 后续观察
17点时，rehash结束，内存降了增加的2G的一半。
五、总结
由于哈希表的特性，Redis 中键值数量大，不会对存取造成性能影响，但是会出现本文提到的问题。控制键个数有几个建议：无用的键值设置过期时间或者定期删除。优化键值设计：例如可以使用 ziplist hash合并优化部分字符串类型。未来改进：内核层面支持 rehash 的审计日志以及增强 rehash 的速度。
以上所述就是小编给大家介绍的《探寻 Redis 内存诡异增长的元凶》，希望对大家有所帮助，如果大家有任何疑问请给我留言，小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持！
猜你喜欢:
- 程序员是制造 Bug 的“元凶”？
- 菠菜类DAPP已成黑客提款机？元凶竟然是它！
- 探寻流式计算
- 探寻 Swift 中的最佳实践
- Koa 生成器函数探寻
- 探寻Metasploit Payload模式背后的秘密
本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。
Head First HTML and CSS
Elisabeth Robson、Eric Freeman / O'Reilly Media / 2012-9-8 / USD 39.99
Tired of reading HTML books that only make sense after you're an expert? Then it's about time you picked up Head First HTML and really learned HTML. You want to learn HTML so you can finally create th......一起来看看 《Head First HTML and CSS》 这本书的介绍吧!
