内容简介:众所周知,redis支持5种基础数据类型,分别是:每种数据类型都存在至少一种encoding方式。redis把上面几种基础类型抽象成为一个结构体叫做本文就重点介绍下hash类型在redis中是如何存储和使用的。
众所周知,redis支持5种基础数据类型,分别是:
- string
- list
- set
- hset
- hash
每种数据类型都存在至少一种encoding方式。redis把上面几种基础类型抽象成为一个结构体叫做 redisObject
typedef struct redisObject { unsigned type:4; //type就是 redis 的基础数据类型 unsigned encoding:4; //这个是具体数据类型的编码方式 unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ int refcount; void *ptr; } robj; 复制代码
本文就重点介绍下hash类型在redis中是如何存储和使用的。
2. redis hash类型
hash类型是一个可以存储多个k-v键值对的结构,典型的样子是这样的:
其实具体的命令查看redis的官方文档是最方便的,但是我还是把常用的总结下,也给自己加深下影响。
2.1 hash的典型命令
典型的命令格式:
hset redis-obj-name k1 v1 k2 v2 ...
hget redis_obj_name k1
注意这个命令是操作hash对象的, 和hset对象没有关系 ,不要搞混淆了。 例如:
redis> HSET myhash field1 "Hello" (integer) 1 redis> HGET myhash field1 "Hello" redis> 复制代码
看上去很简单,那么这个myhash对象在redis的内存中是如何存储的呢?直接上源码,大家看的比较清楚:
void hsetCommand(client *c) { int i, created = 0; robj *o; //首先参数必须是双数,很好理解 if ((c->argc % 2) == 1) { addReplyError(c,"wrong number of arguments for HMSET"); return; } //函数名称写的很清楚,找不到就创建一个redis-obj对象 if ((o = hashTypeLookupWriteOrCreate(c,c->argv[1])) == NULL) return; hashTypeTryConversion(o,c->argv,2,c->argc-1);//这里是两点,它居然会尝试去转换下hash的type for (i = 2; i < c->argc; i += 2) created += !hashTypeSet(o,c->argv[i]->ptr,c->argv[i+1]->ptr,HASH_SET_COPY); /* HMSET (deprecated) and HSET return value is different. */ char *cmdname = c->argv[0]->ptr; if (cmdname[1] == 's' || cmdname[1] == 'S') { /* HSET */ addReplyLongLong(c, created); } else { /* HMSET */ addReply(c, shared.ok); } signalModifiedKey(c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_HASH,"hset",c->argv[1],c->db->id); server.dirty++; } 复制代码
那我们看看 hashTypeLookupWriteOrCreate
和 hashTypeTryConversion
到底干了啥事。
robj *hashTypeLookupWriteOrCreate(client *c, robj *key) { robj *o = lookupKeyWrite(c->db,key); if (o == NULL) { o = createHashObject(); //这里会去创建一个hash objecjt dbAdd(c->db,key,o); } else { if (o->type != OBJ_HASH) { addReply(c,shared.wrongtypeerr); return NULL; } } return o; } robj *createHashObject(void) { unsigned char *zl = ziplistNew(); robj *o = createObject(OBJ_HASH, zl); o->encoding = OBJ_ENCODING_ZIPLIST; return o; } 复制代码
看上面, createHashObject
函数其实创建的redis-obj的type是hash类型,但是encoding却是OBJ_ENCODING_ZIPLIST,看到这里会有点疑惑,既然是hash类型应该用hash table结构来存储,为什么用压缩链表结构呢?其实不用急,还有一个函数 hashTypeTryConversion
这个函数没有看,现在再看看它的实现:
/* Check the length of a number of objects to see if we need to convert a * ziplist to a real hash. Note that we only check string encoded objects * as their string length can be queried in constant time. */ void hashTypeTryConversion(robj *o, robj **argv, int start, int end) { int i; if (o->encoding != OBJ_ENCODING_ZIPLIST) return; for (i = start; i <= end; i++) { if (sdsEncodedObject(argv[i]) && sdslen(argv[i]->ptr) > server.hash_max_ziplist_value) { hashTypeConvert(o, OBJ_ENCODING_HT); break; } } } 复制代码
其实上面的注释写的很清楚,如果是ZIPLIST的编码方式,遍历下ziplist,如果当前的长度已经大于 server.hash_max_ziplist_value
,就把encoding方式改为 OBJ_ENCODING_HT
。还有一种情况是当
hash-max-ziplist-entries 512 hash-max-ziplist-value 64 复制代码
看到这里貌似有点明白了,原来redis对于小数字,短字符串,为了能比较高效的利用内存,都保存到ziplist中,而不是直接放到hash-table结构中,当数字或者字符串超出一定的阈值时候,才会改用hash表的存储方式,这样达到节约内存的作用啊。在这里不得不感叹下redis的作者真不怕麻烦,为了能节约一点内存,可以说费劲了心思。
总结下,redis对于hash对象提供了两种存储方式,也就是 redisObject.encoding
变量的取值是有两个的,分别如下:
- OBJ_ENCODING_ZIPLIST
- OBJ_ENCODING_HT
这两种编码方式内部的数据结构是什么样子的呢? 首先我们先看看 OBJ_ENCODING_ZIPLIST
类型的存储方式
2.2 OBJ_ENCODING_ZIPLIST存储方式
在 createHashObject
函数中,调用了ziplist的创建函数 ziplistNew
,我们来看下这个函数的实现:
/* Create a new empty ziplist. */ unsigned char *ziplistNew(void) { unsigned int bytes = ZIPLIST_HEADER_SIZE+1; unsigned char *zl = zmalloc(bytes); ZIPLIST_BYTES(zl) = intrev32ifbe(bytes); ZIPLIST_TAIL_OFFSET(zl) = intrev32ifbe(ZIPLIST_HEADER_SIZE); ZIPLIST_LENGTH(zl) = 0; zl[bytes-1] = ZIP_END; return zl; } 复制代码
代码里面用了一堆宏,看上去不太直观,画个图看下,就很清晰了:
再附上ziplist的header的注释:
/* The size of a ziplist header: two 32 bit integers for the total * bytes count and last item offset. One 16 bit integer for the number * of items field. */ 复制代码
结合代码很轻松就应该能看懂了。
再看上面的代码 hsetCommand
中,调用了 hashTypeSet
函数进行插入数据 我们再看看对于 OBJ_ENCODING_ZIPLIST
的编码方式,如何插入数据。
int hashTypeSet(robj *o, sds field, sds value, int flags) { int update = 0; if (o->encoding == OBJ_ENCODING_ZIPLIST) { unsigned char *zl, *fptr, *vptr; zl = o->ptr; fptr = ziplistIndex(zl, ZIPLIST_HEAD); if (fptr != NULL) { fptr = ziplistFind(fptr, (unsigned char*)field, sdslen(field), 1); if (fptr != NULL) { /* Grab pointer to the value (fptr points to the field) */ vptr = ziplistNext(zl, fptr); serverAssert(vptr != NULL); update = 1; /* Delete value */ zl = ziplistDelete(zl, &vptr); /* Insert new value */ zl = ziplistInsert(zl, vptr, (unsigned char*)value, sdslen(value)); } } o->ptr = zl; /* Check if the ziplist needs to be converted to a hash table */ if (hashTypeLength(o) > server.hash_max_ziplist_entries) hashTypeConvert(o, OBJ_ENCODING_HT); ... } 复制代码
首次插入的时候, ziplistIndex(zl, ZIPLIST_HEAD);
函数会返回NULL
unsigned char *ziplistIndex(unsigned char *zl, int index) { unsigned char *p; unsigned int prevlensize, prevlen = 0; if (index < 0) { index = (-index)-1; p = ZIPLIST_ENTRY_TAIL(zl); if (p[0] != ZIP_END) { ZIP_DECODE_PREVLEN(p, prevlensize, prevlen); while (prevlen > 0 && index--) { p -= prevlen; ZIP_DECODE_PREVLEN(p, prevlensize, prevlen); } } } else { p = ZIPLIST_ENTRY_HEAD(zl); while (p[0] != ZIP_END && index--) { p += zipRawEntryLength(p); } } return (p[0] == ZIP_END || index > 0) ? NULL : p; } 复制代码
进而直接调用 ziplistPush
把field和value都插入到ziplist中。再插入过后,还再多了一次判断当前的ziplist的长度是不是大于了 server.hash_max_ziplist_entries
,如果是,就需要转换为hashtable结构存储。
unsigned char *ziplistPush(unsigned char *zl, unsigned char *s, unsigned int slen, int where) { unsigned char *p; p = (where == ZIPLIST_HEAD) ? ZIPLIST_ENTRY_HEAD(zl) : ZIPLIST_ENTRY_END(zl); return __ziplistInsert(zl,p,s,slen); } unsigned char *__ziplistInsert(unsigned char *zl, unsigned char *p, unsigned char *s, unsigned int slen) { size_t curlen = intrev32ifbe(ZIPLIST_BYTES(zl)), reqlen; unsigned int prevlensize, prevlen = 0; size_t offset; int nextdiff = 0; unsigned char encoding = 0; long long value = 123456789; /* initialized to avoid warning. Using a value that is easy to see if for some reason we use it uninitialized. */ zlentry tail; /* Find out prevlen for the entry that is inserted. */ if (p[0] != ZIP_END) { ZIP_DECODE_PREVLEN(p, prevlensize, prevlen); } else { unsigned char *ptail = ZIPLIST_ENTRY_TAIL(zl); if (ptail[0] != ZIP_END) { prevlen = zipRawEntryLength(ptail); } } /* See if the entry can be encoded */ if (zipTryEncoding(s,slen,&value,&encoding)) { /* 'encoding' is set to the appropriate integer encoding */ reqlen = zipIntSize(encoding); } else { /* 'encoding' is untouched, however zipStoreEntryEncoding will use the * string length to figure out how to encode it. */ reqlen = slen; } /* We need space for both the length of the previous entry and * the length of the payload. */ reqlen += zipStorePrevEntryLength(NULL,prevlen); reqlen += zipStoreEntryEncoding(NULL,encoding,slen); /* When the insert position is not equal to the tail, we need to * make sure that the next entry can hold this entry's length in * its prevlen field. */ int forcelarge = 0; nextdiff = (p[0] != ZIP_END) ? zipPrevLenByteDiff(p,reqlen) : 0; if (nextdiff == -4 && reqlen < 4) { nextdiff = 0; forcelarge = 1; } /* Store offset because a realloc may change the address of zl. */ offset = p-zl; zl = ziplistResize(zl,curlen+reqlen+nextdiff); p = zl+offset; /* Apply memory move when necessary and update tail offset. */ if (p[0] != ZIP_END) { /* Subtract one because of the ZIP_END bytes */ memmove(p+reqlen,p-nextdiff,curlen-offset-1+nextdiff); /* Encode this entry's raw length in the next entry. */ if (forcelarge) zipStorePrevEntryLengthLarge(p+reqlen,reqlen); else zipStorePrevEntryLength(p+reqlen,reqlen); /* Update offset for tail */ ZIPLIST_TAIL_OFFSET(zl) = intrev32ifbe(intrev32ifbe(ZIPLIST_TAIL_OFFSET(zl))+reqlen); /* When the tail contains more than one entry, we need to take * "nextdiff" in account as well. Otherwise, a change in the * size of prevlen doesn't have an effect on the *tail* offset. */ zipEntry(p+reqlen, &tail); if (p[reqlen+tail.headersize+tail.len] != ZIP_END) { ZIPLIST_TAIL_OFFSET(zl) = intrev32ifbe(intrev32ifbe(ZIPLIST_TAIL_OFFSET(zl))+nextdiff); } } else { /* This element will be the new tail. */ ZIPLIST_TAIL_OFFSET(zl) = intrev32ifbe(p-zl); } /* When nextdiff != 0, the raw length of the next entry has changed, so * we need to cascade the update throughout the ziplist */ if (nextdiff != 0) { offset = p-zl; zl = __ziplistCascadeUpdate(zl,p+reqlen); p = zl+offset; } /* Write the entry */ p += zipStorePrevEntryLength(p,prevlen); p += zipStoreEntryEncoding(p,encoding,slen); if (ZIP_IS_STR(encoding)) { memcpy(p,s,slen); } else { zipSaveInteger(p,value,encoding); } ZIPLIST_INCR_LENGTH(zl,1); return zl; } 复制代码
插入的时候可以看出来,redis对于ziplist的存储数据结构也是比较特殊的。一个item项的结构如下:
p += zipStorePrevEntryLength(p,prevlen); //计算上一个item项的长度 p += zipStoreEntryEncoding(p,encoding,slen); //计算当前自己需要的编码 复制代码
其中 prev_entry_length
存储的是上一个item项的长度,这个也是redis比较特殊的地方,在本次更新item的时候采取计算上一个item项的长度。
encoding是当前这一项的编码方式。ziplist既然是压缩链表,本质上只是是对数字类型的压缩,字符串数字都统一转换为int8, int16, int32, int64 来存储,这样比较节约内存。
具体的代码实现如下:
/* See if the entry can be encoded */ if (zipTryEncoding(s,slen,&value,&encoding)) { /* 'encoding' is set to the appropriate integer encoding */ reqlen = zipIntSize(encoding); } else { /* 'encoding' is untouched, however zipStoreEntryEncoding will use the * string length to figure out how to encode it. */ reqlen = slen; } 复制代码
具体的 zipTryEncoding
代码实现:
/* Check if string pointed to by 'entry' can be encoded as an integer. * Stores the integer value in 'v' and its encoding in 'encoding'. */ int zipTryEncoding(unsigned char *entry, unsigned int entrylen, long long *v, unsigned char *encoding) { long long value; if (entrylen >= 32 || entrylen == 0) return 0; if (string2ll((char*)entry,entrylen,&value)) { /* Great, the string can be encoded. Check what's the smallest * of our encoding types that can hold this value. */ if (value >= 0 && value <= 12) { *encoding = ZIP_INT_IMM_MIN+value; } else if (value >= INT8_MIN && value <= INT8_MAX) { *encoding = ZIP_INT_8B; } else if (value >= INT16_MIN && value <= INT16_MAX) { *encoding = ZIP_INT_16B; } else if (value >= INT24_MIN && value <= INT24_MAX) { *encoding = ZIP_INT_24B; } else if (value >= INT32_MIN && value <= INT32_MAX) { *encoding = ZIP_INT_32B; } else { *encoding = ZIP_INT_64B; } *v = value; return 1; } return 0; } 复制代码
其中string2ll其实就是一个atoi,但是要实现一个没bug的atoi还是很难的,看看redis的实现,觉得考虑的好全面,负数,越界都考虑清楚,感觉还是很难的。
/* Convert a string into a long long. Returns 1 if the string could be parsed * into a (non-overflowing) long long, 0 otherwise. The value will be set to * the parsed value when appropriate. * * Note that this function demands that the string strictly represents * a long long: no spaces or other characters before or after the string * representing the number are accepted, nor zeroes at the start if not * for the string "0" representing the zero number. * * Because of its strictness, it is safe to use this function to check if * you can convert a string into a long long, and obtain back the string * from the number without any loss in the string representation. */ int string2ll(const char *s, size_t slen, long long *value) { const char *p = s; size_t plen = 0; int negative = 0; unsigned long long v; /* A zero length string is not a valid number. */ if (plen == slen) return 0; /* Special case: first and only digit is 0. */ if (slen == 1 && p[0] == '0') { if (value != NULL) *value = 0; return 1; } /* Handle negative numbers: just set a flag and continue like if it * was a positive number. Later convert into negative. */ if (p[0] == '-') { negative = 1; p++; plen++; /* Abort on only a negative sign. */ if (plen == slen) return 0; } /* First digit should be 1-9, otherwise the string should just be 0. */ if (p[0] >= '1' && p[0] <= '9') { v = p[0]-'0'; p++; plen++; } else { return 0; } /* Parse all the other digits, checking for overflow at every step. */ while (plen < slen && p[0] >= '0' && p[0] <= '9') { if (v > (ULLONG_MAX / 10)) /* Overflow. */ return 0; v *= 10; if (v > (ULLONG_MAX - (p[0]-'0'))) /* Overflow. */ return 0; v += p[0]-'0'; p++; plen++; } /* Return if not all bytes were used. */ if (plen < slen) return 0; /* Convert to negative if needed, and do the final overflow check when * converting from unsigned long long to long long. */ if (negative) { if (v > ((unsigned long long)(-(LLONG_MIN+1))+1)) /* Overflow. */ return 0; if (value != NULL) *value = -v; } else { if (v > LLONG_MAX) /* Overflow. */ return 0; if (value != NULL) *value = v; } return 1; } 复制代码
其实每次更新,都会触发内存的realloc,这个地方我感觉其实还是不太好的,如果一次更新n个kv对,就需要调用realloc函数n次,感觉有点浪费啊。
2.2 OBJ_ENCODING_HT存储方式
从上面的代码可以看出来有两种场景会触发hash obj修改encoding方式,分别如下:
hash-max-ziplist-entries 512 hash-max-ziplist-value 64 复制代码
当ziplist的entry个数小于512的时候, 还有一种场景是entry的值长度小于64的时候。当然这其实是redis的一个配置项。
那么hash table存储又是什么样的结构呢?看下面的代码:
void hashTypeConvertZiplist(robj *o, int enc) { serverAssert(o->encoding == OBJ_ENCODING_ZIPLIST); if (enc == OBJ_ENCODING_ZIPLIST) { /* Nothing to do... */ } else if (enc == OBJ_ENCODING_HT) { hashTypeIterator *hi; dict *dict; int ret; hi = hashTypeInitIterator(o); dict = dictCreate(&hashDictType, NULL); while (hashTypeNext(hi) != C_ERR) { sds key, value; key = hashTypeCurrentObjectNewSds(hi,OBJ_HASH_KEY); value = hashTypeCurrentObjectNewSds(hi,OBJ_HASH_VALUE); ret = dictAdd(dict, key, value); if (ret != DICT_OK) { serverLogHexDump(LL_WARNING,"ziplist with dup elements dump", o->ptr,ziplistBlobLen(o->ptr)); serverPanic("Ziplist corruption detected"); } } hashTypeReleaseIterator(hi); zfree(o->ptr); o->encoding = OBJ_ENCODING_HT; o->ptr = dict; } else { serverPanic("Unknown hash encoding"); } } 复制代码
可以看出会创建一个迭代器,遍历当前的ziplist结构,然后放到新创建的dict结构中。
关于dict的结构,可以参看之前我的一篇dict的数据结构分析。
3. 总结
hash对象的存储如果使用的编码是ZipList的时候,感觉效率是不高的,平均复杂度是 O(n)
,如果涉及到内存的连锁移动的话,最差的事件复杂度其实是 o(n^2)
。
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网
猜你喜欢:- 《JavaScript面向对象精要》之三:理解对象
- 面向对象的程序设计之理解对象
- 深入理解Objective-C中实例、类对象、元类对象之间的关系
- Object 对象你真理解了吗?
- 深入理解Python面向对象的三大特性
- 深入理解多线程(三)—— Java的对象头
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Python带我起飞
李金洪 / 电子工业出版社 / 2018-6 / 79
《Python带我起飞——入门、进阶、商业实战》针对Python 3.5 以上版本,采用“理论+实践”的形式编写,通过大量的实例(共42 个),全面而深入地讲解“Python 基础语法”和“Python 项目应用”两方面内容。书中的实例具有很强的实用性,如对医疗影像数据进行分析、制作爬虫获取股票信息、自动化实例、从一组看似混乱的数据中找出规律、制作人脸识别系统等。 《Python带我起飞——......一起来看看 《Python带我起飞》 这本书的介绍吧!