内容简介：Elasticsearch的相关度评分（relevance score）算法采用的是term frequency/inverse document frequency算法，简称为TF/IDF算法。例如：搜索请求：hello world
Elasticsearch的相关度评分（relevance score）算法采用的是term frequency/inverse document frequency算法，简称为TF/IDF算法。
算法介绍
relevance score算法，简单来说就是，就是计算出一个索引中的文本，与搜索文本，它们之间的关联匹配程度。
TF/IDF算法，分为两个部分，IF 和IDF
Term Frequency(TF): 搜索文本中的各个词条在field文本中出现了多少次，出现的次数越多，就越相关
例如：
搜索请求：hello world
doc1: hello you, and world is very good
doc2: hello, how are you
那么此时根据TF算法，doc1的相关度要比doc2的要高
Inverse Document Frequency(IDF): 搜索文本中的各个词条在整个索引的所有文档中出现的次数，出现的次数越多，就越不相关。
搜索请求： hello world
doc1: hello, today is very good.
doc2: hi world, how are you.
比如在index中有1万条document, hello这个单词在所有的document中，一共出现了1000次，world这个单词在所有的document中一共出现100次。那么根据IDF算法此时doc2的相关度要比doc1要高。
对于ES还有一个Field-length norm
field-length norm就是field长度越长，相关度就越弱
搜索请求：hello world
doc1: {"title": "hello article", "content": "1万个单词"}
doc2: {"title": "my article", "content": "1万个单词， hi world"}
此时hello world在整个index中出现的次数是一样多的。但是根据Field-length norm此时doc1比doc2相关度要高。因为title字段更短。
_score是如何被计算出来的
GET /test_index/_search?explain=true
{
"query": {
"match": {
"test_field": "hello"
}
}
}
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 0.20521778,
"hits" : [
{
"_shard" : "[test_index][0]",
"_node" : "P-b-TEvyQOylMyEcMEhApQ",
"_index" : "test_index",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.20521778,
"_source" : {
"test_field" : "hello, how are you"
},
"_explanation" : {
"value" : 0.20521778,
"description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
"details" : [
{
"value" : 0.20521778,
"description" : "score(freq=1.0), product of:",
"details" : [
{
"value" : 2.2,
"description" : "boost",
"details" : [ ]
},
{
"value" : 0.18232156,
"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details" : [
{
"value" : 2,
"description" : "n, number of documents containing term",
"details" : [ ]
},
{
"value" : 2,
"description" : "N, total number of documents with field",
"details" : [ ]
}
]
},
{
"value" : 0.5116279,
"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details" : [
{
"value" : 1.0,
"description" : "freq, occurrences of term within document",
"details" : [ ]
},
{
"value" : 1.2,
"description" : "k1, term saturation parameter",
"details" : [ ]
},
{
"value" : 0.75,
"description" : "b, length normalization parameter",
"details" : [ ]
},
{
"value" : 4.0,
"description" : "dl, length of field",
"details" : [ ]
},
{
"value" : 5.5,
"description" : "avgdl, average length of field",
"details" : [ ]
}
]
}
]
}
]
}
},
{
"_shard" : "[test_index][0]",
"_node" : "P-b-TEvyQOylMyEcMEhApQ",
"_index" : "test_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 0.16402164,
"_source" : {
"test_field" : "hello you, and world is very good"
},
"_explanation" : {
"value" : 0.16402164,
"description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
"details" : [
{
"value" : 0.16402164,
"description" : "score(freq=1.0), product of:",
"details" : [
{
"value" : 2.2,
"description" : "boost",
"details" : [ ]
},
{
"value" : 0.18232156,
"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details" : [
{
"value" : 2,
"description" : "n, number of documents containing term",
"details" : [ ]
},
{
"value" : 2,
"description" : "N, total number of documents with field",
"details" : [ ]
}
]
},
{
"value" : 0.40892193,
"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details" : [
{
"value" : 1.0,
"description" : "freq, occurrences of term within document",
"details" : [ ]
},
{
"value" : 1.2,
"description" : "k1, term saturation parameter",
"details" : [ ]
},
{
"value" : 0.75,
"description" : "b, length normalization parameter",
"details" : [ ]
},
{
"value" : 7.0,
"description" : "dl, length of field",
"details" : [ ]
},
{
"value" : 5.5,
"description" : "avgdl, average length of field",
"details" : [ ]
}
]
}
]
}
]
}
}
]
}
}
匹配的文档有两个，下面直接用一个文档来分析出ES各个算法的公式。
从上面可以看出第一个文档的相关度分数是0.20521778
{
"_shard" : "[test_index][0]",
"_node" : "P-b-TEvyQOylMyEcMEhApQ",
"_index" : "test_index",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.20521778,
"_source" : {
"test_field" : "hello, how are you"
},
"_explanation" : {
"value" : 0.20521778,
"description" : "weight(test_field:hello in 0) [PerFieldSimilarity], result of:",
"details" : [
{
"value" : 0.20521778,
"description" : "score(freq=1.0), product of:",
"details" : [
{
"value" : 2.2,
"description" : "boost",
"details" : [ ]
},
{
"value" : 0.18232156,
"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details" : [
{
"value" : 2,
"description" : "n, number of documents containing term",
"details" : [ ]
},
{
"value" : 2,
"description" : "N, total number of documents with field",
"details" : [ ]
}
]
},
{
"value" : 0.5116279,
"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details" : [
{
"value" : 1.0,
"description" : "freq, occurrences of term within document",
"details" : [ ]
},
{
"value" : 1.2,
"description" : "k1, term saturation parameter",
"details" : [ ]
},
{
"value" : 0.75,
"description" : "b, length normalization parameter",
"details" : [ ]
},
{
"value" : 4.0,
"description" : "dl, length of field",
"details" : [ ]
},
{
"value" : 5.5,
"description" : "avgdl, average length of field",
"details" : [ ]
}
]
}
]
}
]
}
}
通过观察我们可以知道 _score = boost * idf * tf 此时boost = 2.2, idf = 0.18232156, tf = 0.5116279 idf = log(1 + (N - n + 0.5) / (n + 0.5)) 此时n = 2 (n, number of documents containing term), N = 2(N, total number of documents with field) tf = freq / (freq + k1 * (1 - b + b * dl / avgdl)) 此时freq = 1(freq, occurrences of term within document), k1 = 1.2(k1, term saturation parameter), b = 0.75(b, length normalization parameter), d1 = 4 (dl, length of field), avgdl = 5.5(avgdl, average length of field)
以上所述就是小编给大家介绍的《elasticsearch学习笔记（三十）——Elasticsearch 相关度评分 TF&IDF算法》，希望对大家有所帮助，如果大家有任何疑问请给我留言，小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持！
猜你喜欢:
- ElasticSearch 揭秘lucene的相关度分数算法
- 回归中的相关度和决定系数
- Lucene教程--维护索引、查询对象和相关度排序
- elasticsearch学习笔记高级篇（九）——多shard场景下相关度分数不准确问题
- 通俗易懂--决策树算法、随机森林算法讲解(算法+案例)
- 限流算法之漏桶算法、令牌桶算法
本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。
统计自然语言处理
宗成庆 / 清华大学出版社 / 2008-5 / 66.00元
内容简介 本书全面介绍了统计自然语言处理的基本概念、理论方法和最新研究进展，内容包括形式语言与自动机及其在自然语言处理中的应用、语言模型、隐马尔可夫模型、语料库技术、汉语自动分词与词性标注、句法分析、词义消歧、统计机器翻译、语音翻译、文本分类、信息检索与问答系统、自动文摘和信息抽取、口语信息处理与人机对话系统等，既有对基础知识和理论模型的介绍，也有对相关问题的研究背景、实现方法和技术现状的详......一起来看看 《统计自然语言处理》 这本书的介绍吧!
随机密码生成器
多种字符组合密码
Markdown 在线编辑器
Markdown 在线编辑器