elasticsearch-scroll

栏目: 后端 · 发布时间: 5年前

内容简介：碰到一个比较头疼的问题，MySQL数据丢失。有两个办法，一个办法是让DBA找半年前的数据。另一个办法是保存了MySQL数据的ES里找。由于数据量过万，而且ES设置了一次查询数据量最大10000，想想用 scroll 取数据会比较好。

碰到一个比较头疼的问题，MySQL数据丢失。

有两个办法，一个办法是让DBA找半年前的数据。另一个办法是保存了 MySQL 数据的ES里找。

由于数据量过万，而且ES设置了一次查询数据量最大10000，想想用 scroll 取数据会比较好。

1 ElasticSearch 2.x

1.1 查询索引有多少数据

localhost:9200/_nodes/stats/indices/search?pretty

1.1 查看索引信息

curl -XGET 'http://127.0.0.1:9400/dev_index1_20190118/docs/_search?pretty'

1.2 使用游标

curl -XGET 'http://127.0.0.1:9400/dev_index1_20190118/docs/_search?scroll=10m' -d ' 
{ 
    "query": { "match_all": {}},
    "sort" : ["_doc"], 
    "size":  10000
}'  >> es_scroll_data_20190118_1w.txt

1.3 不断取下一页

curl -XGET 'http://127.0.0.1:9400/_search?scroll=10m' -d ' 
{ 
    "scroll": "10m",
    "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAANKLTFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADSi1BY3X1Z6N2NoRlNQaTlGLTFueDk1d0xBAAAAAAA0otYWN19WejdjaEZTUGk5Ri0xbng5NXdMQQAAAAAANKLVFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADvJpxZzcU9YSExnLVRTNk5RY3JfMlNuWU9n"
}' >> es_scroll_data_20190118_2w.txt

curl -XGET 'http://127.0.0.1:9400/_search?scroll=10m' -d ' 
{ 
    "scroll": "10m",
    "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAANKLTFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADSi1BY3X1Z6N2NoRlNQaTlGLTFueDk1d0xBAAAAAAA0otYWN19WejdjaEZTUGk5Ri0xbng5NXdMQQAAAAAANKLVFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADvJpxZzcU9YSExnLVRTNk5RY3JfMlNuWU9n"
}' >> es_scroll_data_20190118_3w.txt

2 ElasticSearch 5.6.x

2.1 查询索引信息

localhost:9200/_nodes/stats/indices/search?pretty

curl -XGET 'http://127.0.0.1:9400/dev_index1_20190118/docs/_search?pretty'

2.2 使用游标

curl -XGET 'http://127.0.0.1:9400/dev_index1_20190118/docs/_search?scroll=10m' -d ' 
{ 
    "query": { "match_all": {}},
    "sort" : ["_doc"], 
    "size":  10000
}'  >> es_scroll_data_20190118_1w.txt

2.3 不断取下一页

curl -XGET 'http://127.0.0.1:9400/_search?scroll=10m' -d ' 
{ 
    "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAANKLTFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADSi1BY3X1Z6N2NoRlNQaTlGLTFueDk1d0xBAAAAAAA0otYWN19WejdjaEZTUGk5Ri0xbng5NXdMQQAAAAAANKLVFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADvJpxZzcU9YSExnLVRTNk5RY3JfMlNuWU9n"
}' >> es_scroll_data_20190118_2w.txt

curl -XGET 'http://127.0.0.1:9400/_search?scroll=10m' -d ' 
{ 
    "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAANKLTFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADSi1BY3X1Z6N2NoRlNQaTlGLTFueDk1d0xBAAAAAAA0otYWN19WejdjaEZTUGk5Ri0xbng5NXdMQQAAAAAANKLVFjdfVno3Y2hGU1BpOUYtMW54OTV3TEEAAAAAADvJpxZzcU9YSExnLVRTNk5RY3JfMlNuWU9n"
}' >> es_scroll_data_20190118_3w.txt
```  




# 遇到的问题

## 3.1 Unknown key for a VALUE_STRING in [scroll_id].

```json
{
    "error": {
        "root_cause": [
            {
                "type": "parsing_exception",
                "reason": "Unknown key for a VALUE_STRING in [scroll_id].",
                "line": 3,
                "col": 19
            }
        ],
        "type": "parsing_exception",
        "reason": "Unknown key for a VALUE_STRING in [scroll_id].",
        "line": 3,
        "col": 19
    },
    "status": 400
}

第二次使用的 scroll_id 和第一次返回的 scroll_id 不一致导致

3.2 Unknown key for a VALUE_STRING in [scroll]

{
    "error": {
        "root_cause": [
            {
                "type": "parsing_exception",
                "reason": "Unknown key for a VALUE_STRING in [scroll].",
                "line": 3,
                "col": 15
            }
        ],
        "type": "parsing_exception",
        "reason": "Unknown key for a VALUE_STRING in [scroll].",
        "line": 3,
        "col": 15
    },
    "status": 400
}

第二次请求时请求参数里多了 scroll 参数

3.3 Batch size is too large, size must be less than or equal to: [10000] but was [1000000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting.

{
    "error": {
        "root_cause": [
            {
                "type": "query_phase_execution_exception",
                "reason": "Batch size is too large, size must be less than or equal to: [10000] but was [1000000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "dev_index1_20190118",
                "node": "8XqKY198S823M78QA43F8g",
                "reason": {
                    "type": "query_phase_execution_exception",
                    "reason": "Batch size is too large, size must be less than or equal to: [10000] but was [1000000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."
                }
            }
        ]
    },
    "status": 500
}

设置的 size 过大，超过10000，配置文件里 index.max_result_window 最大为10000

3.4 search_context_missing_exception

{
    "error": {
        "root_cause": [
            {
                "type": "search_context_missing_exception",
                "reason": "No search context found for id [3540965]"
            },
            {
                "type": "search_context_missing_exception",
                "reason": "No search context found for id [3922089]"
            },
            {
                "type": "search_context_missing_exception",
                "reason": "No search context found for id [3454995]"
            },
            {
                "type": "search_context_missing_exception",
                "reason": "No search context found for id [3454996]"
            },
            {
                "type": "search_context_missing_exception",
                "reason": "No search context found for id [3454994]"
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": -1,
                "index": null,
                "reason": {
                    "type": "search_context_missing_exception",
                    "reason": "No search context found for id [3540965]"
                }
            },
            {
                "shard": -1,
                "index": null,
                "reason": {
                    "type": "search_context_missing_exception",
                    "reason": "No search context found for id [3922089]"
                }
            },
            {
                "shard": -1,
                "index": null,
                "reason": {
                    "type": "search_context_missing_exception",
                    "reason": "No search context found for id [3454995]"
                }
            },
            {
                "shard": -1,
                "index": null,
                "reason": {
                    "type": "search_context_missing_exception",
                    "reason": "No search context found for id [3454996]"
                }
            },
            {
                "shard": -1,
                "index": null,
                "reason": {
                    "type": "search_context_missing_exception",
                    "reason": "No search context found for id [3454994]"
                }
            }
        ],
        "caused_by": {
            "type": "search_context_missing_exception",
            "reason": "No search context found for id [3454994]"
        }
    },
    "status": 404
}

其实是超时了，scroll自动删除了

References

[1] 游标查询

[2] scroll

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持码农网

查看所有标签

猜你喜欢:

elasticsearch-scroll

本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。

码农书籍

ACM国际大学生程序设计竞赛题解

赵端阳//袁鹤 / 电子工业 / 2010-6 / 38.00元

《ACM国际大学生程序设计竞赛题解(1)》可以作为高等院校有关专业的本科和大专学生参加国际大学生程序设计竞赛的辅导教材，或者作为高等院校数据结构、C/C++程序设计或算法设计与分析等相关课程的教学参考书。随着各大专院校参加ACM/ICPC热情的高涨，迫切需要有关介绍ACM国际大学生程序设计竞赛题解的书籍。《ACM国际大学生程序设计竞赛题解(1)》根据浙江大学在线题库的前80题，进行了解答（个别特别......一起来看看《ACM国际大学生程序设计竞赛题解》这本书的介绍吧!

码农工具