ES系列之Elasticsearch-2.3安装笔记

内容简介：ES系列之Elasticsearch-2.3安装笔记

一、安装规划

背景:

某个项目需要三台物理机搭建一个ES集群。

机器大概为24核,128G内存,考虑到ES是纯 java 项目,存在32G内存限制的问题，所以堆内内存一般分配小于等于32G,非堆内内存一般是和堆内内存size一致。

假如我们一个实例堆内内存分配32G,非堆内内存32G,也就是说一个实例会使用64G内存,单机安装两个实例,需要128G内存，这样系统没有可用内存了。

考虑到系统也需要使用一些内存，我们将给每个实例规划25G堆内内存。

实例名称        实例IP                实例端口       软件路径                                                            数据文件路径
instance1    10.16.66.32            9200            /home/bigdata/elasticsearch-2.3.3                    /home/bigdata/es_data
instance2    10.16.66.32            9201            /home/bigdata/elasticsearch-2.3.3_9201         /home/bigdata/es_data_9201
instance3    10.16.66.33            9200            /home/bigdata/elasticsearch-2.3.3                    /home/bigdata/es_data
instance4    10.16.66.33            9201            /home/bigdata/elasticsearch-2.3.3_9201         /home/bigdata/es_data_9201
instance5    10.16.66.34            9200            /home/bigdata/elasticsearch-2.3.3                    /home/bigdata/es_data
instance6    10.16.66.35            9201            /home/bigdata/elasticsearch-2.3.3_9201         /home/bigdata/es_data_9201

二、系统环境设置

1、硬件需求

ES是一个CPU,内存,磁盘使用比较均衡的软件，所以这三个资源不能有明显的瓶颈,不然会影响整体性能。

ES是一个分布式系统,系统自身可以很好的保证数据的不丢失和一致性,所以不用在单台机器上做磁盘RAID,反而会降低磁盘空间利用率和IO性能。

2、内核优化

ES是一个常驻内存的服务，所以需要提前分配及锁定内存,避免内存交换,影响性能。另外就是常见的文件描述符,进程数,tcp连接快速释放和循环利用等参数。

/etc/sysctl.conf 文件如下

kernel.panic = 10
kernel.panic_on_oops = 10
kernel.sysrq = 0
kernel.ctrl-alt-del = 1
kernel.core_pattern = /dev/null
kernel.shmmax = 50696028160
kernel.shmall = 12376960
kernel.msgmnb = 65536
kernel.msgmax = 65536
fs.file-max = 819200
vm.swappiness = 1
vm.dirty_ratio = 60
vm.dirty_background_ratio = 5
vm.min_free_kbytes = 4950780
vm.overcommit_memory = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retries2 = 8
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.ip_local_port_range = 4096 65535
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 65536
net.core.somaxconn = 32768
nee.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_rmem = 4096 131072 16777216
net.core.rmem_default = 131072
net.core.rmem_max = 16777216
net.ipv4.tcp_wmem = 4096 131072 16777216
net.core.wmem_default = 131072
net.core.wmem_max = 16777216
net.core.optmem_max = 65536
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 20000
net.ipv4.udp_rmem_min = 16384
net.ipv4.udp_wmem_min = 16384
net.ipv4.route.flush = 1
net.ipv4.rt_cache_rebuild_count = -1
net.netfilter.nf_conntrack_max = 10000000
net.netfilter.nf_conntrack_tcp_timeout_established = 360
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 13
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

/etc/security/limits.conf文件如下

* soft nofile 1048576
* hard nofile 1048576
* soft nproc 1048576
* hard nproc 1048576
* soft memlock unlimited
* hard memlock unlimited

3、JDK安装

使用JDK8

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

下载64位二进制版本。jdk-8u131-linux-x64.tar.gz

#tar -zxvf jdk-8u131-linux-x64.tar.gz

#mkdir -p /home/jm1/jdk

#mv jdk1.8.0_131/ /home/jm1/jdk/

# cat /etc/profile.d/jdk.sh

#!/bin/bash

export JAVA_HOME=/home/jm1/jdk/jdk1.8.0_131

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$HOME/bin:$PATH

# chmod 755 /etc/profile.d/jdk.sh

由于我们公司初始化系统对profile做了大量改动,所以需要在/etc/bashrc在导入一次。

在/etc/bashrc最后加入一行

source /etc/profile.d/jdk.sh

然后退出重新登陆,执行

#java -version

java version “1.8.0_131″

Java(TM) SE Runtime Environment (build 1.8.0_131-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

4、cgorup安装配置

稍后补充

三、下载软件

https://www.elastic.co/downloads/elasticsearch

将软件分发到每台机器的/tmp/目录

四、安装ES

useradd bigdata

mkdir /home/bigdata/es_data/

mkdir /home/bigdata/es_logs/

mkdir /home/bigdata/es_data_9201/

mkdir /home/bigdata/es_logs_9201/

cp -r elasticsearch-2.3.3/ /home/bigdata/elasticsearch-2.3.3

cp -r elasticsearch-2.3.3/ /home/bigdata/elasticsearch-2.3.3_9201/

修改配置文件:

配置文件示例一:

╰─># cat elasticsearch.yml

cluster.name: frist_es

processors: 40

node.name: “frist_es(10.16.66.32)"
node.master: true
node.data: true
node.max_local_storage_nodes: 1

bootstrap.mlockall: true

threadpool.bulk.type: fixed
threadpool.bulk.size: 40
threadpool.bulk.queue_size: 2000

threadpool.index.type: fixed
threadpool.index.size: 40
threadpool.index.queue_size: 2000

threadpool.search.type: fixed
threadpool.search.size: 120
threadpool.search.queue_size: 4000

index.refresh_interval: 10s
index.merge.scheduler.max_thread_count: 1

#path.shared_data:  /home/jm/es/
path.data: /home/bigdata/es_data/
#path.data: /home/jm8/es_data
path.logs: /home/bigdata/es_logs/

network.host: 10.16.66.32
network.publish_host: 10.16.66.32
transport.tcp.port: 9300
http.port: 9200

http.enabled: true
http.cors.allow-origin: "/.*/"
http.cors.enabled: true

index.number_of_shards: 6
index.number_of_replicas: 1

#gateway.type: local
#gateway.recover_after_nodes: 5
#gateway.expected_nodes: 6

cluster.routing.allocation.node_initial_primaries_recoveries: 8
cluster.routing.allocation.node_concurrent_recoveries: 4

indices.recovery.max_bytes_per_sec: 80mb
indices.recovery.concurrent_streams: 5
indices.fielddata.cache.size: 30%
indices.ttl.interval: 43200s
indices.ttl.bulk_size: 100000

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 60s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.16.66.32:9300","10.16.66.32:9301","10.16.66.33:9300","10.16.66.33:9301","10.16.66.34:9300","10.16.66.34:9301"]

index.translog.flush_threshold_size: 512m
index.translog.flush_threshold_ops: 100000
index.translog.sync_interval: 10s

index.search.slowlog.level: INFO
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.level: INFO
index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms

action.destructive_requires_name: true

script.groovy.sandbox.enabled: true
script.inline: on
script.indexed: on
script.search: on
script.engine.groovy.inline.aggs: on

配置文件示例二:

cluster.name: frist_es

processors: 40

node.name: “frist_es(10.16.66.32_9201)"
node.master: false
node.data: true
node.max_local_storage_nodes: 1

bootstrap.mlockall: true

threadpool.bulk.type: fixed
threadpool.bulk.size: 40
threadpool.bulk.queue_size: 2000

threadpool.index.type: fixed
threadpool.index.size: 40
threadpool.index.queue_size: 2000

threadpool.search.type: fixed
threadpool.search.size: 120
threadpool.search.queue_size: 4000

index.refresh_interval: 10s
index.merge.scheduler.max_thread_count: 1

#path.shared_data:  /home/jm/es/
path.data: /home/bigdata/es_data_9201/
#path.data: /home/jm8/es_data
path.logs: /home/bigdata/es_logs_9201/

network.host: 10.16.66.32
network.publish_host: 10.16.66.32
transport.tcp.port: 9301
http.port: 9201

http.enabled: true
http.cors.allow-origin: "/.*/"
http.cors.enabled: true

index.number_of_shards: 6
index.number_of_replicas: 1

#gateway.type: local
#gateway.recover_after_nodes: 5
#gateway.expected_nodes: 6

cluster.routing.allocation.node_initial_primaries_recoveries: 8
cluster.routing.allocation.node_concurrent_recoveries: 4

indices.recovery.max_bytes_per_sec: 80mb
indices.recovery.concurrent_streams: 5
indices.fielddata.cache.size: 30%
indices.ttl.interval: 43200s
indices.ttl.bulk_size: 100000

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 60s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.16.66.32:9300","10.16.66.32:9301","10.16.66.33:9300","10.16.66.33:9301","10.16.66.34:9300","10.16.66.34:9301"]

index.translog.flush_threshold_size: 512m
index.translog.flush_threshold_ops: 100000
index.translog.sync_interval: 10s

index.search.slowlog.level: INFO
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.level: INFO
index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms

action.destructive_requires_name: true

script.groovy.sandbox.enabled: true
script.inline: on
script.indexed: on
script.search: on
script.engine.groovy.inline.aggs: on

++++++++++++++++++++++++++++++++++

JVM优化

设置JVM内存SIZE

15 if [ "x$ES_MIN_MEM" = "x" ]; then

16 ES_MIN_MEM=25g

17 fi

18 if [ "x$ES_MAX_MEM" = "x" ]; then

19 ES_MAX_MEM=25g

20 fi

21 if [ "x$ES_HEAP_SIZE" != "x" ]; then

22 ES_MIN_MEM=$ES_HEAP_SIZE

23 ES_MAX_MEM=$ES_HEAP_SIZE

24 fi

设置额外的JVM参数

88 JAVA_OPTS=”$JAVA_OPTS -Xss256k -XX:NewRatio=4 -XX:SurvivorRatio=4 -Djna.nosys=true”

五、启动实例

chow -R bigdata:bigdata /home/bigdata

su – bigdata

/home/bigdata/elasticsearch-2.3.3/bin/elasticsearch -d

/home/bigdata/elasticsearch-2.3.3_9201/bin/elasticsearch -d

六、安装插件

su – bigdata

cd elasticsearch-2.3.3

在线安装插件

bin/plugin install hlstudio/bigdesk

bin/plugin install lmenezes/elasticsearch-kopf

bin/plugin install mobz/elasticsearch-head

离线安装插件

./bin/plugin -u file:///tmp/elasticsearch-sql-1.4.6.zip –install sql

注意,在不同的es版本里面,安装插件的命令有所差异。

七、基本使用

在浏览器访问

10.16.66.32:9200/_plugin/kopf //适合运维方面的插件

10.16.66.32:9200/_plugin/head //适合开发管理查询数据的插件

10.16.66.32:9200/_plugin/bigdesk //适合监控集群性能的插件

以上所述就是小编给大家介绍的《ES系列之Elasticsearch-2.3安装笔记》，希望对大家有所帮助，如果大家有任何疑问请给我留言，小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持！

查看所有标签

猜你喜欢:

• Kafka读书笔记 -- 安装与配置
• Kafka读书笔记 -- 集群安装与配置
• Kafka读书笔记 -- 单节点安装与配置
• Docker笔记（三）：Docker安装与配置
• Kafka读书笔记 -- 集群安装与配置（Ubuntu）
• Kafka读书笔记 -- 集群安装与配置（Docker）

本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。