宜人贷蜂巢ELK Stack之elasticsearch权限探索

栏目: Ruby · 发布时间: 5年前

内容简介：上文宜人贷蜂巢API网关技术解密之Netty使用实践提到了，API网关“承外对内”，将外部请求，转发到内部各个抓取服务。在网关中，不仅可以做鉴权、加解密、路由、限流功能；如果想了解各服务接口的调用次数、调用时间、成功次数、失败次数等接口相关的统计，都可以网关中埋点，采集相关信息，通过logstash放入elasticsearch（简称es），在kibana中或通过es的http请求获得各服务接口的统计信息。图前段时间有一个需求，我们的es需要作为一个服务提供出来。考虑到es内保存着敏感数据，需要对es添加权

前言

上文宜人贷蜂巢API网关技术解密之Netty使用实践提到了，API网关“承外对内”，将外部请求，转发到内部各个抓取服务。在网关中，不仅可以做鉴权、加解密、路由、限流功能；如果想了解各服务接口的调用次数、调用时间、成功次数、失败次数等接口相关的统计，都可以网关中埋点，采集相关信息，通过logstash放入elasticsearch（简称es），在kibana中或通过es的http请求获得各服务接口的统计信息。

ELK框架

图

前段时间有一个需求，我们的es需要作为一个服务提供出来。考虑到es内保存着敏感数据，需要对es添加权限管理功能。搜寻es权限管理，大概有三套方案：

ELK Stack官方安全插件X-Pack Securit
es安全插件Search Guard
自实现权限管理

考虑到X-Pack Security是收费软件，自开发一个权限管理模块成本比较高，最后选用了Search Guard的社区版本，其支持的功能已经满足当前需求。

图

Search Guard安装过程

Stop Elasticsearch

ps -ef | grep elasticsearch | grep -v grep |

awk {'print $2'} | xargs kill -9

Install Search Guard

/opt/soft/elasticsearch-5.5.1-guard/bin/

elasticsearch-plugin install -b

com.floragunn:search-guard-5:5.5.1-14

如果线上服务器不能访问外网，可以事先下载对应版本的 Search Guard，再以本地模式安装

/opt/soft/elasticsearch-5.5.1-guard/bin/

elasticsearch-plugin install -b

file:///opt/soft/search-guard-5-5.5.1-15.zip

产生证书及配置es

如果在测试环境安装es，可以使用软件自带的脚本，生成CA及相应证书，并自动配置了es

sh /opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/tools/install_demo_configuration.sh

但以上方式不适用于线上环境，我们的es希望在线上环境能提供权限访问控制，所以采用了search-guard-ssl生成线上证书
首先，下载了search-guard-ssl项目
git clone https://github.com/floragunnc...

生成证书，可以参考官方文档详细说明
建议修改证书信息及密码

example-pki-scripts/etc/root-ca.conf

example-pki-scripts/etc/signing-ca.conf

最后，执行证书生成脚本

sh ./example.sh

将相应的证书拷贝到es的config文件夹下
修改es配置

cluster.name: honeycomb-es-guard-5.5.1# node 1 to 5
node.name: node-1 
path.data: /data01/elasticsearch-guard
path.logs: /opt/logs/elasticsearch-guard

network.host: 0.0.0.0

http.cors.enabled: true
http.cors.allow-origin: "*"
thread_pool.bulk.queue_size: 50

searchguard.ssl.transport.keystore_filepath: node-1-keystore.jks
searchguard.ssl.transport.keystore_password: node_es
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: r_ca_honeycomb
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.enable_openssl_if_available: true
searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node-1-keystore.jks
searchguard.ssl.http.keystore_password: node_es
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: r_ca_honeycomb

searchguard.authcz.admin_dn:
  - CN=*,OU=client,O=client,L=test, C=de
  - CN=kirk,OU=client,O=client,L=test,C=DE
searchguard.ssl.http.clientauth_mode: NONE

Restart Elasticsearch.
su elk

/opt/soft/elasticsearch-5.5.1-guard/bin/elasticsearch -d
Initialise the Search Guard index by running sgadmin

<span style="font-family:monospace;font-size:14px;">/opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/sgconfig/ -cn honeycomb-es-guard-5.5.1 -ks /opt/soft/elasticsearch-5.5.1-guard/config/kirk-keystore.jks -kspass ki_ca_1 -ts /opt/soft/elasticsearch-5.5.1-guard/config/truststore.jks -tspass r_ca_honeycomb –nhnv</span>

测试结果

curl -k -u admin:admin ' https://localhost :9200/_searchguard/authinfo?pretty'

图

es装上Search Guard后的ELK框架

图

kibana修改步骤

安装插件

/opt/soft/kibana-5.5.1-linux-x86_64-guard/bin/kibana-plugin install file:///opt/soft/searchguard-kibana-5.5.1-4.zip

kibana配置

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.url: "https://localhost:9200"

kibana.index: ".kibana"

elasticsearch.username: "kibanaserver"

elasticsearch.password: "kibanaserver"

elasticsearch.ssl.verificationMode: none

start kibana

/opt/soft/kibana-5.5.1-linux-x86_64-guard/bin/kibana &

- 访问页面

![图][5]

##logstash修改步骤

logstash在按照官方文档配置，且咨询Search Guard官方后，并没有安装成功；现列出操作过程及问题现象，如有哪位读者了解，还望不吝赐教

- Configure logstash to use HTTPS instead of HTTP.
例如，hosts => "https://..."
- If you want to verify the server’s certificate (optional, but recommended), you need to provide the path to the keystore containing your Root CA as well.
logstash不需要验证es的证书，所以设置ssl_certificate_verification => false
如果进行证书验证，需配置CA路径及密码
- Configure the logstash username and password.

####最终配置

output {

elasticsearch {

hosts => ["https://xx.xx.xx.32:9200"]
    index => "%{log_project}-%{+YYYY-MM-dd}"

    ssl => true
    ssl_certificate_verification => false
    #truststore => "/opt/soft/elasticsearch-5.5.1-guard/config/truststore.jks"
    #truststore_password => r_ca_honeycomb

    user => logstash
    password => logstash

}

- start logstash

    /opt/soft/logstash-2.4.1-guard/bin/logstash -f /opt/soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf


##问题

####1. Search Guard 建议logstash不做证书验证且不需要提供truststore和truststore_password，但按操作后，进程启动成功，但仍有fail提示

WARNINGDetected UNSAFE options in elasticsearch output configuration!

WARNINGYou have enabled encryption but DISABLED certificate verification.

WARNINGTo make sure your data is secure change :ssl_certificate_verification to true {:level=>:warn}

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:class=>"Manticore::ClientProtocolException", :level=>:error}

Pipeline main started

####2. logstash在启动后，发送数据给es时，始终报错

Attempted to send a bulk request to Elasticsearch configured at '["https://xx.xx.xx.32:9200"]',

but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?

{:error_message=>"PKIX path building failed:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested

target", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/soft/logstash-

2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'",

"org/jruby/RubyProc.java:281:in `call'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'",

"org/jruby/RubyProc.java:281:in `call'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'",

"org/jruby/ext/thread/Mutex.java:149:in `synchronize'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in retrying_submit'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:in each_slice'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in multi_receive'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'",

"org/jruby/RubyHash.java:1342:in each'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'",

"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

看日志似乎是目的不可达，但在logstash机器上，通过命令测试，是可以访问的
curl -k -u logstash:logstash 'https://xx.xx.xx.32:9200/_searchguard/authinfo?pretty'

![图][6]

另外es有打印错误日志

2017-08-24T10:39:20,834[c.f.s.h.SearchGuardHttpServerTransport] [node-2] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
    at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

2017-08-24T10:39:21,663[c.f.s.h.SearchGuardHttpServerTransport] [node-2] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
    at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

####3、上文es的错误日志，显示es不知道证书（应该是客户端证书），但是已经配置了不做客户端认证，这又是为什么？
es部分配置


    searchguard.ssl.http.clientauth_mode: NONE


##总结

虽然logstash与es通信暂时没有成功，但是在尝试开通es权限的过程中，也积累了些经验和成果
1. 使用类似文中curl的方式（http请求），直接使用es服务
2. 添加了Search Guard的es如何对外提供域名访问，实现后端多点服务/路由
nginx可以处理来自外部的https请求，但目前不支持https代理，即nginx不能访问后端的https服务。如果仍然使用Search Guard，可以考虑使用四层负载均衡器lvs或者硬件负载均衡器，将请求透传给后端服务器；另外知乎文章上有推荐使用Squid。

##参考

https://floragunn.com/

https://github.com/floragunncom/search-guard-docs

https://github.com/floragunncom/search-guard-docs/blob/master/installation.md

https://github.com/floragunncom/search-guard-docs/blob/master/tls_generate_demo_certificates.md

https://github.com/floragunncom/search-guard-ssl/tree/5.5.0

https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md

https://github.com/floragunncom/search-guard-docs/blob/master/logstash.md

https://forum.nginx.org/read.php?2,15124,15256

https://www.zhihu.com/question/19871146

作者：蜂巢团队

[宜信技术学院][7]


  [1]: /img/bVbpnWX
  [2]: /img/bVbpnW4
  [3]: /img/bVbpnW8
  [4]: /img/bVbpnXd
  [5]: /img/bVbpnXp
  [6]: /img/bVbpnXr

以上就是本文的全部内容，希望本文的内容对大家的学习或者工作能带来一定的帮助，也希望大家多多支持码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。

码农书籍

Learning JavaScript

Shelley Powers / Oreilly & Associates Inc / 2006-10-17 / $29.99

As web browsers have become more capable and standards compliant, JavaScript has grown in prominence. JavaScript lets designers add sparkle and life to web pages, while more complex JavaScript has led......一起来看看《Learning JavaScript》这本书的介绍吧!

码农工具