kafka集群搭建 - ZooKeeper

栏目: 编程工具 · 发布时间: 6年前

内容简介：ZooKeeper（动物园管理员），顾名思义，是用来管理Hadoop（大象）、Hive（蜜蜂）、Pig（小猪）的管理员，同时Apache HBase、Apache Solr、LinkedIn Sensei等众多项目中都采用了ZooKeeper。ZooKeeper是一个集中式服务，用于维护配置信息、命名、提供分布式同步和提供组服务。所有这些类型的服务都以某种形式被分布式应用程序使用。每次它们被实现时，都有大量的工作用于修复不可避免的bug和竞选条件。由于实现这类服务的困难，应用程序最初通常跳过它们，这使得它

ZooKeeper（动物园管理员），顾名思义，是用来管理Hadoop（大象）、Hive（蜜蜂）、Pig（小猪）的管理员，同时Apache HBase、Apache Solr、LinkedIn Sensei等众多项目中都采用了ZooKeeper。

ZooKeeper是一个集中式服务，用于维护配置信息、命名、提供分布式同步和提供组服务。所有这些类型的服务都以某种形式被分布式应用程序使用。每次它们被实现时，都有大量的工作用于修复不可避免的bug和竞选条件。由于实现这类服务的困难，应用程序最初通常跳过它们，这使得它们在存在更改时变得脆弱，并且难以管理。即使正确完成，这些服务的不同实现也会在部署应用程序时导致管理复杂性。

二、Zookeeper集群

Kafka使用Zookeeper来存储集群元数据以及消费者元数据。Zookeeper集群被称为群组。Zookeeper使用的是一致性协议，所以建议每个群组里应该包含奇数个节点（比如3个、5个等），因为只有当群组里的大多数节点（也就是法定人数）处于可用状态，Zookeeper才能处理外部的请求。也就是说，如果3个节点的集群，允许1个节点失效。如果5个节点的集群，允许2个节点失效。

三、 Docker 方式配置ZooKeeper

myid文件：标识要写到快照目录下面myid文件里

采用Docker镜像：wurstmeister/zookeeper，部署一个3节点的集群。

节点一：

  • docker-compose.yml
version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./zoo.cfg:/opt/zookeeper-3.4.9/conf/zoo.cfg
      - ./data:/opt/zookeeper-3.4.9/data
      - ./logs:/opt/zookeeper-3.4.9/logs
      - ./myid:/opt/zookeeper-3.4.9/data/myid
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
    restart: always

  • myid文件

    1

  • zoo.cfg文件

    # The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper-3.4.9/data
dataLogDir=/opt/zookeeper-3.4.9/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#kafka01 (10.100.158.245)
server.1=0.0.0.0:2888:3888
#kafka02 (10.100.19.15)
server.2=10.150.19.15:2888:3888
#kafka03 (10.100.124.217)
server.3=10.100.124.217:2888:3888

节点二：

  • docker-compose.yml

    version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./zoo.cfg:/opt/zookeeper-3.4.9/conf/zoo.cfg
      - ./data:/opt/zookeeper-3.4.9/data
      - ./logs:/opt/zookeeper-3.4.9/logs
      - ./myid:/opt/zookeeper-3.4.9/data/myid
    ports:
      - "2180:2181"
      - "2888:2888"
      - "3888:3888"
    restart: always

  • myid文件

    2

  • zoo.cfg文件

    # The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper-3.4.9/data
dataLogDir=/opt/zookeeper-3.4.9/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#kafka01 (10.100.158.245)
server.1=10.100.158.245:2888:3888
#kafka02 (10.100.19.15)
server.2=0.0.0.0:2888:3888
#kafka03 (10.100.124.217)
server.3=10.100.124.217:2888:3888

节点三：

  • docker-compose.yml

    version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./zoo.cfg:/opt/zookeeper-3.4.9/conf/zoo.cfg
      - ./data:/opt/zookeeper-3.4.9/data
      - ./logs:/opt/zookeeper-3.4.9/logs
      - ./myid:/opt/zookeeper-3.4.9/data/myid
    ports:
      - "2179:2181"
      - "2888:2888"
      - "3888:3888"
    restart: always

  • myid文件

    3

  • zoo.cfg文件

    # The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/zookeeper-3.4.9/data
dataLogDir=/opt/zookeeper-3.4.9/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#kafka01 (10.100.158.245)
server.1=10.100.158.245:2888:3888
#kafka02 (10.100.19.15)
server.2=10.100.19.15:2888:3888
#kafka03 (10.100.124.217)
server.3=0.0.0.0:2888:3888

四、查看ZooKeeper状态 

# 服务器1

[root@VM_158_24_centos ~]# docker exec -it s1_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: leader
 
 
 
#服务器2
 
[root@VM_19_15_centos ~]# docker exec -it s2_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower
 
 
 
#服务器3
 
[root@VM_124_27_centos ~]# docker exec -it s3_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower

停掉leader，2个follower的状态有变化，如下：

# 服务器1
 
[root@VM_158_245_centos ~]# docker stop s1_zookeeper_1
s1_zookeeper_1
 
 
# 服务器2
 
2018-10-30 10:54:44,555 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 1 at election address /10.154.158.245:3888
java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
 at java.lang.Thread.run(Thread.java:745)
2018-10-30 10:54:44,556 [myid:2] - INFO [WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.154.158.245 to address: /10.154.158.245
2018-10-30 10:54:44,556 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x500000000 (n.zxid), 0xf4 (n.round), LOOKING (n.state), 2 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
2018-10-30 10:54:44,759 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:QuorumPeer@844] - FOLLOWING
2018-10-30 10:54:44,759 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:ZooKeeperServer@173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /opt/zookeeper-3.4.9/logs/version-2 snapdir /opt/zookeeper-3.4.9/data/version-2
2018-10-30 10:54:44,759 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:Follower@61] - FOLLOWING - LEADER ELECTION TOOK - 215
2018-10-30 10:54:44,760 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.105.124.217 to address: /10.105.124.217
2018-10-30 10:54:44,761 [myid:2] - WARN [QuorumPeer[myid=2]/0.0.0.0:2181:Learner@236] - Unexpected exception, tries=0, connecting to /10.105.124.217:2888
java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:228)
 at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:69)
 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)
2018-10-30 10:54:45,794 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:Learner@329] - Getting a snapshot from leader
2018-10-30 10:54:45,796 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:FileTxnSnapLog@240] - Snapshotting: 0x500000000 to /opt/zookeeper-3.4.9/data/version-2/snapshot.500000000
 
 
# 服务器3
2018-10-30 10:54:44,567 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:FastLeaderElection@818] - New election. My id = 3, proposed zxid=0x500000000
2018-10-30 10:54:44,568 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 1 at election address /10.154.158.245:3888
java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:579)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
 at java.lang.Thread.run(Thread.java:745)
2018-10-30 10:54:44,569 [myid:3] - INFO [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: 10.154.158.245 to address: /10.154.158.245
2018-10-30 10:54:44,569 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x500000000 (n.zxid), 0xf4 (n.round), LOOKING (n.state), 3 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
2018-10-30 10:54:44,575 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x500000000 (n.zxid), 0xf4 (n.round), LOOKING (n.state), 2 (n.sid), 0x5 (n.peerEpoch) LOOKING (my state)
2018-10-30 10:54:44,776 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer@856] - LEADING
2018-10-30 10:54:44,779 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@59] - TCP NoDelay set to: true
2018-10-30 10:54:44,780 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:ZooKeeperServer@173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /opt/zookeeper-3.4.9/logs/version-2 snapdir /opt/zookeeper-3.4.9/data/version-2
2018-10-30 10:54:44,780 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@361] - LEADING - LEADER ELECTION TOOK - 219
2018-10-30 10:54:45,782 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@329] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@7f54169
2018-10-30 10:54:45,808 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@384] - Synchronizing with Follower sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x300000000
2018-10-30 10:54:45,809 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@458] - Sending SNAP
2018-10-30 10:54:45,809 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@482] - Sending snapshot last zxid of peer is 0x300000000 zxid of leader is 0x600000000sent zxid of db as 0x500000000
2018-10-30 10:54:45,820 [myid:3] - INFO [LearnerHandler-/10.154.19.15:42938:LearnerHandler@518] - Received NEWLEADER-ACK message from 2
2018-10-30 10:54:45,820 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@952] - Have quorum of supporters, sids: [ 2,3 ]; starting up and setting last processed zxid: 0x600000000

再次查看status:

 
#服务器2
 
[root@VM_19_15_centos ~]# docker exec -it s2_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower
 
 
 
#服务器3
 
[root@VM_124_217_centos ~]# docker exec -it s3_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: leader

服务器3变成leader了！

重新启动服务器1，服务器1是follower

 
[root@VM_158_245_centos ~]# docker start s1_zookeeper_1
 
s1_zookeeper_1
 
[root@VM_158_245_centos ~]# docker exec -it s1_zookeeper_1 bin/zkServer.sh status
 
ZooKeeper JMX enabled by default
 
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
 
Mode: follower

参考

Configuring Zookeeper Cluster

多台虚拟机搭建zookeeper集群

搭建kafka集群详细教程

zookeeper.apache.org


以上所述就是小编给大家介绍的《kafka集群搭建 - ZooKeeper》，希望对大家有所帮助，如果大家有任何疑问请给我留言，小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持！

查看所有标签

猜你喜欢:

本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。

码农书籍
Google's PageRank and Beyond

Google's PageRank and Beyond

Amy N. Langville、Carl D. Meyer / Princeton University Press / 2006-7-23 / USD 57.50

Why doesn't your home page appear on the first page of search results, even when you query your own name? How do other web pages always appear at the top? What creates these powerful rankings? And how......一起来看看 《Google's PageRank and Beyond》 这本书的介绍吧!

码农工具
URL 编码/解码
URL 编码/解码

URL 编码/解码

HSV CMYK 转换工具
HSV CMYK 转换工具

HSV CMYK互换工具

  • New
  • 文章
  • 话题
  • 教程
    • 关注 码农网 公众号
    版权所有，保留一切权利！© 2018-2025 码农网 粤ICP备17054400号-3