kafka 修改topic partitions

栏目: 后端 · 发布时间: 5年前

内容简介:kafka topic的partitions为1时,存取的数据,只会在kafka集群中的一台机器上面,时间一长,必然会造成,资源不均衡。分区由原来一个变成了三个,并且集群里面的所有kafka,data目录都会产生相应的文件夹注意:

kafka topic的partitions为1时,存取的数据,只会在kafka集群中的一台机器上面,时间一长,必然会造成,资源不均衡。

1,修改topic partitions数量

[root@bigserver2 kafka]# ./bin/kafka-topics.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --alter --topic track_pc --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@bigserver2 kafka]# bin/kafka-topics.sh --describe --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --topic track_pc
Topic:track_pc PartitionCount:3 ReplicationFactor:1 Configs:
 Topic: track_pc Partition: 0 Leader: 1 Replicas: 1 Isr: 1
 Topic: track_pc Partition: 1 Leader: 2 Replicas: 2 Isr: 2
 Topic: track_pc Partition: 2 Leader: 0 Replicas: 0 Isr: 0

分区由原来一个变成了三个,并且集群里面的所有kafka,data目录都会产生相应的文件夹

2,生成均衡配置文件

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --topics-to-move-json-file track_pc.json --broker-list "0,1,2" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[1],"log_dirs":["any"]}]}

注意: 0,1,2是kafka配置文件中的broker.id,在这里只是生成配置文件,并没有真正均衡

3,均衡partitions

将2中,生成的json,copy到一个json文件中,执行以下命令

# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"track_pc","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"track_pc","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"track_pc","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --verify
Status of partition reassignment:
Reassignment of partition track_pc-0 is still in progress
Reassignment of partition track_pc-2 is still in progress
Reassignment of partition track_pc-1 is still in progress

[root@bigserver2 kafka]# bin/kafka-reassign-partitions.sh --zookeeper bigserver1:2181,bigserver2:2181,testing:2181 --reassignment-json-file exec.json --verify
Status of partition reassignment:
Reassignment of partition track_pc-0 completed successfully
Reassignment of partition track_pc-2 completed successfully
Reassignment of partition track_pc-1 completed successfully

注意: 到这里均衡已结束,但是对已有的数据,并不能起到均衡作用。


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

成功由我

成功由我

刘世英、彭征 / 湖南人民出版社 / 2010-2 / 28.00元

《成功由我:李彦宏快乐成功之道》讲述:他,18岁高分考入北京大学,毕业后到美国名校学习最热门的计算机专业,然后闯荡于华尔街和硅谷这两大金融和技术圣地,31岁回国创立百度……到如今身价数十亿美元,领导的百度发展成为全球第二大搜索引擎,在国内搜索市场占据近八成的市场份额,将有“上帝”之称的Google抛在身后,最近他又掀起了“框计算”风暴,并雄心万丈宣称“未来十年,要让百度在全球一半以上国家成为家喻户......一起来看看 《成功由我》 这本书的介绍吧!

在线进制转换器
在线进制转换器

各进制数互转换器

html转js在线工具
html转js在线工具

html转js在线工具

HEX CMYK 转换工具
HEX CMYK 转换工具

HEX CMYK 互转工具