kafka学习

kafka学习 -- 服务端命令学习

写在前面一、服务端所有脚本二、常用的脚本2.1、服务端启动配置2.2、broker 节点2.3、Topic2.4、producer2.5、consumer 和 group2.6、zookeeper2.7、性能测试2.8、日志查询三、更多Kafka,，客户端、服务端，监控可参考下文

写在前面

这里 介绍一下 kafka_2.11-1.0.2 版本服务端命令

一、服务端所有脚本

connect-distributed.sh
kafka-consumer-perf-test.sh
kafka-replica-verification.sh
kafka-verifiable-producer.sh

connect-standalone.sh
kafka-delete-records.sh
kafka-run-class.sh
trogdor.sh

kafka-acls.sh
kafka-log-dirs.sh
kafka-server-start.sh
windows
kafka-broker-api-versions.sh
kafka-mirror-maker.sh
kafka-server-stop.sh
zookeeper-security-migration.sh

kafka-configs.sh
kafka-preferred-replica-election.sh
kafka-simple-consumer-shell.sh
zookeeper-server-start.sh

kafka-console-consumer.sh
kafka-producer-perf-test.sh
kafka-streams-application-reset.sh
zookeeper-server-stop.sh

kafka-console-producer.sh
kafka-reassign-partitions.sh
kafka-topics.sh
zookeeper-shell.sh

kafka-consumer-groups.sh
kafka-replay-log-producer.sh
kafka-verifiable-consumer.sh

二、常用的脚本

2.1、服务端启动配置

bin/zookeeper-server-start.sh config/zookeeper.propertiesbin/kafka-server-start.sh config/server.properties
bin/kafka-server-start.sh config/server1.properties
bin/kafka-server-start.sh config/server2.properties

2.2、broker 节点

在这里插入代码片

2.3、Topic

bin/kafka-topics.sh --create --zookeeper 192.168.1.74:2181 --replication-factor 3 --partitions 1 --topic topic2
bin/kafka-topics.sh  --delete --zookeeper 192.168.1.74:2181  --topic topic1bin/kafka-topics.sh --list --zookeeper 192.168.1.74:2181bin/kafka-topics.sh --describe --zookeeper 192.168.1.74:2181 --topic topic1

2.4、producer

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic1

2.5、consumer 和 group

consumer start

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic topic1
bin/kafka-console-consumer.sh --zookeeper 192.168.1.74:2181 --from-beginning --topic topic1

group命令，这里可看到 消费者组的Topic 管理（分区）

bin/kafka-consumer-groups.sh --zookeeper 192.168.1.74:2181 --list
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --listbin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group dltGroup --describe
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group program --describe
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group fooGroup --describe
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group multiGroup --describe

2.6、zookeeper

启动 zookeeper 客户端，管理 kafka

bin/zookeeper-shell.sh 192.168.1.74:2181
ls /
ls /brokers/topics

查看 节点

ls /brokers/ids

查看 topic

ls /brokers/topics

查看删除的消息

ls /admin/delete_topics

其他

ls /log_dir_event_notification
ls /consumers
ls /consumers/console-consumer-45795
ls /consumers/console-consumer-45795/offsets
ls /consumers/console-consumer-45795/offsets/topic2

ls /consumers/console-consumer-97962/offsets/topic1
ls /consumers/console-consumer-97962/owners/topic1
ls /consumers/console-consumer-97962/ids

ls /vod_clip_dev_new
ls /latest_producer_id_block
ls /controller_epoch
ls /isr_change_notification
ls /admin
ls /admin/delete_topics
ls /zookeeper
ls /zookeeper/quota
ls /config
ls /cluster
ls /cluster/id
ls /controller
ls /brokers
ls /dubbo-dev

2.7、性能测试

bin/kafka-consumer-perf-test.sh --broker-list localhost:9092 --group fooGroup --topic topic1 --messages 1024

2.8、日志查询

bin/kafka-log-dirs.sh --describe --bootstrap-server 127.0.0.1:9092 --topic-list foos

结果

{"version": 1,"brokers": [{"broker": 0,"logDirs": [{"logDir": "/kafka/kafka-logs-3d0b0150932f","error": null,"partitions": [{"partition": "foos-0","size": 384,"offsetLag": 0,"isFuture": false}]}]}]
}

然后，查看日志

ls  /kafka/kafka-logs-3d0b0150932f/foos-0tail -f /kafka/kafka-logs-3d0b0150932f/foos-0/00000000000000000000.log

三、更多Kafka,，客户端、服务端，监控可参考下文

kafka学习 – kafka connectkafka学习 – 客户端Consumer API学习kafka学习 – 客户端Producer API学习kafka学习 – 服务端命令学习Springboot集成Kafka