docker部署kafka
1、拉取镜像 docker pull wurstmeister/zookeeper docker pull wurstmeister/kafka 2、创建容器创建并运行zookeeper
docker run -d --network=docker-network --name=zookeeper -p 2181:2181 --privileged=true --restart always -v /etc/localtime:/etc/localtime wurstmeister/zookeeper创建并运行kafka
docker run -d --restart=always --privileged=true --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.139.187:2181/kafka -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.139.187:9092 -e KAFKA_LISTENERS=PLAINTEXT://:9092 --restart always -v /etc/localtime:/etc/localtime wurstmeister/kafka这里我遇到了个问题
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTINGat kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361)at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)at kafka.server.KafkaServer.startup(KafkaServer.scala:205)at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)at kafka.Kafka$.main(Kafka.scala:75)at kafka.Kafka.main(Kafka.scala)这个问题是因为我原先的 KAFKA_ZOOKEEPER_CONNECT 配置的问题
我曾尝试过这些,这些对我来说都有问题
server.properties配置文件的内容如下所示
broker.id=0 # 表示broker的编号,如果集群中有多个broker,则每个broker的编号需要设置的不同listeners=PLAINTEXT://:9092 # brokder对外提供的服务入口地址log.dirs=/tmp/kafka-logs # 设置存放消息日志文件的地址zookeeper.connect=192.168.139.187:2181/kafka # Kafka所需Zookeeper集群地址,学习中Zookeeper和Kafka都安装本机 4、创建主题这个个指令在我安装配置的kafka会发生新手级别错误:
温暖的冬天/kafka-topics.sh --zookeeper 192.168.139.187:2181 --create --topic itheima --partitions 2 --replication-factor 1# 创建主题,表示副本因子,表示每一个分区具有的副本数(包含分区本身),副本因子数不能超过broker的数量这里我遇到这个问题
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.[2020-10-29 14:24:53,137] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0. (kafka.admin.TopicCommand$)这个问题只因为指定的zookeeper地址不对,创建kafka的时候配置的是KAFKA_ZOOKEEPER_CONNECT=192.168.139.187:2181/kafka所以正确的指定令为:
温暖的冬天/kafka-topics.sh --zookeeper 192.168.139.187:2181/kafka --create --topic itheima --partitions 2 --replication-factor 1 # 查看所有的主题温暖的冬天/kafka-topics.sh --zookeeper 192.168.139.187:2181/kafka --list# 查看主题的详细信息温暖的冬天/kafka-topics.sh --zookeeper 192.168.139.187:2181/kafka --describe --topic itheima# 删除主题温暖的冬天/kafka-topics.sh --delete --zookeeper 192.168.139.187:2181/kafka --topic itheima# 开启一个消费者(使用新shell窗口,用来测试消费数据)温暖的冬天/kafka-console-consumer.sh --bootstrap-server 192.168.139.187:9092 --topic itheima# 开启一个生产者(使用新shell窗口,用来测试接收数据)温暖的冬天/kafka-console-producer.sh --broker-list 192.168.139.187:9092 --topic itheima修改分区数增加分区数温暖的冬天/kafka-topics.sh --alter --zookeeper 192.168.139.187:2181/kafka --topic itheima --partitions 3修改分区数时,仅能增加分区个数。温暖的冬天/kafka-topics.sh --alter --zookeeper 192.168.139.187:2181/kafka --topic heima --partitions 2