环境信息操作系统CentOS 7.9 (x86_64)硬件4C8G4核8GB用户rootJDK 路径/usr/local/jdk1.8.0_491已安装Kafka 版本2.8.2Scala 2.13虚拟机 IP192.168.10.5一、配置 Java 环境变量# 设置 JAVA_HOME 和 PATH立即生效 export JAVA_HOME/usr/local/jdk1.8.0_491 export PATH$JAVA_HOME/bin:$PATH # 永久写入配置文件 echo export JAVA_HOME/usr/local/jdk1.8.0_491 /etc/profile echo export PATH\$JAVA_HOME/bin:\$PATH /etc/profile source /etc/profile # 验证 java -version # 应显示 java version 1.8.0_491 echo $JAVA_HOME二、系统优化针对 4C8G# 1. 内核参数优化 cat /etc/sysctl.conf EOF vm.swappiness 1 vm.max_map_count 262144 net.core.somaxconn 65535 net.core.rmem_default 262144 net.core.rmem_max 4194304 net.core.wmem_default 262144 net.core.wmem_max 1048576 EOF sysctl -p # 2. 文件描述符限制 cat /etc/security/limits.conf EOF * soft nofile 65535 * hard nofile 65535 EOF # 当前会话生效 ulimit -n 65535三、下载并安装 Kafkacd /usr/local wget https://archive.apache.org/dist/kafka/2.8.2/kafka_2.13-2.8.2.tgz tar -xzf kafka_2.13-2.8.2.tgz mv kafka_2.13-2.8.2 /usr/local/kafka # 创建数据目录 mkdir -p /data/zookeeper mkdir -p /data/kafka-logs四、配置 Kafka 堆内存4GB通过环境变量设置不修改启动脚本echo export KAFKA_HEAP_OPTS-Xmx4G -Xms4G /etc/profile source /etc/profile echo $KAFKA_HEAP_OPTS # 验证输出 -Xmx4G -Xms4G五、编写配置文件5.1 ZooKeeper 配置cat /usr/local/kafka/config/zookeeper.properties EOF dataDir/data/zookeeper clientPort2181 tickTime2000 initLimit5 syncLimit2 EOF5.2 Kafka Broker 配置vim /usr/local/kafka/config/server.properties将内容替换为以下注意advertised.listeners改为实际 IPbroker.id0 listenersPLAINTEXT://0.0.0.0:9092 advertised.listenersPLAINTEXT://192.168.10.5:9092 log.dirs/data/kafka-logs zookeeper.connectlocalhost:2181 num.network.threads5 num.io.threads8 socket.send.buffer.bytes1048576 socket.receive.buffer.bytes1048576 socket.request.max.bytes104857600 offsets.topic.replication.factor1 transaction.state.log.replication.factor1 transaction.state.log.min.isr1踩坑说明如果不设置offsets.topic.replication.factor1Kafka 会尝试用默认副本因子 3 创建__consumer_offsets主题但单机只有 1 个 broker导致消费者无法正常工作。六、配置 Systemd 服务6.1 ZooKeeper 服务vim /etc/systemd/system/zookeeper.service内容[Unit] DescriptionApache ZooKeeper Afternetwork.target [Service] Typesimple EnvironmentJAVA_HOME/usr/local/jdk1.8.0_491 EnvironmentKAFKA_HEAP_OPTS-Xmx1G -Xms1G ExecStart/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop/usr/local/kafka/bin/zookeeper-server-stop.sh Restarton-abnormal [Install] WantedBymulti-user.target6.2 Kafka 服务vim /etc/systemd/system/kafka.service内容[Unit] DescriptionApache Kafka Afterzookeeper.service network.target [Service] Typesimple EnvironmentJAVA_HOME/usr/local/jdk1.8.0_491 EnvironmentKAFKA_HEAP_OPTS-Xmx4G -Xms4G ExecStart/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop/usr/local/kafka/bin/kafka-server-stop.sh Restarton-abnormal [Install] WantedBymulti-user.target6.3 启动并设置开机自启systemctl daemon-reload systemctl start zookeeper systemctl start kafka #设置服务开机自启 systemctl enable zookeeper systemctl enable kafka systemctl list-unit-files |grep -Ew zookeeper|kafka # 检查状态 systemctl status zookeeper systemctl status kafka七、验证安装7.1 检查进程和端口jps # 应看到 QuorumPeerMain 和 Kafka 进程 netstat -tulnp | grep -E 2181|90927.2 手动创建主题/usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server 192.168.10.5:9092 --replication-factor 1 --partitions 1 --topic test7.3 消息收发测试生产者任选一种地址都行# 使用 IP /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 192.168.10.5:9092 --topic test # 或使用 localhost /usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test输入hello后回车。消费者同样可以使用 IP 或 localhost/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.10.5:9092 --topic test --from-beginning # 或 /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning应该立即看到hello。经验总结只要advertised.listeners配置为可路由的 IP如192.168.10.5客户端无论是用该 IP 还是localhost连接都能正常工作。这是因为listeners0.0.0.0:9092绑定了所有接口而advertised.listeners返回的地址客户端也能访问本机访问 localhost 和 192.168.10.5 都通。八、日志查看与故障排查8.1 Kafka 日志# 实时查看 Kafka 日志 tail -100f /usr/local/kafka/logs/server.log # 查看最近 100 行错误 tail -100 /usr/local/kafka/logs/server.log | grep -i error # 查看全部日志分页 less /usr/local/kafka/logs/server.log8.2 ZooKeeper 日志# 实时查看 ZooKeeper 日志 journalctl -u zookeeper -f # 查看最近 100 行 journalctl -u zookeeper -n 100 # 查看所有错误 journalctl -u zookeeper -p err8.3 查看 systemd 服务状态和日志# Kafka 服务日志 journalctl -u kafka -f # 查看服务启动失败详情 journalctl -u kafka -n 50 --no-pager九、常用管理命令主题操作# 创建主题test /usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server 192.168.10.5:9092 --replication-factor 1 --partitions 1 --topic test # 列出所有主题 /usr/local/kafka/bin/kafka-topics.sh --list --bootstrap-server 192.168.10.5:9092 # 查看主题详情 /usr/local/kafka/bin/kafka-topics.sh --describe --bootstrap-server 192.168.10.5:9092 --topic test # 删除主题需在 server.properties 中设置 delete.topic.enabletrue /usr/local/kafka/bin/kafka-topics.sh --delete --bootstrap-server 192.168.10.5:9092 --topic test查看消费进度偏移量/usr/local/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.10.5:9092 --topic test --time -1 # 输出格式 test:0:6 表示分区0有6条消息查看消费者组/usr/local/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 192.168.10.5:9092 --list查看 broker 配置/usr/local/kafka/bin/kafka-configs.sh --bootstrap-server 192.168.10.5:9092 --describe --broker 0服务管理systemctl start kafka zookeeper systemctl stop kafka zookeeper systemctl restart kafka systemctl status kafka systemctl enable kafka # 开机自启十、常见踩坑与解决方法问题现象原因解决方法Kafka 启动失败日志提示InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1未设置offsets.topic.replication.factor1在server.properties中添加该配置并重启消费者收不到消息但生产者无报错消费者使用了localhost而advertised.listeners是 IP导致网络不通或内部主题创建失败确保消费者地址与advertised.listeners中地址可达检查日志中的副本因子错误jps找不到命令JAVA_HOME未加入PATH重新执行export PATH$JAVA_HOME/bin:$PATH并写入/etc/profilekafka-run-class.sh报UnrecognizedOptionException某些工具参数名不同如GetOffsetShell用--broker-list而非--bootstrap-server查阅官方文档或使用--help生产者发送成功但消费者收不到GetOffsetShell显示 offset0消息实际未写入可能由于auto.create.topics.enablefalse且主题不存在先手动创建主题十一、重置环境清除所有数据systemctl stop kafka zookeeper rm -rf /data/kafka-logs/* /data/zookeeper/* systemctl start zookeeper kafka # 重新创建主题 /usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server 192.168.10.5:9092 --replication-factor 1 --partitions 1 --topic test附录完整配置文件清单/etc/profile中与 Kafka 相关的行export JAVA_HOME/usr/local/jdk1.8.0_491 export JRE_HOME$JAVA_HOME/jre export CLASSPATH.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_PATH$JAVA_HOME/bin:$JRE_HOME/bin export PATH$PATH:$JAVA_PATH export KAFKA_HEAP_OPTS-Xmx4G -Xms4G/usr/local/kafka/config/server.properties最终版broker.id0 listenersPLAINTEXT://0.0.0.0:9092 advertised.listenersPLAINTEXT://192.168.10.5:9092 log.dirs/data/kafka-logs zookeeper.connectlocalhost:2181 num.network.threads5 num.io.threads8 socket.send.buffer.bytes1048576 socket.receive.buffer.bytes1048576 socket.request.max.bytes104857600 auto.create.topics.enablefalse offsets.topic.replication.factor1 transaction.state.log.replication.factor1 transaction.state.log.min.isr1/usr/local/kafka/config/zookeeper.properties最终版dataDir/data/zookeeper clientPort2181 tickTime2000 initLimit5 syncLimit2
