1. 概述提供在 Rocky Linux 9.5 环境下从零搭建Filebeat - Kafka - Logstash - Elasticsearch - Kibana架构的标准化操作指南。该架构通过 Kafka 实现日志的缓冲与削峰利用 ELK 栈完成日志的清洗、存储与可视化最终实现全链路日志的浏览器端实时监控。执行范围所有节点2.1 关闭防火墙与 SELinux为降低测试环境的网络配置复杂度需关闭系统安全策略systemctl stop firewalld systemctl disable firewalldsetenforce 0sed -i s/^SELINUX.*/SELINUXdisabled/ /etc/selinux/config2.2 调整系统资源限制Elasticsearch 对虚拟内存与文件描述符有严格要求需执行以下调优# 调整虚拟内存映射echo vm.max_map_count262144 /etc/sysctl.conf sysctl -p# 调整文件描述符和进程数限制cat EOF /etc/security/limits.conf* soft nofile 65535* hard nofile 65535* soft nproc 4096* hard nproc 4096EOF2.3 安装 Java 运行环境 (JDK 17)wget https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.11%2B9/OpenJDK17U-jdk_x64_linux_hotspot_17.0.11_9.tar.gztar -zxvf OpenJDK17U-jdk_x64_linux_hotspot_17.0.11_9.tar.gz -C /usr/local/mv /usr/local/jdk-17.0.119 /usr/local/jdk17cat EOF /etc/profileexport JAVA_HOME/usr/local/jdk17export PATH\$JAVA_HOME/bin:\$PATHEOFsource /etc/profilejava -version # 验证输出3. 核心组件部署3.1 部署 Elasticsearch (存储与检索)1.下载与解压wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.15.0-linux-x86_64.tar.gztar -zxvf elasticsearch-8.15.0-linux-x86_64.tar.gz -C /usr/local/mv /usr/local/elasticsearch-8.15.0 /usr/local/elasticsearch2创建专用用户ES 严禁 root 运行useradd eschown -R es:es /usr/local/elasticsearch3修改核心配置(/usr/local/elasticsearch/config/elasticsearch.yml)cluster.name: elk-clusternode.name: node-1network.host: 0.0.0.0http.port: 9200discovery.type: single-nodexpack.security.enabled: falsexpack.security.http.ssl.enabled: false4启动服务su - es -c /usr/local/elasticsearch/bin/elasticsearch -dcurl http://localhost:9200 # 验证返回 JSON 信息3.2 部署 Kafka (消息缓冲)5下载与解压:wget https://downloads.apache.org/kafka/4.0.1/kafka_2.13-4.0.1.tgztar -zxvf kafka_2.13-4.0.1.tgz -C /opt/mv /opt/kafka_2.13-4.0.1 /opt/kafka6初始化 KRaft 模式KAFKA_CLUSTER_ID$(/opt/kafka/bin/kafka-storage.sh random-uuid)/opt/kafka/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /opt/kafka/config/kraft/server.properties --standalone7启动服务/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/kraft/server.properties3.3 部署 Logstash (数据清洗与转发)8.下载与解压wget https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gztar -zxvf logstash-8.15.0-linux-x86_64.tar.gz -C /usr/local/mv /usr/local/logstash-8.15.0 /usr/local/logstash9编写数据流转配置(/usr/local/logstash/config/kafka-to-es.conf)input {kafka {bootstrap_servers localhost:9092topics [app-logs]codec json}}output {elasticsearch {hosts [http://localhost:9200]index app-logs-%{YYYY.MM.dd}}}10.启动服务/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/kafka-to-es.conf 3.4 部署 Filebeat (日志采集)11下载与解压wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.15.0-linux-x86_64.tar.gztar -zxvf filebeat-8.15.0-linux-x86_64.tar.gz -C /usr/local/mv /usr/local/filebeat-8.15.0-linux-x86_64 /usr/local/filebeat12修改配置文件(/usr/local/filebeat/filebeat.yml)filebeat.inputs:- type: logenabled: truepaths:- /var/log/*.log # 替换为实际日志路径output.kafka:hosts: [localhost:9092]topic: app-logsrequired_acks: 113启动服务/usr/local/filebeat/filebeat -e -c /usr/local/filebeat/filebeat.yml 4. 浏览器可视化方案4.1 部署 Kibana (查看清洗后的日志)14下载解压wget https://artifacts.elastic.co/downloads/kibana/kibana-8.15.0-linux-x86_64.tar.gztar -zxvf kibana-8.15.0-linux-x86_64.tar.gz -C /usr/local/mv /usr/local/kibana-8.15.0-linux-x86_64 /usr/local/kibana15修改配置(/usr/local/kibana/config/kibana.yml)server.host: 0.0.0.0elasticsearch.hosts: [http://localhost:9200]i18n.locale: zh-CN16启动与访问/usr/local/kibana/bin/kibana 浏览器访问http://服务器IP:5601在 Stack Management 中创建app-logs-*索引模式即可在 Discover 中查看日志。4.2 部署 Kafka-UI (查看 Kafka 原始消息)若需在浏览器中直接监控 Kafka 缓冲区的消息可通过 Docker 快速部署 Kafka-UIdocker run -d --name kafka-ui -p 8080:8080 \-e KAFKA_CLUSTERS_0_NAMElocal \-e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERSlocalhost:9092 \provectuslabs/kafka-ui:latest浏览器访问http://服务器IP:8080点击对应 Topic 即可浏览实时消息内容。17.版本一致性Elasticsearch、Logstash、Kibana、Filebeat 的主版本号必须完全一致本文统一为 8.15.0否则会导致通信协议不兼容。18.内存分配Elasticsearch 默认占用 2G 内存。若服务器内存较小需修改/usr/local/elasticsearch/config/jvm.options中的-Xms和-Xmx参数。19.权限问题若 ES 启动报错首先检查目录归属权是否为es用户严禁使用root启动 Elasticsearch。20.路径检查:确保所有配置文件路径正确特别是 Kafka 的配置文件路径可能需要根据实际安装位置调整。21.网络连通性确保各组件之间的网络端口可以互相访问特别是 9092(Kafka)、9200(Elasticsearch)、5601(Kibana) 端口