ELK收集网络设备日志

分步走1、新建一台Linux虚拟机做为rsyslog服务器接收网络设备日志2、在上述虚拟机上部署filebeat将接收到的网络日志发送到logstash上3、最后通过logstash针对性配置将日志发送给es并在kibana上显示。1、部署rsyslog服务器1、新建rsyslog配置文件 [rootlocalhost ~]# vi /etc/rsyslog.d/10-network-output.conf # 定义模板按来源IP和日期分割日志文件 # 格式/var/log/network-devices/源IP/年-月-日.log template(nameNetworkDeviceLog typestring string/home/network-devices/%fromhost-ip%/%$YEAR%-%$MONTH%-%$DAY%.log ) # 规则将所有来自网络的日志排除本地 localhost写入上述模板 # 注意这里假设网络设备日志没有特定的 facility 限制如果有可改为 local0.* 等 :fromhost-ip, !isequal, 127.0.0.1 ?NetworkDeviceLog # 确保目录存在并设置权限 # mkdir -p /var/log/network-devices # chown -R syslog:adm /var/log/network-devices (CentOS可能是 root:root 或 syslogs:syslogs)2、启动rsyslog服务并检查端口514是否存在[rootlocalhost ~]# systemctl enable rsyslog [rootlocalhost ~]# systemctl start rsyslog [rootlocalhost ~]# systemctl status rsyslog ● rsyslog.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2026-06-24 14:22:55 CST; 1h 45min ago Docs: man:rsyslogd(8) http://www.rsyslog.com/doc/ Main PID: 28876 (rsyslogd) CGroup: /system.slice/rsyslog.service └─28876 /usr/sbin/rsyslogd -n Jun 24 14:22:55 localhost.localdomain systemd[1]: Starting System Logging Service... Jun 24 14:22:55 localhost.localdomain rsyslogd[28876]: [origin softwarersyslogd swVersion8.24.0-55.el7 x-pid28876 x-infohttp://www.rsyslog.com] start Jun 24 14:22:55 localhost.localdomain systemd[1]: Started System Logging Service. [rootlocalhost ~]# netstat -lntup |grep 514 tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 28876/rsyslogd tcp6 0 0 :::514 :::* LISTEN 28876/rsyslogd udp 0 0 0.0.0.0:514 0.0.0.0:* 28876/rsyslogd udp6 0 0 :::514 :::* 28876/rsyslogd3、网络设备上设置策略将选定日志发送到rsyslog的514端口(可由网工配置)4、检查是否有日志接收到[rootlocalhost home]# ll /home/network-devices/10.10.254.10/ total 32512 -rw-------. 1 root root 20922019 Jun 24 16:11 2026-06-24.log 有日志写入5、在rsyslog上部署filebeat并修改配置文件部署过程忽略 [rootlocalhost home]# vim /opt/filebeat/filebeat.yml filebeat.inputs: - type: log id: logs-app-network-devices #ID自定义 enabled: true paths: - /home/network-devices/*/*.log #日志路径自定义 #tags: [network-device] encoding: utf-8 fields: log_source: network-devices #系统-必填 区分索引 log_source_env: prod #环境 fields_under_root: false # multiline: # pattern: ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} # negate: true # match: after # max_lines: 1000 scan_frequency: 10s close_inactive: 1m output.logstash: hosts: [10.10.200.35:8888] compression_level: 3 loadbalance: true bulk_max_size: 2048 logging.level: info logging.to_files: true logging.files: path: /opt/filebeat/logs #目录自定义 name: filebeat.log keepfiles: 76、转到logstash服务器新建专门接收网络日志的conf[rootelk-lo-node03 pipeline]# pwd /opt/logstash/config/pipeline [rootelk-lo-node03 pipeline]# vim network-device-log.conf input { beats { port 8888 #与filebeat配置的发送端口一致 # 可选限制只接受来自特定 IP 的连接增强安全性 # host 0.0.0.0 } } filter { # 1. 识别来源根据 Filebeat 中定义的 fields.log_source 进行判断 if [fields][log_source] network-devices { # 2. 添加标签便于在 Kibana 中快速筛选或创建专用视图 mutate { add_tag [network_device_log] add_field { [metadata][target_index] logs-app-network-devices } } } # 其他来源的日志可以在这里添加 else if 分支处理 } output { # 调试输出确认数据结构和字段是否正确生产环境稳定后可注释掉 stdout { codec rubydebug } # 输出到 Elasticsearch elasticsearch { hosts [https://10.10.200.31:9200] # 动态索引名称 # 最终生成的索引名为: logs-app-network-2026.06.24 index %{[metadata][target_index]}-%{yyyy.MM.dd} # 认证信息 user elastic password JcJv*N7rUT6fE6fik4oY # SSL 配置根据你的 ES 集群实际情况调整 ssl_certificate_verification false # 模板管理可选如果希望自动应用映射模板 # manage_template true # template_name network-log-template } }7、因为pipeline包含多个conf需要额外修改pipelines.yml注明[rootelk-lo-node03 config]# ll total 48 -rw-r--r-- 1 root root 2924 Apr 1 17:49 jvm.options -rw-r--r-- 1 root root 8680 Apr 1 17:49 log4j2.properties -rw-r--r-- 1 root root 502 Jun 4 14:37 logstash.conf -rw-r--r-- 1 root root 342 Apr 1 17:49 logstash-sample.conf -rw-r--r-- 1 root root 15745 Apr 1 17:49 logstash.yml drwxr-xr-x 2 root root 98 Jun 24 16:21 pipeline -rw-r--r-- 1 root root 837 Jun 16 16:54 pipelines.yml -rw-r--r-- 1 root root 1696 Apr 1 17:49 startup.options [rootelk-lo-node03 config]# pwd /opt/logstash/config [rootelk-lo-node03 config]# vim pipelines.yml - pipeline.id: beats-elk-log path.config: /opt/logstash/config/pipeline/beats-elk-log.conf pipeline.workers: 2 pipeline.batch.size: 125 pipeline.batch.delay: 50 queue.type: persisted queue.max_bytes: 1gb queue.checkpoint.acks: 1024 queue.drain: false - pipeline.id: windows-winlogbeat-log path.config: /opt/logstash/config/pipeline/windows-winlogbeat-log.conf pipeline.workers: 2 pipeline.batch.size: 125 pipeline.batch.delay: 50 queue.type: persisted queue.max_bytes: 1gb queue.checkpoint.acks: 1024 queue.drain: false - pipeline.id: network-device-log path.config: /opt/logstash/config/pipeline/network-device-log.conf pipeline.workers: 2 pipeline.batch.size: 125 pipeline.batch.delay: 50 queue.type: persisted queue.max_bytes: 1gb queue.checkpoint.acks: 1024 queue.drain: false8、重启logstash并新开窗口检查是否有接收到网络设备日志[rootelk-lo-node03 config]# systemctl restart logstash [rootelk-lo-node03 config]# journalctl -u logstash.service -f Jun 24 16:23:29 elk-lo-node03 logstash[193222]: message Jun 24 16:23:29 10.10.254.10 5823434235018048(root) 43240501 HillstoneNetworks#EventNET: ARP entry is created, 10.10.254.65, 8840.33e8.c0d7, trust-vr, Jun 24 16:23:29 elk-lo-node03 logstash[193222]: timestamp 2026-06-24T08:16:47.271Z, Jun 24 16:23:29 elk-lo-node03 logstash[193222]: host { Jun 24 16:23:29 elk-lo-node03 logstash[193222]: name localhost.localdomain Jun 24 16:23:29 elk-lo-node03 logstash[193222]: }, Jun 24 16:23:29 elk-lo-node03 logstash[193222]: input { Jun 24 16:23:29 elk-lo-node03 logstash[193222]: type log Jun 24 16:23:29 elk-lo-node03 logstash[193222]: }, Jun 24 16:23:29 elk-lo-node03 logstash[193222]: ecs { Jun 24 16:23:29 elk-lo-node03 logstash[193222]: version 8.0.0 Jun 24 16:23:29 elk-lo-node03 logstash[193222]: }, Jun 24 16:23:29 elk-lo-node03 logstash[193222]: log { Jun 24 16:23:29 elk-lo-node03 logstash[193222]: offset 21896295, Jun 24 16:23:29 elk-lo-node03 logstash[193222]: file { Jun 24 16:23:29 elk-lo-node03 logstash[193222]: path /home/network-devices/10.10.254.10/2026-06-24.log 检查看到日志被logstash成功接收到9、kibana上检查索引管理上是否有自动创建索引(若没有则说明需要修改logstash上的对应conf文件)10、新建对应网络设备日志的数据视图11、进入discover检查日志以上完成~