ELK学习实验014：Nginx日志JSON格式收集
2021-04-27 08:28
标签：height mat cert rate core hat nbsp soft received https://demo.elastic.co/app/kibana#/dashboard/welcome_dashboard 环境先处理干净 安装nginx和httpd-tools [root@node4 ~]# ab -n 100 -c 100 http://192.168.132.134/ [root@node4 ~]# tail -f /usr/local/nginx/logs/access.log [root@node4 src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-x86_64.rpm [root@node4 src]# rpm -ivh filebeat-7.5.1-x86_64.rpm [root@node4 src]# rpm -qc filebeat [root@node4 src]# cd /etc/filebeat/ [root@node4 filebeat]# cp filebeat.yml /opt/ [root@node4 filebeat]# grep -Ev "^$|[#;]" filebeat.yml [root@node4 filebeat]# systemctl restart filebeat 使用kibana查看 访问：http://192.168.132.131:5601/app/kibana 对于一些基本的操作，再前面的学习中已经演示，参看前面的学习实验 添加日志格式 访问后 安装jq工具 [root@node4 filebeat]# yum -y install jq [root@node4 ~]# tail -f /usr/local/nginx/logs/access.log |jq . 但是读取的依然是整个的message日志 [root@node4 ~]# grep -Ev "^$|[#;]" /etc/filebeat/filebeat.yml [root@node4 ~]# systemctl restart filebeat 得到格式的日志 ES原始数据 Kibana显示 使用json格式收集nginx的日志介绍到这里 ELK学习实验014：Nginx日志JSON格式收集 标签：height mat cert rate core hat nbsp soft received 原文地址：https://www.cnblogs.com/zyxnhr/p/12210938.html1 Kibana的显示配置
2 使用压测工具产生日志
This is ApacheBench, Version 2.3 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.132.134 (be patient).....done
Server Software: nginx/1.16.1
Server Hostname: 192.168.132.134
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 100
Time taken for tests: 0.011 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 84500 bytes
HTML transferred: 61200 bytes
Requests per second: 9071.12 [#/sec] (mean)
Time per request: 11.024 [ms] (mean)
Time per request: 0.110 [ms] (mean, across all concurrent requests)
Transfer rate: 7485.44 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 3 0.5 3 3
Processing: 2 3 1.1 2 5
Waiting: 0 3 1.2 2 5
Total: 4 6 0.7 5 7
WARNING: The median and mean for the total time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 5
66% 6
75% 6
80% 7
90% 7
95% 7
98% 7
99% 7
100% 7 (longest request)
192.168.132.181 - - [18/Jan/2020:21:47:23 -0500] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"
192.168.132.181 - - [18/Jan/2020:21:47:23 -0500] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"
192.168.132.181 - - [18/Jan/2020:21:47:23 -0500] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"
3 安装filebeat
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/access.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
hosts: ["192.168.132.131:9200","192.168.132.132:9200","192.168.132.133:9200"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
4 配置nginx的日志改为json格式
log_format access_json ‘{"@timestamp":"$time_iso8601",‘
‘"host":"$server_addr",‘
‘"clientip":"$remote_addr",‘
‘"size":$body_bytes_sent,‘
‘"responsetime":$request_time,‘
‘"upstreamtime":"$upstream_response_time",‘
‘"upstreamhost":"$upstream_addr",‘
‘"http_host":"$host",‘
‘"url":"$uri",‘
‘"domain":"$host",‘
‘"xff":"$http_x_forwarded_for",‘
‘"referer":"$http_referer",‘
‘"status":"$status"}‘;
{"@timestamp":"2020-01-18T23:33:57-05:00","host":"192.168.132.134","clientip":"192.168.132.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.132.134","url":"/index.html","domain":"192.168.132.134","xff":"-","referer":"-","status":"304"}
5 filebeat解析成json格式
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/access.log
json.key_under_root: true
json.overwrite_keys: true
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
hosts: ["192.168.132.131:9200","192.168.132.132:9200","192.168.132.133:9200"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
{
"_index": "filebeat-7.5.1",
"_type": "_doc",
"_id": "oq03vG8BcnOPLK2r_2wa",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2020-01-19T05:13:02.912Z",
"json": {
"upstreamtime": "-",
"host": "192.168.132.134",
"@timestamp": "2020-01-19T00:13:01-05:00",
"Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36",
"clientip": "192.168.132.1",
"domain": "192.168.132.134",
"http_host": "192.168.132.134",
"status": "304",
"referer": "-",
"xff": "-",
"responsetime": 0,
"url": "/index.html",
"size": 0,
"upstreamhost": "-"
},
"input": {
"type": "log"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "node4",
"hostname": "node4",
"architecture": "x86_64",
"os": {
"platform": "centos",
"version": "7 (Core)",
"family": "redhat",
"name": "CentOS Linux",
"kernel": "3.10.0-957.27.2.el7.x86_64",
"codename": "Core"
},
"id": "a833bbe7e3634c75ab1b427c4049d056",
"containerized": false
},
"agent": {
"ephemeral_id": "bab448ec-726f-48bf-aa62-0b8fe39c2684",
"hostname": "node4",
"id": "3d856c84-4859-4b20-a25f-4b6c1a8a8108",
"version": "7.5.1",
"type": "filebeat"
},
"log": {
"file": {
"path": "/usr/local/nginx/logs/access.log"
},
"offset": 15601
}
}
}
