轻量型日志采集器 Filebeat基本使用

tar -zxvf filebeat-7.17.0-linux-x86_64.tar.gz

在这里插入代码片

chown root filebeat.yml
chown root modules.d/nginx.yml 
./filebeat -e

./filebeat -e -c filebeat.yml -d "publish"

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  fields:
    name: ccccccccc
  fields_under_root: true
output.console:
  pretty: true

./filebeat -e -c filebeat_log.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  tags: ["nginx"]
setup.template.settings:
  index.number_of_shards: 1
output.elasticsearch:
  hosts: ["118.195.175.213:9200"]
  username: "elastic"
  password: "EtWvHc8lGh05YS7vS1mq"

./filebeat -e -c filebeat_nginx.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  tags: ["nginx"]
    #keys_under_root可以让字段位于根节点，默认为false
  json.keys_under_root: true
  #对于同名的key，覆盖原有key值
  json.overwrite_keys: true
  #将解析错误的消息记录储存在error.message字段中
  json.add_error_key: true
setup.template.settings:
  index.number_of_shards: 1
output.elasticsearch:
  hosts: ["118.19*.17*.213:9200"]
  username: "elastic"
  password: "EtWvHc8lGh05YS7vS1mq"

## 查看module集合
./filebeat modules list
## 启用module
./filebeat modules enable module名称
## 禁用module
./filebeat modules disable module名称

./filebeat modules enable nginx

./filebeat modules list

- module: nginx
  access:
    enabled: true
    var.paths: ["/usr/local/nginx/logs/access.log"]

  error:
    enabled: true
    var.paths: ["/usr/local/nginx/logs/error.log"]

filebeat.inputs:
setup.template.settings:
  index.number_of_shards: 3
output.elasticsearch:
  hosts: ["http://118.195.175.213:9200"]
  username: "elastic"
  password: "EtWvHc8lGh05YS7vS1mq"
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

./filebeat -e -c filebeat_nginx_module.yml

./filebeat -e -c filebeat_nginx_module.yml -d 'publish'

<h2><a id="1__0"></a>1 介绍</h2> <p>Filebeat是一个日志文件托运工具，安装客户端后，filebeat会监控指令日志，<br /> 下载地址：<br /> <a href="https://www.elastic.co/cn/beats/filebeat" target="_blank">https://www.elastic.co/cn/beats/filebeat</a></p> <p><a href="https://www.elastic.co/cn/downloads/past-releases/filebeat-7-17-0" target="_blank">https://www.elastic.co/cn/downloads/past-releases/filebeat-7-17-0</a></p> <p>注意：版本需要和 Elasticearch 版本相同</p> <h2><a id="2__10"></a>2 安装</h2> <h3><a id="21__11"></a>2.1 下载安装</h3> <pre><code class="lang-powershell">tar -zxvf filebeat-7.17.0-linux-x86_64.tar.gz </code></pre> <h2><a id="3__15"></a>3 收集日志配置</h2> <p>安装目录下新建<code>conf/star_module_log.yml</code></p> <pre><code class="lang-powershell">在这里插入代码片 </code></pre> <h2><a id="4__23"></a>4 启动</h2> <pre><code class="lang-powershell">chown root filebeat.yml chown root modules.d/nginx.yml ./filebeat -e </code></pre> <p>./filebeat -e -c filebeat_log.yml</p> <pre><code class="lang-powershell">./filebeat -e -c filebeat.yml -d &quot;publish&quot; </code></pre> <h3><a id="5__39"></a>5 举例</h3> <h3><a id="51_Filebeat_40"></a>5.1 Filebeat收集日志并输出到控制台</h3> <p><code>&quot;name&quot;: &quot;ccccccccc&quot;</code>：自定义name字段<br /> <code>&quot;message&quot;: &quot;你好&quot;</code> ：日志中输出：你好<br /> <code>fields_under_root: true</code>：将name定义为顶级字段</p> <pre><code class="lang-powershell">filebeat.inputs: - type: log enabled: true paths: - /usr/local/nginx/logs/access.log fields: name: ccccccccc fields_under_root: true output.console: pretty: true </code></pre> <pre><code class="lang-powershell">./filebeat -e -c filebeat_log.yml </code></pre> <h3><a id="52_FilebeatNginxes_60"></a>5.2 Filebeat收集Nginx运行日志并输出到es</h3> <p><code>filebeat_nginx.yml</code></p> <pre><code class="lang-powershell">filebeat.inputs: - type: log enabled: true paths: - /usr/local/nginx/logs/access.log tags: [&quot;nginx&quot;] setup.template.settings: index.number_of_shards: 1 output.elasticsearch: hosts: [&quot;118.195.175.213:9200&quot;] username: &quot;elastic&quot; password: &quot;EtWvHc8lGh05YS7vS1mq&quot; </code></pre> <pre><code class="lang-powershell">./filebeat -e -c filebeat_nginx.yml </code></pre> <p>查看es,默认会按照filebeat-版本号-日期格式生成一个索引：</p> <ul> <li>nginx日志数据在字段<mark>message</mark>中</li> </ul> <p><img src="https://static.couragesteak.com/article/4c0e85b7f5665ccf488fa2c928423f1f.png" alt="image.png" /></p> <p>到这里，仍然属于原始数据</p> <p><code>filebeat_nginx.yml</code></p> <pre><code class="lang-powershell">filebeat.inputs: - type: log enabled: true paths: - /usr/local/nginx/logs/access.log tags: [&quot;nginx&quot;] #keys_under_root可以让字段位于根节点，默认为false json.keys_under_root: true #对于同名的key，覆盖原有key值 json.overwrite_keys: true #将解析错误的消息记录储存在error.message字段中 json.add_error_key: true setup.template.settings: index.number_of_shards: 1 output.elasticsearch: hosts: [&quot;118.19*.17*.213:9200&quot;] username: &quot;elastic&quot; password: &quot;EtWvHc8lGh05YS7vS1mq&quot; </code></pre> <h3><a id="53_Nginx_moduleFilebeatNginxes_111"></a>5.3 基于Nginx module使用Filebeat收集Nginx运行日志并输出到es</h3> <pre><code class="lang-powershell">## 查看module集合 ./filebeat modules list ## 启用module ./filebeat modules enable module名称 ## 禁用module ./filebeat modules disable module名称 </code></pre> <p>启动nginx module</p> <pre><code class="lang-powershell">./filebeat modules enable nginx </code></pre> <p>查看module列表</p> <pre><code class="lang-powershell">./filebeat modules list </code></pre> <p>配置Nginx Module</p> <p><code> ${FILEBEAT_HOME}/modules.d/</code> <code>nginx.yml</code></p> <pre><code class="lang-powershell">- module: nginx access: enabled: true var.paths: [&quot;/usr/local/nginx/logs/access.log&quot;] error: enabled: true var.paths: [&quot;/usr/local/nginx/logs/error.log&quot;] </code></pre> <p>配置filebeat：<code>filebeat_nginx_module.yml</code></p> <pre><code class="lang-powershell">filebeat.inputs: setup.template.settings: index.number_of_shards: 3 output.elasticsearch: hosts: [&quot;http://118.195.175.213:9200&quot;] username: &quot;elastic&quot; password: &quot;EtWvHc8lGh05YS7vS1mq&quot; filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false </code></pre> <p>启动filebeat</p> <pre><code class="lang-powershell">./filebeat -e -c filebeat_nginx_module.yml </code></pre> <pre><code class="lang-powershell">./filebeat -e -c filebeat_nginx_module.yml -d 'publish' </code></pre> <p>到此，可以查看数据索引</p> <p>参考地址：<br /> https://blog.csdn.net/zyxwvuuvwxyz/article/details/108831962</p>