提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档
文章目录
- 背景
- filebeat主体配置
- filebeat.inputs部分
- filebeat.output部分
- filebeat完整配置
背景
今天收到需求，生产环境中通需要优化filebeat的输出，将不同的内容输出到kafka不同的topic中，没看过上集的兄弟可以去看上集
filebeat主体配置
这次filebeat主体部分也是参考 File beat官方文档 配置如下↓
filebeat.inputs:- type: log paths:- /data/filebeat-data/*.logprocessors:- add_fields:target: ""fields:log_type: "bizlog"# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:filebeat.autodiscover:providers:- type: kuberneteshints.enabled: truehints.default_config:type: containerpaths:- /var/log/containers/*.logprocessors:- add_fields:target: ""fields:log_type: "bizlog"#output.elasticsearch:# hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']# username: ${ELASTICSEARCH_USERNAME}# password: ${ELASTICSEARCH_PASSWORD}output.kafka:hosts: ['${KAFKA_HOST:kafka}:${KAFKA_PORT:9092}']topic: log_topic_alltopics:- topic: "bizlog-%{[agent.version]}"when.contains:log_type: "bizlog"- topic: "k8slog-%{[agent.version]}"when.contains:log_type: "k8slog"
---
filebeat.inputs部分
我们在input中添加了processors模块，它可以将自定义的标签注入到输出文档中
processors:- add_fields:target: "" # target为空可以让文档直接加入根节点fields:log_type: "bizlog" # 自定义标签
输出文本效果如下
filebeat.output部分
输出直接指向kafka接口
本例中使用了环境变量方式
输出到不同文档可以直接在topics模块中配置
output.kafka:hosts: ['${KAFKA_HOST:kafka}:${KAFKA_PORT:9092}']topic: log_topic_all # 文本没有有符合下面条件时，则输出到这个标题topics: - topic: "bizlog-%{[agent.version]}" # 指定输出topicwhen.contains:log_type: "bizlog" # input中我们注入的标签这里可以作为判断条件使用- topic: "k8slog-%{[agent.version]}"when.contains:log_type: "k8slog"
输出效果如下
完美符合需求
filebeat完整配置
本文采用daemonset方式部署了filebeat，完整的yaml文件如下↓
apiVersion: v1
data:.dockerconfigjson: ewogICJhdXRocyI6IHsKICAgICJmdmhiLmZqZWNsb3VkLmNvbSI6IHsKICAgICAgInVzZXJuYW1lIjogImFkbWluIiwKICAgICAgInBhc3N3b3JkIjogIkF0eG41WW5MWG5KS3JsVFciCiAgICB9CiAgfQp9Cg==
kind: Secret
metadata:name: harbor-loginnamespace: kube-system
type: kubernetes.io/dockerconfigjson
---
apiVersion: v1
kind: ServiceAccount
metadata:name: filebeatnamespace: kube-systemlabels:k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: filebeatlabels:k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API groupresources:- namespaces- pods- nodesverbs:- get- watch- list
- apiGroups: ["apps"]resources:- replicasetsverbs: ["get", "list", "watch"]
- apiGroups: ["batch"]resources:- jobsverbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:name: filebeat# should be the namespace where filebeat is runningnamespace: kube-systemlabels:k8s-app: filebeat
rules:- apiGroups:- coordination.k8s.ioresources:- leasesverbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:name: filebeat-kubeadm-confignamespace: kube-systemlabels:k8s-app: filebeat
rules:- apiGroups: [""]resources:- configmapsresourceNames:- kubeadm-configverbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: filebeat
subjects:
- kind: ServiceAccountname: filebeatnamespace: kube-system
roleRef:kind: ClusterRolename: filebeatapiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: filebeatnamespace: kube-system
subjects:- kind: ServiceAccountname: filebeatnamespace: kube-system
roleRef:kind: Rolename: filebeatapiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: filebeat-kubeadm-confignamespace: kube-system
subjects:- kind: ServiceAccountname: filebeatnamespace: kube-system
roleRef:kind: Rolename: filebeat-kubeadm-configapiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-confignamespace: kube-systemlabels:k8s-app: filebeat
data:filebeat.yml: |-filebeat.inputs:- type: log paths:- /data/filebeat-data/*.logprocessors:- add_fields:target: ""fields:log_type: "bizlog"# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:filebeat.autodiscover:providers:- type: kuberneteshints.enabled: truehints.default_config:type: containerpaths:- /var/log/containers/*.logprocessors:- add_fields:target: ""fields:log_type: "bizlog"#output.elasticsearch:# hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']# username: ${ELASTICSEARCH_USERNAME}# password: ${ELASTICSEARCH_PASSWORD}output.kafka:hosts: ['${KAFKA_HOST:kafka}:${KAFKA_PORT:9092}']topic: log_topic_alltopics:- topic: "bizlog-%{[agent.version]}"when.contains:log_type: "bizlog"- topic: "k8slog-%{[agent.version]}"when.contains:log_type: "k8slog"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: filebeatnamespace: kube-systemlabels:k8s-app: filebeat
spec:selector:matchLabels:k8s-app: filebeattemplate:metadata:labels:k8s-app: filebeatspec:serviceAccountName: filebeatterminationGracePeriodSeconds: 30hostNetwork: truednsPolicy: ClusterFirstWithHostNetcontainers:- name: filebeatimage: fvhb.fjecloud.com/beats/filebeat:8.9.2securityContext:runAsUser: 0args: ["-c", "/etc/filebeat.yml","-e",]env:- name: KAFKA_HOSTvalue: "100.99.17.19"- name: KAFKA_PORTvalue: "9092" - name: ELASTICSEARCH_HOSTvalue: "100.99.17.19"- name: ELASTICSEARCH_PORTvalue: "19200"- name: ELASTICSEARCH_USERNAMEvalue: elastic- name: ELASTICSEARCH_PASSWORDvalue: qianyue@2024- name: ELASTIC_CLOUD_IDvalue:- name: ELASTIC_CLOUD_AUTHvalue:- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeNamesecurityContext:runAsUser: 0# If using Red Hat OpenShift uncomment this:#privileged: trueresources:limits:memory: 200Mirequests:cpu: 100mmemory: 100MivolumeMounts:- name: configmountPath: /etc/filebeat.ymlreadOnly: truesubPath: filebeat.yml- name: datamountPath: /usr/share/filebeat/data- name: varlibdockercontainersmountPath: /data/docker/containersreadOnly: true- name: filebeat-datamountPath: /data/filebeat-data- name: varlogmountPath: /var/logreadOnly: trueimagePullSecrets:- name: harbor-loginvolumes:- name: configconfigMap:defaultMode: 0640name: filebeat-config- name: varlibdockercontainershostPath:path: /data/docker/containers- name: filebeat-datahostPath:path: /data/filebeat-datatype: DirectoryOrCreate- name: varloghostPath:path: /var/log# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart- name: datahostPath:# When filebeat runs as non-root user, this directory needs to be writable by group (g+w).path: /var/lib/filebeat-datatype: DirectoryOrCreate
---
