Intro

This article continues on the last one about the Logstash and describes the Filebeat as Log scraping agent for your Kubernetes cluster.

Our Elasticsearch and Kibana are managed outside of the Kubernetes cluster (AWS Elasticsearch Service). So we can go straight to the configs.

Filebeat

Consult official proposal to the configuration as as well our setup is derived from that but has some differences.

OSS Build

Here we use Open Distro compatible docker.elastic.co/beats/filebeat-oss:7.7.1 image. I recommend to use this one to avoid unexpected problems with AWS Elasticsearch Service.

Check all OSS Filebeat releases.

DaemonSet

Let’s start with a DaemonSet resource. A DaemonSet makes sure that Filebeat runs on every Kubernetes Node. This is of course inevitable to get entire logs from the every service running one every cluster node.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat-oss:7.7.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: prospectors
          mountPath: /usr/share/filebeat/prospectors.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: prospectors
        configMap:
          defaultMode: 0600
          name: filebeat-prospectors
      - name: data
        emptyDir: {}

that’s not much more. Of course there are mounted ConfigMaps and here they are:

ConfigMaps

First of all, the general Filebeat Settings need to know where Logstash is running. In previous article we exposed Logstash as: logstash-service:5044 to the cluster, this is what goes under output.logstash:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    output.logstash:
      hosts: ['logstash-service:5044']

Further we need to say a little bit more about our environment, which is in fact Kuberntes cluster full of Docker containers. That’s done by so called prospector configurations :

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:|^\s*$'
      multiline.negate: false
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true

Also here for typ Docker we we have a configuration for handling multiline Logs.. It’s hard to bring such stuff to perfection. If interested consult multiline configuration docu for more details.

So we are nearly done here, except that Filebeat need to have some more rights as usual container. So the following ServiceAccount, ClusterRole,ClusterRoleBinding resources should do the job.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

Evaluation

This works very well several month for us an do not require any maintenance so far. Only the multilines are still ugly sometimes. Feel free to provide any feedback or ask for details.