Deploying Filebeat as a Kubernetes Daemonset for Multi-line Log Ingestion into Elasticsearch

Deploying Filebeat as a Kubernetes Daemonset for Multi-line Log Ingestion into Elasticsearch

As data scientists, we often find ourselves dealing with a vast amount of data, including logs. These logs can be single-line or multi-line, and managing them efficiently is crucial for our work. In this blog post, we’ll explore how to ingest multi-line logs into Elasticsearch (ES) using Filebeat deployed as a Kubernetes Daemonset.

What is Filebeat?

Filebeat is a lightweight, open-source log shipper from Elastic that forwards and centralizes log data. It’s part of the Elastic Stack (formerly ELK Stack) — a set of open-source tools including Elasticsearch, Logstash, and Kibana, designed to help users take data from any type of source and visualize it in a meaningful way.

Why Use Filebeat with Kubernetes?

Kubernetes, an open-source platform for managing containerized workloads, produces a significant amount of log data. Filebeat, when deployed as a Kubernetes Daemonset, can efficiently collect and ship these logs to Elasticsearch. This setup allows for centralized logging, making it easier to monitor the health and performance of your Kubernetes clusters.

Setting Up Filebeat as a Kubernetes Daemonset

Let’s dive into the steps required to set up Filebeat as a Kubernetes Daemonset for multi-line log ingestion into Elasticsearch.

Prerequisites

• A running Kubernetes cluster
• Elasticsearch and Kibana set up and running
• Helm installed on your Kubernetes cluster

Step 1: Add Elastic Helm Charts

First, we need to add the Elastic Helm charts that contain Filebeat:

helm repo add elastic https://helm.elastic.co

Step 2: Configure Filebeat

Next, we need to configure Filebeat to handle multi-line logs. Create a filebeat-values.yaml file and add the following configuration:

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths: 
        - /var/log/containers/*.log
      multiline.pattern: '^[[:space:]]'
      multiline.negate: false
      multiline.match: after    

This configuration tells Filebeat to treat lines starting with a space as part of the previous line, which is a common pattern in multi-line logs.

Step 3: Deploy Filebeat

Now we’re ready to deploy Filebeat using Helm:

helm install filebeat elastic/filebeat -f filebeat-values.yaml

Step 4: Verify the Deployment

To verify that Filebeat is running correctly, use the following command:

kubectl get pods -l app=filebeat

You should see a list of Filebeat pods running on each node of your Kubernetes cluster.

Viewing Logs in Kibana

Once Filebeat is up and running, it will start shipping logs to Elasticsearch. You can then use Kibana to visualize and analyze the logs.

Conclusion

Deploying Filebeat as a Kubernetes Daemonset is an efficient way to handle multi-line log ingestion into Elasticsearch. It provides a robust solution for managing the vast amount of log data produced by Kubernetes, enabling data scientists to focus on extracting valuable insights from the data.

Remember, this is just a basic setup. Depending on your specific needs, you might need to adjust the Filebeat configuration. For example, you might want to add more complex multi-line patterns or filter out specific types of logs.

Keywords

• Filebeat
• Kubernetes
• Daemonset
• Elasticsearch
• Multi-line logs
• Log ingestion
• Elastic Stack
• Kibana
• Helm
• Centralized logging
• Containerized workloads
• Log data
• Log shipper
• Data scientists
• Kubernetes clusters
• Helm charts
• Filebeat configuration
• Log visualization
• Log analysis
• Log patterns
• Log filtering

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.