← Back to Blog

How to Mount Volume of Airflow Worker to Airflow Kubernetes Pod Operator

In the world of data engineering, Apache Airflow has emerged as a de facto tool for orchestrating complex data pipelines. With the advent of Kubernetes, the ability to scale and manage applications has become significantly easier. In this blog post, we will delve into the process of mounting a volume of an Airflow worker to an Airflow Kubernetes Pod Operator. This guide is optimized for data scientists and engineers who are familiar with Kubernetes and Airflow.
By Saturn Cloud | | Miscellaneous

How to Mount Volume of Airflow Worker to Airflow Kubernetes Pod Operator

In the world of data engineering, Apache Airflow has emerged as a de facto tool for orchestrating complex data pipelines. With the advent of Kubernetes, the ability to scale and manage applications has become significantly easier. In this blog post, we will delve into the process of mounting a volume of an Airflow worker to an Airflow Kubernetes Pod Operator. This guide is optimized for data scientists and engineers who are familiar with Kubernetes and Airflow.

Prerequisites

Before we begin, ensure you have the following:

  • A working Kubernetes cluster
  • Helm installed on your local machine
  • Apache Airflow installed in your Kubernetes cluster

Step 1: Define Persistent Volume and Persistent Volume Claim

First, we need to define a Persistent Volume (PV) and a Persistent Volume Claim (PVC). The PV is a piece of storage in the cluster that has been provisioned by an administrator. The PVC is a request for storage by a user.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: airflow-worker-volume
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  hostPath:
    path: "/data/airflow-worker"
---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: airflow-worker-volume-claim
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Save this as airflow-worker-pv-pvc.yaml and apply it to your Kubernetes cluster using kubectl apply -f airflow-worker-pv-pvc.yaml.

Step 2: Configure Airflow KubernetesPodOperator

Next, we need to configure the Airflow KubernetesPodOperator to use the PVC. This operator allows you to run Kubernetes pods as tasks in your Airflow DAGs.

from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator

KubernetesPodOperator(
    namespace='default',
    image=""my-image"",
    cmds=[""python"",""-c""],
    arguments=[""print('hello world')""],
    labels={""foo"": ""bar""},
    name=""airflow-test-pod"",
    in_cluster=True,
    task_id=""task"",
    get_logs=True,
    volumes=[PersistentVolumeClaim('airflow-worker-volume-claim')],
    volume_mounts=[VolumeMount('airflow-worker-volume', mount_path='/data')],
    is_delete_operator_pod=True,
    dag=dag
)

In this configuration, we are creating a KubernetesPodOperator task that runs a pod using the image ““my-image””. The pod prints ““hello world”” to the console. The important part here is the volumes and volume_mounts parameters. We are passing the PVC we created earlier to the volumes parameter and specifying a mount path for the volume.

Step 3: Test Your Configuration

Now that we have our Airflow worker volume mounted to our KubernetesPodOperator, we can test our configuration. Run your Airflow DAG and check the logs of the pod. You should see the output of the print('hello world') command.

Conclusion

Mounting a volume of an Airflow worker to an Airflow Kubernetes Pod Operator is a powerful way to share data between your Airflow tasks and your Kubernetes pods. This guide has shown you how to define a Persistent Volume and Persistent Volume Claim, configure the Airflow KubernetesPodOperator to use the PVC, and test your configuration.

Remember, while this guide provides a basic example, the real power comes from leveraging this setup in complex data pipelines. You can store intermediate data, share data between tasks, and even use the volume for caching purposes.

In the ever-evolving landscape of data engineering, mastering tools like Apache Airflow and Kubernetes is essential. Stay tuned for more guides and tutorials to help you navigate this exciting field.

Keywords: Apache Airflow, Kubernetes, KubernetesPodOperator, Persistent Volume, Persistent Volume Claim, Data Engineering, Data Pipelines, Data Science, Mount Volume, Airflow Worker

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.

Try Saturn Cloud Now