← Back to Blog

Monitor Kubernetes Cluster from Another Kubernetes Cluster with Prometheus

Prometheus, an open-source systems monitoring and alerting toolkit, is a popular choice for monitoring Kubernetes clusters. But what if you need to monitor a Kubernetes cluster from another Kubernetes cluster? This post will guide you through the process, step by step.
By Saturn Cloud | | Miscellaneous

Monitor Kubernetes Cluster from Another Kubernetes Cluster with Prometheus

Prometheus, an open-source systems monitoring and alerting toolkit, is a popular choice for monitoring Kubernetes clusters. But what if you need to monitor a Kubernetes cluster from another Kubernetes cluster? This post will guide you through the process, step by step.

Introduction

Monitoring is a crucial aspect of maintaining the health and performance of your Kubernetes clusters. With Prometheus, you can collect and visualize metrics, set up alerts, and more. This guide will show you how to set up Prometheus in one Kubernetes cluster to monitor another.

Prerequisites

Before we start, ensure you have the following:

  • Two Kubernetes clusters: one to install Prometheus (Monitoring Cluster), and another to be monitored (Target Cluster).
  • Helm installed on your Monitoring Cluster.
  • kubectl installed and configured to manage both clusters.

Step 1: Install Prometheus on the Monitoring Cluster

First, we need to install Prometheus on the Monitoring Cluster. We’ll use the stable Prometheus Helm chart for this.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus

This will install Prometheus and its components, including Alertmanager and Pushgateway.

Step 2: Configure Prometheus to Monitor the Target Cluster

Next, we need to configure Prometheus to scrape metrics from the Target Cluster. We’ll do this by creating a ServiceAccount, ClusterRole, and ClusterRoleBinding in the Target Cluster, and then adding the Target Cluster’s API server to Prometheus' scrape configs.

First, create the ServiceAccount and ClusterRoleBinding:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: default

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default
EOF

Next, get the ServiceAccount’s token and the Target Cluster’s API server address:

TOKEN=$(kubectl get secret $(kubectl get serviceaccount prometheus -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode)
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

Finally, add the Target Cluster’s API server to Prometheus' scrape configs:

cat <<EOF | helm upgrade prometheus prometheus-community/prometheus -f -
scrape_configs:
  - job_name: 'kubernetes'
    kubernetes_sd_configs:
      - api_server: ${APISERVER}
        role: node
        bearer_token: ${TOKEN}
        tls_config:
          insecure_skip_verify: true
EOF

Step 3: Verify the Setup

To verify that Prometheus is correctly scraping metrics from the Target Cluster, you can check the Targets page in the Prometheus UI. You should see the Target Cluster’s nodes listed there.

Conclusion

Monitoring Kubernetes clusters is essential for maintaining their health and performance. With Prometheus, you can easily set up monitoring for multiple clusters, even from another Kubernetes cluster. This guide showed you how to do just that. Remember to secure your setup by enabling TLS verification and using a secure token for the ServiceAccount.

References

Keywords: Kubernetes, Prometheus, Monitoring, Cluster, Helm, ServiceAccount, ClusterRole, ClusterRoleBinding, API Server, Scrape Configs, Metrics, Alertmanager, Pushgateway, Nodes, Services, Endpoints, Pods, TLS, Token

Meta Description: Learn how to monitor a Kubernetes cluster from another Kubernetes cluster using Prometheus. This guide covers installing Prometheus with Helm, configuring Prometheus to scrape metrics from the target cluster, and verifying the setup.

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.

Try Saturn Cloud Now