Spegel (Image Pull Acceleration)

Spegel is a stateless peer-to-peer OCI registry mirror that runs as a DaemonSet on every node. When a node needs to pull a layer that another node in the cluster has already pulled, Spegel serves it from the peer over the local network instead of from the upstream registry. If no peer has the layer, the pull falls through to the upstream as normal.

Saturn Cloud Enterprise can install and manage Spegel as a saturn-helm-operator component. It is enabled by default on Nebius deployments and disabled by default elsewhere. For background on what Spegel does and why we use it, see Speeding Up Kubernetes Image Pulls with Spegel.

When to enable it

Enable Spegel when:

• You scale nodes up and down frequently, and the same large images get pulled to many nodes
• Your cluster’s intra-node network is faster than its path to the upstream registry (this is the common case on a cloud-provider-managed cluster)
• You are not already using a pull-through cache or another peer-to-peer image distribution system

Leave it disabled when:

• You only ever run a fixed set of nodes that already have the images cached locally
• You already operate a pull-through cache (e.g. Harbor) and pulls already go through it
• You are running on a managed Kubernetes that does not let you configure containerd (see the prerequisites below)

Prerequisites

Spegel requires two settings in containerd’s config.toml on every node:

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

This tells containerd to read per-registry mirror configuration from /etc/containerd/certs.d, which is where Spegel writes its mirror config at runtime.

discard_unpacked_layers = false

This keeps layer blobs in the content store after they are unpacked. With discard_unpacked_layers = true (the default on some distributions), Spegel has no layers left to serve to peers.

Saturn Cloud’s Spegel component ships with a containerd-prep DaemonSet that patches both settings and restarts containerd on each node. The patch is idempotent: containerd is only restarted when the config actually changed. If your nodes already have the correct containerd configuration (some managed Kubernetes setups do this for you), you can disable the prep step (see below).

Spegel does not need any other dependencies. It does not run a control plane, does not need persistent storage, and does not interact with the rest of the cluster outside of writing files to /etc/containerd/certs.d and serving HTTP on the host network.

Enabling it

In the saturn-helm-operator values, set:

saturnComponents:
  spegel:
    enabled: true
    values:
      # Registries to mirror. Spegel only writes containerd config for these.
      # Leaving the list empty would tell Spegel to mirror every registry,
      # which is rarely what you want. List the registry hosts your cluster
      # actually pulls images from in volume.
      mirroredRegistries:
      - https://<registry-you-pull-from>
      - https://<another-registry-you-pull-from>
      # Patch containerd on each node so Spegel can read/write mirror config.
      # Set enabled: false if your nodes already have config_path set and
      # discard_unpacked_layers disabled.
      containerdPrep:
        enabled: true

Apply the operator chart as usual. The operator will create:

• A spegel namespace
• A Spegel CR (charts.saturncloud.io/v1alpha1) which the operator reconciles into a Helm release of the bundled Spegel chart
• The Spegel DaemonSet on every node
• The containerd-prep DaemonSet (if enabled) on every node

Configurable values

Anything under values: is passed through to the upstream Spegel chart, so see the chart’s values.yaml for the full list. Common overrides:

• values.serviceMonitor.enabled: true to expose Prometheus metrics if you run kube-prometheus-stack.
• values.spegel.registryFilters: ["^registry\\.k8s\\.io/"] to exclude registries Spegel should not touch.

The Spegel image itself is set centrally in the operator’s images.spegel field and is automatically rewritten through imageMirror for air-gapped deployments. You should not need to override it in the Spegel CR.

Verifying

After enabling Spegel and waiting for both DaemonSets to roll out:

kubectl -n spegel get ds

You should see spegel and (if enabled) spegel-containerd-prep, both with the desired number of pods ready.

Check that containerd is reading the mirror config. On a node:

ls /etc/containerd/certs.d/

You should see a directory per mirrored registry, each containing a hosts.toml that points the registry’s host at localhost:5000 (Spegel’s local port) with the upstream as fallback.

Spegel exposes a metrics endpoint and a debug web page on each pod. Port-forward to see them:

kubectl -n spegel port-forward ds/spegel 9090:9090
# Then visit http://localhost:9090/metrics

Useful metrics:

• spegel_mirror_requests_total{cache_type="hit"}: pulls served from a peer
• spegel_mirror_requests_total{cache_type="miss"}: pulls that fell through to the upstream
• spegel_advertised_images: number of images this node is advertising to peers

A healthy Spegel deployment shows the hit count climbing as nodes pull the same images.

Disabling

Set saturnComponents.spegel.enabled: false in the operator values and re-apply. The operator uninstalls the Helm release, which removes both DaemonSets, the Spegel namespace, and the mirror configuration Spegel had written to /etc/containerd/certs.d.

The containerd-prep DaemonSet is also removed, but it does not roll back the config_path setting it added to config.toml. Leaving config_path set is harmless (containerd just finds no mirror entries) and means you can re-enable Spegel later without another containerd restart. If you want to fully undo the change, edit config.toml and restart containerd manually.

After Spegel is gone, all image pulls fall through to the upstream registry exactly as they did before Spegel was installed.

Troubleshooting

Spegel pods running, but pulls still take the same time as before. Check /etc/containerd/certs.d on a node. If it is empty or missing, containerd does not have config_path set, and Spegel’s mirror config is being ignored. Either enable containerdPrep or set config_path via your node-provisioning mechanism.

Spegel metrics show miss but never hit. Check discard_unpacked_layers on the node (grep discard_unpacked /etc/containerd/config.toml). If it is true, containerd is throwing layers away after unpacking, so peers have nothing to serve.

ImagePullBackOff on gcr.io/google-containers/pause. This registry path was retired by Google in March 2025. If you copied the containerd-prep manifest from an older example that hardcodes it, switch to the Saturn-managed component (which uses saturn-k8s-utils for the keepalive) or update the manifest to use registry.k8s.io/pause.