Bootstrap an Air-Gapped Cluster

Bootstrap a Cozystack cluster in an isolated (air-gapped) environment with container registry mirrors.

Introduction

This guide outlines the steps to bootstrap a Cozystack cluster in an air-gapped environment.

Air-gapped installation means that the cluster has no direct access to the Internet. All necessary resources, such as images and metadata, must be available on the private network.

Configuring Talos Nodes

1. Configure NTP Servers

Accurate time synchronization is critical for the cluster. In your Talos machine configuration, set local NTP servers that are accessible inside your private network:

machine:
  time:
    servers:
      # example values
      - 192.168.0.4
      - 10.10.0.5

Ensure that these NTP servers are reachable from the first Talos node.

2. Configure Container Registry Mirrors

Since the cluster cannot access public container registries, it needs to use their local mirrors. Creating such mirrors is out of the scope of this guide.

Update your machine configuration in the following way, providing the IP addresses and ports of your local mirrors for each registry:

machine:
  registries:
    mirrors:
      docker.io:
        endpoints:
          - http://10.0.0.1:8082
      ghcr.io:
        endpoints:
          - http://10.0.0.1:8083
      gcr.io:
        endpoints:
          - http://10.0.0.1:8084
      registry.k8s.io:
        endpoints:
          - http://10.0.0.1:8085
      quay.io:
        endpoints:
          - http://10.0.0.1:8086
      cr.fluentbit.io:
        endpoints:
          - http://10.0.0.1:8087
      docker-registry3.mariadb.com:
        endpoints:
          - http://10.0.0.1:8088
    config:
      "10.0.0.1:8082":
        tls:
          insecureSkipVerify: true
        auth:
          username: myuser
          password: mypass

Of course, the values for config.[0].auth.* are given as examples, and you need to use real credentials. Make sure your local registry proxies mirror all required images for Talos and Kubernetes components.

3. Add CA Certificate

To use a private Certificate Authority, you need to add its certificate to the nodes.

# talm: nodes=["10.10.10.10"], endpoints=["10.10.10.10"], templates=["templates/controlplane.yaml"]
# THIS FILE IS AUTOGENERATED. PREFER TEMPLATE EDITS OVER MANUAL ONES.
machine:
# ...
# ...
  discovery:
    enabled: false
  etcd:
    advertisedSubnets:
      - 10.4.100.10/24
  allowSchedulingOnControlPlanes: true
---
apiVersion: v1alpha1
kind: TrustedRootsConfig
name: my-enterprise-ca
certificates: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

4. Apply Changes

After you have made the changes above, you can apply the configuration and bootstrap a cluster:

Using Talm

Rebuild the node configuration files and apply them to each node:

talm template -e <ip> -n <ip> -t templates/controlplane.yaml -i > nodes/node1.yaml
talm apply -f nodes/node1.yaml
# repeat for each node

Finally, bootstrap the cluster as usual:

talm bootstrap -f nodes/node1.yaml

Read the Talm configuration guide to learn more.

Using talosctl

Apply the configuration to each node:

talosctl apply -f controlplane.yaml -n <ip> -e <ip> -i

Finally, bootstrap the cluster using one of the nodes:

talosctl bootstrap -n <ip> -e <ip>

Read the talosctl configuration guide to learn more.

5. Configure Container Registry Mirrors for Tenant Kubernetes

Tenant Kubernetes clusters in Cozystack use Kamaji for the control plane. The control plane components run as pods on the management cluster nodes, so they automatically use the registry mirrors configured in step 2 for Talos.

However, tenant worker nodes run as separate virtual machines with their own containerd instance. These worker nodes need a separate registry mirror configuration.

To perform this configuration, you first need to deploy a Cozystack cluster of (or upgrade your cluster to) version v0.32.0 or later. Check your current cluster version with:

kubectl get deploy -n cozy-system cozystack -oyaml | grep installer

Option A: Configure via platform package

The platform package can automatically generate the patch-containerd secret from the registries section in the platform values.

Add the registries section to your cozystack-platform.yaml:

apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
  name: cozystack.cozystack-platform
spec:
  variant: isp-full
  components:
    platform:
      values:
        # ... your existing publishing, networking, etc. ...
        registries:
          mirrors:
            docker.io:
              endpoints:
                - http://10.0.0.1:8082
            ghcr.io:
              endpoints:
                - http://10.0.0.1:8083
            gcr.io:
              endpoints:
                - http://10.0.0.1:8084
            registry.k8s.io:
              endpoints:
                - http://10.0.0.1:8085
            quay.io:
              endpoints:
                - http://10.0.0.1:8086
            cr.fluentbit.io:
              endpoints:
                - http://10.0.0.1:8087
            docker-registry3.mariadb.com:
              endpoints:
                - http://10.0.0.1:8088
          config:
            "10.0.0.1:8082":
              tls:
                insecureSkipVerify: true
              auth:
                username: myuser
                password: mypass

Then apply it:

kubectl apply -f cozystack-platform.yaml

This will create a patch-containerd secret in the cozy-system namespace, which is automatically copied to every tenant Kubernetes cluster.

Alternatively, patch an existing platform package

If the platform package is already deployed, you can add registry mirrors with a patch:

kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{
  "spec": {
    "components": {
      "platform": {
        "values": {
          "registries": {
            "mirrors": {
              "docker.io": {
                "endpoints": ["http://10.0.0.1:8082"]
              },
              "ghcr.io": {
                "endpoints": ["http://10.0.0.1:8083"]
              },
              "gcr.io": {
                "endpoints": ["http://10.0.0.1:8084"]
              },
              "registry.k8s.io": {
                "endpoints": ["http://10.0.0.1:8085"]
              },
              "quay.io": {
                "endpoints": ["http://10.0.0.1:8086"]
              },
              "cr.fluentbit.io": {
                "endpoints": ["http://10.0.0.1:8087"]
              },
              "docker-registry3.mariadb.com": {
                "endpoints": ["http://10.0.0.1:8088"]
              }
            },
            "config": {
              "10.0.0.1:8082": {
                "tls": {
                  "insecureSkipVerify": true
                },
                "auth": {
                  "username": "myuser",
                  "password": "mypass"
                }
              }
            }
          }
        }
      }
    }
  }
}'

Option B: Create the secret manually

Alternatively, create a Kubernetes Secret named patch-containerd directly:

apiVersion: v1
kind: Secret
metadata:
  name: patch-containerd
  namespace: cozy-system
type: Opaque
stringData:
  docker.io.toml: |
    server = "https://registry-1.docker.io"
    [host."http://10.0.0.1:8082"]
      capabilities = ["pull", "resolve"]
      skip_verify = true
  ghcr.io.toml: |
    server = "https://ghcr.io"
    [host."http://10.0.0.1:8083"]
      capabilities = ["pull", "resolve"]
      skip_verify = true
  gcr.io.toml: |
    server = "https://gcr.io"
    [host."http://10.0.0.1:8084"]
      capabilities = ["pull", "resolve"]
      skip_verify = true
  registry.k8s.io.toml: |
    server = "https://registry.k8s.io"
    [host."http://10.0.0.1:8085"]
      capabilities = ["pull", "resolve"]
      skip_verify = true
  quay.io.toml: |
    server = "https://quay.io"
    [host."http://10.0.0.1:8086"]
      capabilities = ["pull", "resolve"]
      skip_verify = true
  cr.fluentbit.io.toml: |
    server = "https://cr.fluentbit.io"
    [host."http://10.0.0.1:8087"]
      capabilities = ["pull", "resolve"]
      skip_verify = true
  docker-registry3.mariadb.com.toml: |
    server = "https://docker-registry3.mariadb.com"
    [host."http://10.0.0.1:8088"]
      capabilities = ["pull", "resolve"]
      skip_verify = true

If your registry mirrors require authentication, add a custom Authorization header with Base64-encoded credentials:

server = "https://registry-1.docker.io"
[host."http://10.0.0.1:8082"]
  capabilities = ["pull", "resolve"]
  skip_verify = true
  [host."http://10.0.0.1:8082".header]
    Authorization = "Basic bXl1c2VyOm15cGFzcw=="

To generate the Base64-encoded value, run:

echo -n 'myuser:mypass' | base64

For dynamic or token-based authentication (e.g., Docker Hub), use Kubernetes image pull secrets instead of plaintext credentials.

How it works

The patch-containerd secret from the cozy-system namespace is automatically copied to every tenant Kubernetes cluster namespace during deployment. The secret data is mounted into worker node VMs as containerd registry configuration files at /etc/containerd/certs.d/<registry>/hosts.toml.

Per-cluster configuration

It is possible to configure registry mirrors for a particular tenant Kubernetes cluster instead of using the global patch-containerd secret:

  • The tenant cluster must be deployed with a Kubernetes package version 0.23.1 or later, which is available since Cozystack 0.32.1.
  • Before deploying the tenant cluster, create a Kubernetes Secret named kubernetes-<cluster-name>-patch-containerd in the tenant cluster namespace, using the same format as the examples above.

To learn more about registry configuration values, read the CRI Plugin configuration guide

Last modified 2026-03-26: docs(air-gapped): address review comments for registry mirrors (dc96ecf)