Backup and Recovery

flex.plane's stateless architecture means there is surprisingly little to back up. The orchestrator has no database. If it crashes and restarts, it reconstructs everything from the systems around it. But there are a few components that do hold state and need protection.

Understand what needs backing up

The orchestrator is stateless by design. It derives all state at request time from:

• Proxmox: VMs, networks, storage, node status
• Kubernetes: cluster state, ConfigMaps (profiles, VDC mappings)
• Headscale: mesh network membership, node keys
• Zitadel: users, roles, VDC memberships

If you lose the orchestrator pod, just restart it. There is nothing to restore.

The components that do hold state:

Back up identity

Zitadel's PostgreSQL database is the most critical piece to back up. It contains all user accounts, roles, and OIDC configuration.

Automated PostgreSQL backups

Set up a CronJob or use your managed database provider's backup feature:

# Manual backup
kubectl exec -n flex-identity postgresql-0 -- \
  pg_dump -U zitadel zitadel > zitadel-backup-$(date +%Y%m%d).sql

For production, use a continuous backup solution like:

• Managed database (RDS, Cloud SQL, etc.) with automated daily backups and point-in-time recovery.
• pgBackRest or Barman for self-managed PostgreSQL.
• Velero to back up the PostgreSQL PVC at the Kubernetes level.

# Back up the Zitadel master key secret
kubectl get secret zitadel -n flex-identity -o yaml > zitadel-masterkey-backup.yaml

Back up mesh state

Headscale stores its state in a SQLite database on the orchestrator's persistent volume. This contains node registrations, WireGuard keys, and mesh topology.

The orchestrator's persistence volume holds this data:

orchestrator:
  persistence:
    enabled: true
    size: 10Gi

Back up options:

• Velero PVC snapshots on a schedule.
• Volume snapshots if your storage class supports CSI snapshots.
• File-level backup by copying the SQLite database from the PVC.

If you lose the Headscale state, all agents need to re-enroll with new auth keys. The nodes and their VMs are unaffected. Only the mesh connectivity between nodes and the orchestrator needs to be re-established.

Back up Kubernetes ConfigMaps

Compute profiles, storage profiles, VDC-to-zone mappings, and the image catalog are stored in Kubernetes ConfigMaps in the tenant namespace.

# Back up all ConfigMaps in the tenant namespace
kubectl get configmaps -n <tenant-namespace> -o yaml > configmaps-backup.yaml

Include this in your regular Kubernetes backup strategy (Velero, etcd snapshots, etc.).

Recovery procedure

Scenario: Orchestrator pod crashes

Impact: API is temporarily unavailable. VMs and workloads are unaffected.

Recovery: Kubernetes restarts the pod automatically. No action needed. The orchestrator reconnects to Headscale and starts serving requests within seconds.

Scenario: Headscale state lost

Impact: Agents cannot communicate with the orchestrator via the mesh network.

Recovery:

1. Restore the Headscale SQLite database from backup to the PVC.
2. Restart the orchestrator pod.
3. If no backup is available, generate new auth keys and reconfigure each agent.

Scenario: Zitadel database lost

Impact: No one can authenticate. This is the most severe failure.

Recovery:

1. Restore the PostgreSQL database from backup.
2. Ensure the Zitadel master key secret is present.
3. Restart the Zitadel pods.
4. Verify login works through the portal.

Scenario: Kubernetes cluster lost

Impact: All flex.plane management components are down. VMs continue running on Proxmox but cannot be managed.

Recovery:

1. Stand up a new Kubernetes cluster.
2. Restore secrets (Zitadel master key, PostgreSQL passwords, registry credentials).
3. Restore the PostgreSQL database.
4. Redeploy all Helm charts with the same values.
5. Restore the Headscale PVC or re-enroll agents.
6. Restore ConfigMaps or recreate profiles manually.