Backup and recovery

A complete recovery needs both the leader data and its matching sealing key, which decrypts stored secrets. Keep separate, protected copies of both.

Take a consistent offline backup

Stop the leader before copying its files so the backup is consistent. Schedule a maintenance window. Workers running in separate processes normally keep running their current configuration, but workers configured to stop when the leader is unreachable may stop. Browser sessions end. In standalone mode, stopping Smeltr also stops its worker and data processing.

Complete this checklist in order:

  • Record the installed version, checksum or container-image ID, configuration file, environment settings, and service account. Keep the matching executable or container image for recovery.
  • Stop the leader through its service manager or gracefully terminate the foreground process. Confirm it has exited before copying files.
  • Copy the entire leader.storage.data_dir: leader.db, any accompanying SQLite files, repos/ and tls/. Do not select just the database or just Git. Each start that finds leader.db sound also keeps a copy beside it as leader.db.previous.
  • Back up the exact leader.secrets.key separately under restricted access. Label it so you can match it to the data backup. Also keep the service configuration and any private certificate authority or proxy configuration needed to restore access.
  • Encrypt and restrict both backups, preserve ownership and permissions, and verify checksums. Restart the original leader and verify login and heartbeats.

The key lives outside the data directory by default. Keep it separate so a stolen data backup does not also provide the key to decrypt stored secrets. TLS private keys inside the data directory also make that backup sensitive.

Restore without creating two leaders

  1. Stop or isolate the original leader and prevent its automatic restart.
  2. On the replacement host, install the same binary version or container-image ID first. Restore the complete data directory and matching sealing key to the configured paths, preserving restrictive permissions.
  3. Restore the configuration and verify DNS, public origin, ports and certificate names. Do not initialize an empty leader over the recovery target.
  4. Start it as the original service account. Sign in again; a backup holds no sessions. If you lose account access, use administrator recovery.
  5. Check Git history, worker groups, certificate status, and any errors on the Secrets page. Confirm workers reconnect and compare each worker's reported Commit with the restored main commit.

A restored Git history may be older than the fleet's cached configuration. Review that difference before allowing production workers to fetch from it.

Test a restore safely

Restore into a network-isolated environment with no production workers and no access to production sources or destinations. Use disposable worker state and non-production credentials. Check login, history, secret usability, trust and a sample deployment, then record the measured recovery time. A restored leader is not passive: do not expose a test restore to the production fleet.

Availability and worker state

A standalone worker keeps its state in worker/ beneath the leader data directory. Include that directory in the stopped-process backup. An isolated restore also starts that worker when worker.enabled is true, so block access to production data systems before starting it.

Run one leader. If it is lost, restore a backup using the procedure above; Smeltr does not currently switch automatically to another leader. Preserve worker state across upgrades, including its identity, credential, trust files and local configuration repository. Never start two workers from the same copied identity. A rebuilt host should enroll afresh.