Sendense Documentation

Appliance Disaster Recovery

How the SHA protects itself: the recovery secret, automatic encrypted recovery bundles delivered into your backup repositories, and estate-wide replication concurrency limits. Distinct from workload DR failover.

Documents Home

Operations

Appliance Disaster Recovery

How the SHA protects itself: the recovery secret, automatic encrypted recovery bundles delivered into your backup repositories, and estate-wide replication concurrency limits. Distinct from workload DR failover.

ReadyCurrentdisaster-recoveryrecovery-bundlesrecovery-secretreplication

What This Page Protects

Settings, Disaster Recovery protects the SHA itself - not your workloads. Workload disaster recovery (replication patterns, failover, Sendense Controllers) is documented under Recovery & DR; this page is about making sure the appliance that orchestrates all of that can be rebuilt if it is lost.

It owns two estate-wide concerns: recovery bundles - automatic, encrypted daily snapshots of the SHA's configuration and its metadata database, delivered into every eligible backup repository so a destroyed appliance can be rebuilt from any surviving copy - and the replication concurrency limits that cap how many DR syncs run in parallel.

The Recovery Secret

The recovery secret is the encryption passphrase for every recovery bundle. It is created during first-time appliance setup and shown once. Until it is configured, scheduled bundle automation is disabled entirely and manual bundles fail - the page shows a red notice while the secret is missing.

Treat the recovery secret like a root credential

Losing the recovery secret makes every recovery bundle unrestorable - there is no other way to open one. Anyone holding the secret plus access to a repository copy can rebuild the appliance. Store it outside Sendense, in a password manager or offline escrow, never only on the appliance it protects.

Recovery Bundles

Each bundle contains the appliance's encryption key material, a configuration snapshot including the repository inventory, and a consistent recovery point of the appliance's metadata database - full or incremental. Bundles are encrypted with AES-256 using a key derived from the recovery secret.

Delivery targets are chosen automatically: every enabled local or network-share backup repository, plus every active EBA repository, receives a copy. You do not pick targets - repositories are configured on the repositories page, not here.

Retention is fixed at seven days and is not configurable. Within a chain, a fresh full anchor is forced when the previous chain is older than seven days, exceeds 48 incrementals, or is unusable.

  • Create Recovery Bundle Now starts an on-demand bundle in the background against all eligible repositories.
  • The success message only means the run was accepted - the actual outcome, including per-repository delivery failures, appears in the Latest run section after a refresh.
  • The button is disabled while a run is already in progress, and each run appears as a tracked maintenance job.

Schedule And Automatic Triggers

Scheduled bundles run daily at a time you choose (default 02:00 UTC). Enter the run time as HH:MM and the timezone as a standard region/city name - unrecognized timezone names are rejected on save. A Use browser timezone helper fills the field for you, and the page shows the computed next run. Schedule changes take effect within about a minute, no restart needed.

Two kinds of runs happen automatically outside the daily window: configuration-change bundles - changing repositories, encryption keys, or site credentials queues a bundle roughly ten minutes later, with multiple changes batched into one run - and automatic retries of partially delivered runs, by default every 30 minutes, until every repository has its copy.

Monitoring Delivery Health

The page rolls the latest run up into four tiles: total recovery repositories, how many are healthy, total bundles retained, and how many repositories hold a healthy database recovery chain. The last tile also shows whether the mandatory recovery secret is configured.

The Latest run section shows the newest run's trigger (manual, scheduled, retry, or configuration change), status (completed, failed, or partial), per-repository counts, and the last error. The repositories table shows per-repository delivery status, retained bundle count, database-chain health, last delivery attempt, and retry count. A repository picker lets you inspect the bundle history actually present in any repository's copy - up to the seven most recent, each marked Full or Incremental and Restorable or Degraded.

Review this periodically, not just during an incident: a repository whose database chain shows broken, or a run stuck in partial, cannot currently rebuild the appliance on its own.

Replication Concurrency Limits

The same page carries the estate-wide admission limits for DR replication syncs. All of them queue extra work rather than failing it - setting a cap too low does not break anything, it silently slows all DR replication while syncs wait their turn. Save writes all five values at once; a blank field falls back to the built-in default, and the page re-reads and displays the values actually in effect. The first four apply on the next admission cycle without a restart.

Per-pattern default (default 2)
Caps concurrent syncs within one replication pattern. A per-pattern override set in the replication wizard always wins over this default; 0 disables the check.
Per source SNA (default 3)
Caps concurrent syncs pulling from one source SNA, protecting it and its virtualization-manager connection from oversubscription. 0 disables the check.
Global ceiling (default 0)
Overall cap across the whole appliance. 0 defers to the service-level default (4); any value above 0 overrides it.
Setup-phase heartbeat timeout (default 300s, valid 30-3600)
Grace window for syncs still in pre-transfer setup (source snapshot creation, disk-export setup) before they are declared stalled. Raise it when snapshot creation is slow under load; setting it too high delays detection of genuinely stalled syncs.
Snapshot scan interval (default 900s, valid 60-86400)
How often the appliance scans for leftover source-side VM snapshots whose sync already ended. The scan reports leftovers for review - it does not delete them. Unlike the other values, a change takes effect only after the appliance's replication service next restarts, for example after an appliance update or reboot.

Rebuilding A Lost Appliance

The appliance recovery flow consumes a recovery bundle plus the recovery secret to rebuild a lost SHA. Any surviving repository copy is sufficient - that is why bundles are delivered everywhere automatically. Keep the secret escrowed outside Sendense and confirm delivery health regularly so at least one restorable copy always exists.

Related Docs