Multi-Tenant Provider API Onboarding

1. Purpose, Audience And Scope

This page is the integration guide for a cloud or managed service provider that wants to run migration and disaster recovery for its tenants on one multi-tenant Sendense Hub Appliance (SHA) from its own server-side automation, rather than through the appliance's web interface.

It is written so that a competent integration engineer can implement the full workflow — tenant onboarding through irreversible cutover — without a call with Sendense.

In scope. Provider-driven onboarding, credential provisioning, discovery, replication patterns, sync, test failover, rollback, planned and unplanned cutover, commit, licence and health monitoring, error handling and retry rules.

Out of scope. Tenant-facing self-service automation; the SHA web interface; backup (protection patterns) and Enterprise Block Archive (EBA) operations; fleet administration across many appliances; and any Sendense-hosted outbound webhook, OAuth or workflow engine — none exists, and section 12 explains what your service owns instead.

The contract. Every operation named here is defined in the generated provider API profile, SHA_MULTI_TENANT_CSP_OPENAPI.yaml. That file is the contract; this page is the workflow. See section 16.

The provider credential is a server-side credential. It reaches every tenant on the appliance. Never place it in a browser, a mobile app, a tenant's system, client-side JavaScript, a source repository, a shell history, a ticket, a screenshot or an ordinary application log. Section 4 explains how to hold it safely.

2. Architecture And Responsibility Boundary

Your automation talks to one SHA over HTTPS, on the same host and the same /api/v1/... paths the appliance serves to its own web interface. There is no separate provider endpoint, no separate port and no separate API version.

Sendense exposes state and identifiers precisely enough for an external orchestrator to make decisions. It does not make them for you, and it does not call you back.

2.1 Tenancy model

Two identifiers matter and they are not interchangeable:

• site_id authorises. Every tenancy gate on the appliance resolves against the site that owns the object. A replication pattern's site_id is its recovery site and its tenancy anchor.
• tenant_id groups. It is an organisational and reporting key. It never authorises anything.

A provider credential reaches the whole appliance by an explicit provider-scope claim, not by an absent site list. An empty site list means no sites, never all of them.

Cross-tenant denial renders as 404, not 403, everywhere. A denial never confirms that something exists. Plan for this in your error handling — see section 14.

3. Before You Begin

You need:

1. A multi-tenant SHA, with multi-tenancy activated and site-scope enforcement on. Provider scope, staff-only routes and the 404-not-403 denial behaviour described here are the enforced-mode behaviours.
2. A human administrator account on that appliance holding settings.write. You need it once, to mint the provider credential, and again whenever you revoke one. Automation cannot do either — see section 4.4.
3. A server-side secret manager your automation can read at runtime and you can audit.
4. Durable storage for your own workflow state, including identifiers and idempotency keys. Your orchestrator must survive its own restart without losing them.
5. A destination platform for each tenant's recovery site, and credentials for it.
6. Licence capacity for the workloads you intend to protect. Capacity is licensed to the appliance, not to a tenant — see section 13.

Confirm you can reach the appliance and that your credential works before building anything else: GET /api/v1/licensing/status (getLicenseStatus) is a good first call, because it is a read, it requires only settings.read, and it proves both authentication and provider scope.

4. The Provider Credential

4.1 What it is

A provider service credential is an explicit, appliance-wide automation credential. It is a distinct credential class with its own subject type and its own positive provider-scope claim. It is not a user session, and it is not a site API token.

All four require settings.write and a human session. See section 4.4.

4.2 Minting one

POST /api/v1/provider-tokens HTTP/1.1
Host: sha.example.test
Authorization: Bearer <administrator session>
Content-Type: application/json

{
  "name": "provider-example migration orchestrator",
  "description": "Server-side automation that onboards tenants and drives cutovers",
  "permissions": ["settings.write", "settings.read", "users.write", "dr.replication.read"],
  "expires_in_days": 180
}

Two rules the appliance enforces at mint time:

• Exactly one of expires_in_days and never_expires. Omitting both is refused. A permanent, appliance-wide credential must be a deliberate decision, not the result of leaving a field out.
• Requested permissions are intersected with the minting administrator's own. Nobody can mint a credential wider than themselves, and at least one permission must survive the intersection. A site-bounded administrator is refused outright: permissions can be intersected but site reach cannot, and this credential carries no site list.
• Permissions do not imply one another. The check is an exact string match, so settings.write does not grant settings.read. Grant read and write separately when the integration needs both — the example above does, because §3's first call is a read.
• Use exact permission keys, and read them back. The intersection silently drops anything it does not recognise, so a typo does not fail the request — it mints a narrower credential than you asked for. provider_token.permissions in the response is the set that was actually granted: assert it matches what you sent. The keys are the ones in §5.2. There is no tenants.* or sites.* permission — tenant and site creation are both settings.write.

The 201 response carries the clear token, a server-supplied warning, and the credential's safe metadata projection:

{
  "token": "<clear value — shown once, never retrievable>",
  "warning": "This is the only time the token is shown. It is stored as a hash and cannot be retrieved again — if it is lost, revoke it and mint another. This credential reaches every tenant on this appliance: keep it server-side and never issue it to a tenant or place it in a browser.",
  "provider_token": {
    "id": "ptok-3f9c2a7b41d84e6fa0b5c8d213e47f96",
    "name": "provider-example migration orchestrator",
    "token_prefix": "a1b2c3d4e5f60718",
    "scope": "provider",
    "permissions": ["settings.write", "settings.read", "users.write", "dr.replication.read"],
    "never_expires": false,
    "expires_at": "2027-02-14T09:00:00Z",
    "last_used_at": null,
    "created_by_user_id": "usr-provider-admin",
    "created_at": "2026-08-18T09:00:00Z"
  }
}

4.3 Holding it safely

The clear value exists exactly once, in that response. Sendense stores only a one-way digest of it and cannot reproduce it. If you lose it, revoke the credential and mint another.

Move the clear value directly into a server-side secret manager, in the same process that received it. Then, in order of how often these go wrong:

• Never send it to a browser or a mobile client. It is not a user credential and there is no scope in which a client-side holder of it is acting on its own behalf.
• Never issue it to a tenant, or embed it in anything a tenant runs. Provider scope reaches every tenant on the appliance, so a tenant holding it can read and act on every other tenant's estate.
• Never commit it, to your repository or anyone else's, and never paste it into a ticket, a chat message or a screenshot.
• Never put it on a command line where it enters shell history or a process listing, and never log the request header. Log the token_prefix if you need to identify which credential a call used — that is what it is for.
• Grant only the permissions the integration needs. Section 5 and section 6 are about exactly this.
• Choose expiry deliberately. A finite expires_in_days is the safer default; explicit never_expires is available and is a decision you should be able to justify.

Compromise of a provider credential is an estate-wide event, not a single-tenant one. Size your controls accordingly.

Sendense does not provide automatic credential rotation. Rotation is mint-new, cut-over, revoke-old, performed by an administrator. Build that into your operational procedures rather than expecting the appliance to do it.

4.4 What still needs a human

Minting and revoking a provider credential require a human administrator session. A provider service credential presented to any of the four provider-tokens operations is refused with 403, including when it targets itself.

This is deliberate and load-bearing in two directions:

• A service credential cannot mint a successor. Credential issuance stays attributable to a person, and revoking a compromised credential cannot be undone by that credential having already issued another.
• A leaked credential cannot kill the incident response that would contain it. It cannot revoke itself or anything else.

An administrator can do all four from the appliance's administration interface; no shell access, environment variable or compose-file edit is involved.

4.5 Revoking one

DELETE /api/v1/provider-tokens/ptok-3f9c2a7b41d84e6fa0b5c8d213e47f96 HTTP/1.1
Authorization: Bearer <administrator session>
Content-Type: application/json

{"reason": "orchestrator decommissioned"}

A reason is required — it is the record of why the credential was killed.

Revocation takes effect on the credential's next request, on both the SHA and the DR control plane, because liveness is re-checked per request rather than waiting for anything to expire. Revoking never touches the administrator account that created it, which is the whole point of the credential class: provider automation can be stopped without disabling a human login.

Revoking is safe to retry. The update matches only an unrevoked record, so a replay reports that there is no active credential with that id (400) rather than overwriting the original revocation's actor, reason or timestamp.

5. Designing Least-Privilege Permission Sets

5.1 How authorisation actually resolves

Three things gate every request, in this order:

1. Authentication at the appliance edge. Your provider credential authenticates on any route.
2. The permission map. The same map that gates a user session gates your credential. A route requires a named permission; you either hold it or you get 403.
3. Site scope. A provider credential holds appliance-wide reach by explicit claim. A credential whose provider-scope claim is absent falls through to the ordinary site check, finds no sites, and is denied — absence denies, it never widens.

Reach is therefore decided by the permissions the credential was minted with, not by a per-route allowlist.

Reading security on an operation. Where an operation in the profile lists only sessionBearer, that reflects how the canonical contract's family declares itself. It is not proof that a provider credential is refused there. The real exceptions are the ones above: all four provider-credential operations — mint, list, read and revoke — genuinely require a human session, because the list alone discloses the provider's whole automation estate. The profile's own preamble states this rule; treat it as part of the contract.

5.2 The permissions that matter

5.3 How the graded verbs compose

This is the part integrators most often get wrong, so state it exactly:

Every DR operation passes two gates. The appliance edge applies one ungraded rule to the whole /api/v1/dr/ surface — read for GET, write for everything else — and only then does the DR plane apply the graded verb. So a graded verb is additional, never a substitute:

A credential holding dr.failover.commit but not dr.replication.write is refused at the edge and can never commit.

Every rollback entry point takes dr.rollback. That includes the two under a failover job — POST /api/v1/dr/failover/jobs/{job_id}/rollback/retry (retryDRRollbackJob) and .../rollback/cleanup (cleanupDRRollbackJob). Retry is additionally bounded to a failover job whose status is rollback_failed; see Section 8.2.

Where that stops. It covers a failover that has REACHED a rollback-able state — pending_commit or completed. It does not cover a cutover still in flight: the failover compensation routes (cleanupDRFailoverJob, forceCleanupDRFailoverJob) are ungraded and run on dr.replication.write, and their admission set includes the mid-cutover states in which a planned source VM is already powered off. Compensation deletes the destination VM and powers the source back on, which for an in-flight cutover is the same physical outcome as a rollback. It deliberately refuses pending_commit and completed, so it cannot touch a completed failover or a working replica. If your threat model needs a credential that provably cannot undo a cutover at any stage, withhold dr.replication.write — which also means withholding every DR write, because the appliance edge requires it on all of them.

Failover is graded in the handler, not the middleware, because the type is in the request body — which is why a caller can pass the edge and still be refused. Commit and rollback are distinguishable by path and are graded in the DR middleware.

The consequence for you: a credential holding dr.replication.write and dr.failover.test can rehearse but cannot cut over; add dr.failover.live and it can cut over but still cannot commit. The graded verbs are what you withhold, and dr.replication.write is the floor underneath all of them.

inventory.discover is a similarly graded slice: it runs a rescan of a site the caller can already see, without unlocking the inventory writes that inventory.write also grants.

6. Seven Permission Examples

These are illustrative permission sets, not built-in Sendense roles. Sendense ships no provider-credential presets; you choose a permission list at mint time and the appliance intersects it with the minting administrator's own. Nothing here is a server-side name you can ask for.

Split your automation into as many credentials as these examples suggest. One credential that holds the union of all six is the thing least privilege exists to prevent.

Example 1 — Onboarding automation

Responsibility. Stand up a new tenant: tenant record, sites, tenant logins, destination credentials, discovery, and a dormant replication pattern ready for workloads.

Permissions. settings.write, users.write, inventory.write, inventory.discover, inventory.read, dr.replication.write, dr.replication.read

Can. createTenant, createSite, createUser, setUserSites, createVaultCredential, testStoredVaultCredential, unifiedDiscoverVMs, executeDiscovery, getDiscoveryExecution, listVMContexts, createDRPattern, addDRPatternVMs, and the DR reads that confirm each step landed — and, because they are ungraded, the failover-compensation routes cleanupDRFailoverJob and forceCleanupDRFailoverJob (§5.3).

Cannot. Trigger any failover (triggerDRTargetFailover, triggerDRPatternFailover) or commit (commitDRTargetFailover, commitDRPatternFailover) — it holds none of the four graded verbs — and it cannot reach any rollback entry point: rollbackDRTargetFailover, rollbackDRPatternFailover, retryDRRollbackJob and cleanupDRRollbackJob all require dr.rollback. It also cannot mint, list, read or revoke a provider credential: all four need a human session regardless of permission.

Caveat. The compensation routes it CAN reach undo a cutover that is still in flight, which for a planned source VM already powered off is the same physical outcome as a rollback (§5.3).

Human session still required? Yes, once, to mint this credential in the first place.

Why least privilege. Onboarding is the widest creating role on the appliance, so it is the one that most needs to be unable to cut over a production workload. Separating it from every failover verb means a defect or compromise in onboarding automation cannot power off a tenant's production VM.

Size its controls for settings.write, not for tenant creation. As §5.2 says, that one permission also reaches certificates, encryption-key management, system and admin routes and control-plane recovery. This credential's blast radius is appliance administration, so give it the monitoring, storage and review that implies — and prefer a short expiry.

It can also offboard. settings.write and users.write are the same floor the offboarding operations of Example 7 sit on (deleteTenant, deleteSite, deleteUser, deleteVaultCredential), so this credential can remove what it creates. The permission map does not separate creating a tenant from deleting one; if your design needs that separation, it comes from holding two credentials in two components, not from two permission sets.

Example 2 — Read-only observation and monitoring

Responsibility. Feed a provider dashboard, an SLA report or an alerting pipeline.

Permissions. dr.replication.read, inventory.read, settings.read

Can. Every DR read in the profile — listDRPatterns, getDRPattern, listDRPatternVMs, getDRAssignment, listDRTargets, getDRTarget, listDRTargetDisks, getDRSyncJob, getDRSyncJobProgress, getDRTargetSyncSummary, getDRPatternRPOStatus, getDRTargetFailoverReadiness, getDRTargetHealth, getDRTargetCommitReadiness, getDRFailoverJob, listDRVMReplicationStatus, getDRSuccessRateMetrics — plus listVMContexts, getDiscoveryExecution, getLicenseStatus and listLicenseAssignments.

Cannot. Change anything at all. No pattern edits, no sync, no acknowledgements, no cancellations.

Human session still required? No.

Why least privilege. A monitoring integration is the credential most likely to be widely deployed, longest-lived and least closely watched. It should be incapable of mutation, so that its blast radius is disclosure only.

Example 3 — Replication and sync operation

Responsibility. Day-to-day replication: keep patterns current, attach and detach workloads, run and retry syncs, manage schedules, clear transient errors.

Permissions. dr.replication.read, dr.replication.write, inventory.read

Can. createDRPattern, updateDRPattern, addDRPatternVMs, updateDRPatternAssignment, removeDRPatternVM, bulkRemoveDRPatternVMs (but see below), triggerDRTargetSync, triggerDRPatternSync, retryDRSyncJob, cancelDRSyncJob, retryDRAssignment, cancelDRAssignment, retryDRTargetProvisioning, validateDRTarget, updateDRTargetSchedule, acknowledgeDRTargetError (outside the published provider profile), acknowledgeDRTargetHealth, plus every DR read — and, because they are ungraded, the failover-compensation routes cleanupDRFailoverJob and forceCleanupDRFailoverJob (§5.3).

Cannot. Trigger a test or live failover (triggerDRTargetFailover, triggerDRPatternFailover) or commit (commitDRTargetFailover, commitDRPatternFailover), and it cannot reach any rollback entry point: rollbackDRTargetFailover, rollbackDRPatternFailover, retryDRRollbackJob and cleanupDRRollbackJob all require dr.rollback (§5.3). It also cannot create tenants, sites, users or credentials — it holds no settings.write or users.write.

Caveats. The compensation routes it CAN reach undo a cutover that is still in flight — see the second callout in §5.3. And bulk removal with destroy_controller=true additionally requires fleet-administrator access, which none of these example credentials hold: detaching VMs works, destroying the controller VM alongside it does not. That is deliberate — before commit the controller VM holds the only copy of the replicated data.

Human session still required? No.

Why least privilege. This is the busiest credential and the one that runs unattended on a schedule. Replication is not cutover; a credential that keeps replicas current has no business being able to promote one.

Example 4 — Test failover (rehearsal) operation

Responsibility. Run scheduled or on-demand DR rehearsals and clean them up afterwards.

Permissions. dr.replication.read, dr.replication.write, dr.failover.test, dr.rollback

Can. triggerDRTargetFailover and triggerDRPatternFailover with failover_type: "test"; rollbackDRTargetFailover and rollbackDRPatternFailover to undo them; getDRFailoverJob, getDRPatternActiveFailover, getDRFailoverJournal and getDRTargetCommitReadiness to follow and verify them; retryDRPatternFailover and retryDRRollbackJob where a rehearsal or its cleanup fails part-way.

Cannot. Trigger a planned or unplanned cutover — the handler refuses with 403 naming dr.failover.live. It cannot commit anything, so it cannot make a rehearsal permanent even by mistake.

Human session still required? No.

Why least privilege. Rehearsal is reversible and cutover is not, and this pair of permissions is exactly the line between them. Note that dr.rollback is included: without it the rehearsal credential can start a test failover it cannot clean up.

Example 5 — Live failover operation

Responsibility. Execute the cutover itself, planned or unplanned, once a human or a business process has approved it.

Permissions. dr.replication.read, dr.replication.write, dr.failover.live, dr.rollback

Can. triggerDRTargetFailover and triggerDRPatternFailover with failover_type: "planned" or "unplanned"; roll them back while rollback is still possible through rollbackDRTargetFailover and rollbackDRPatternFailover, resume a failed rollback with retryDRRollbackJob and remediate one with cleanupDRRollbackJob; read failover jobs, journals and commit readiness (getDRFailoverJob, getDRFailoverJournal, getDRTargetCommitReadiness).

Cannot. Commit — commitDRTargetFailover and commitDRPatternFailover need dr.failover.commit, which it does not hold — so it can bring a workload up on the destination and it can put it back, but it cannot discard the rollback path. It also cannot run a test failover unless you additionally grant dr.failover.test — the two are separately grantable in both directions.

Human session still required? No.

Why least privilege. A live failover is disruptive but recoverable. Keeping commit out of this credential means the decision to make a cutover permanent is a separate act, by a separate identity, at a separate moment.

Example 6 — Commit and irreversible cutover authority

Responsibility. Perform the single irreversible step, after the business decision is recorded.

Permissions. dr.replication.write, dr.failover.commit, dr.replication.read

Can. commitDRTargetFailover and commitDRPatternFailover, plus the reads that make a commit decision defensible: getDRTargetCommitReadiness, getDRTarget, getDRFailoverJob.

Cannot. Start a failover — it holds neither dr.failover.live nor dr.failover.test — and it cannot reach any rollback entry point — rollbackDRTargetFailover, rollbackDRPatternFailover, retryDRRollbackJob and cleanupDRRollbackJob all require dr.rollback, though the ungraded failover-compensation routes remain reachable for a cutover still in flight (§5.3). It cannot create tenants, sites, users or credentials.

It does hold dr.replication.write, because the appliance edge requires it on every DR write and commit cannot be reached without it (§5.3). That means it can also attach VMs and sync targets. If that is unacceptable, do not automate commit at all — see below.

Human session still required? No — but see below.

Why least privilege. This is the narrowest and most dangerous credential on the appliance. It should be held by the smallest possible component, used the fewest possible times, and its use should be traceable to a recorded approval. Consider not automating it at all: a commit performed by an administrator in the appliance's interface, after your orchestrator has proved readiness, is a defensible design and does not need this credential to exist.

Example 7 — Offboarding automation

Responsibility. Reconcile what the appliance holds against your own records, and decommission a tenant end to end: its replicas, any orphaned replication targets, its stored credentials, its logins, the tenant itself, and — if you choose — the sites it leaves behind. Section 17 is the procedure.

Permissions. settings.read, settings.write, users.read, users.write, dr.replication.read, dr.replication.write

Can. The reconciliation reads — listTenants, getTenant, listSites, getSite, listUsers, getUserSites — and the offboarding writes: deleteTenant (with its dry run), deleteSite, deleteUser, deleteVaultCredential, and the DR teardown primitives deleteDRPattern, removeDRPatternVM, bulkRemoveDRPatternVMs, listDROrphanedTargets, getDROrphanedTarget, removeDROrphanedTarget, and — outside the published provider profile, for appliance operators — deleteDRTarget and retryDRTargetDestroy. Because it holds dr.replication.write it can also sync and attach, and — like every credential with that permission — reach the ungraded failover-compensation routes cleanupDRFailoverJob and forceCleanupDRFailoverJob (§5.3).

Cannot. Trigger any failover (triggerDRTargetFailover, triggerDRPatternFailover) or commit (commitDRTargetFailover, commitDRPatternFailover), and it cannot reach any rollback entry point — rollbackDRTargetFailover, rollbackDRPatternFailover, retryDRRollbackJob and cleanupDRRollbackJob all require dr.rollback. It cannot mint, list, read or revoke a provider credential: all four need a human session regardless of permission.

Human session still required? No. Every step of section 17 runs on this credential.

Why least privilege. The floor is as wide as Example 1's — settings.write and users.write — because the appliance does not grade "delete a tenant" more finely than "administer settings". What this credential must not hold is any failover verb: a credential that destroys controller VMs and deletes tenants must never also be able to power off a tenant's production source. Give it the shortest expiry that covers one decommission, mint it for the job, and revoke it when the job is done (§4.5).

What it cannot bypass. deleteTenant refuses while any of the tenant's sites still holds a protected workload, however wide the credential; a site-scoped login cannot reach the tenant routes at all; and deleteDRPattern refuses to cascade through a target that carries failover history until that target's removal is confirmed through its plan token (§17.4).

6.1 Summary

* Example 6 must hold dr.replication.write to reach the commit route at all (§5.3), which carries ordinary DR write access with it. Provider-credential administration is absent from every row: all four of its operations need a human session.

† Examples 1 and 7 share the settings.write / users.write floor, so each can both create and delete tenants, sites, logins and credentials, and both can attach (they hold dr.replication.write); Example 1 additionally discovers (inventory.discover, inventory.write). Separation, where you need it, is by credential and component.

7. End-To-End Onboarding Sequence

Each step names the operation, what to persist, and how to know it finished.

7.1 Create the tenant

POST /api/v1/tenants — createTenant — settings.write

This one call creates the tenant, its sites and its logins in a single all-or-nothing transaction. If that suits you, steps 7.2 and 7.3 are already done and you can skip to 7.4.

POST /api/v1/tenants HTTP/1.1
Authorization: Bearer <provider credential>
Idempotency-Key: 0f0f9d6a-1f1b-4a4a-9f2a-6b1a2c3d4e5f
Content-Type: application/json

{
  "tenant_id": "acme-2026-prod",
  "name": "Acme Corp",
  "description": "Acme production tenant",
  "sites": [{"site_id": "site-london", "name": "London"}],
  "users": [{"email": "[email protected]", "full_name": "Acme Admin", "role": "tenant-replication-admin"}]
}

• Send an Idempotency-Key, and use a fresh one per tenant. A create that times out at the proxy is indistinguishable from one that never ran; without the header a retry provisions a second tenant with a second set of one-time credentials. A repeat with the same key returns 200 with replayed: true and no credentials — they existed only in the original 201 and cannot be reproduced.
• This key is matched on the key ALONE — the body is not fingerprinted, and the key is appliance-wide. Unlike the two DR commits, reusing a key for a materially different tenant is not refused: it replays the original tenant and creates nothing, so a caller who recycles a key believes it provisioned a tenant that does not exist. Nor is the key scoped to your credential: a key any administrator already used replays that tenant. Generate a UUID per attempt, and always read tenant_id back off the response rather than assuming it is the one you asked for. Verified against a running appliance.
• ?dry_run=true returns the plan and writes nothing. The apply re-plans inside its own transaction, so what you approve is what executes.
• Generated passwords are returned exactly once, in the 201 body. The appliance stores only hashes. Treat them as you would the provider credential.
• 409 normally carries a structured conflict list; a plan-to-apply race returns a bare error instead, so read conflicts defensively.

Tenant user roles are tenant-replication-admin, tenant-replication-operator and tenant-replication-viewer.

Persist: tenant_id, every site_id, every user_id, and the idempotency key you used.

7.2 Create a site

POST /api/v1/sites — createSite — settings.write

Only name is required. The id is a server-assigned UUID.

• Site names are unique appliance-wide (uk_site_name), across every tenant. One tenant's site name blocks another's, so prefix or namespace the names you generate.
• A duplicate name answers 500, not 409. The handler maps every create error to 500 with a plain-text body and does not distinguish the constraint violation. Do not parse it; check your own records.
• The blank check is exact, so a whitespace-only name is accepted with a 201. Trim your own input.
• There is no idempotency key here. A blind retry after a lost response either creates a second site or answers 500, and you cannot tell which from the response. Read your own records first, or create sites through createTenant, which is idempotency-key protected.

The response is a minimal seven-field shape, not the list or detail shape.

Persist: id — this is the site_id every later step anchors on.

7.3 Create a tenant user and assign sites

POST /api/v1/users — createUser — users.write PUT /api/v1/users/{id}/sites — setUserSites — users.write

A provider credential cannot be issued to a tenant, so a tenant operator who needs to see the appliance needs a user account.

createUser takes an email, a password and optional role names. The password is an input only; no credential is returned. A blank email or password, an unknown role and a duplicate email are all 400 — there is no 409.

setUserSites replaces the assignment list:

{"site_ids": ["site-london"]}

• site_ids is required. Omitting it or sending null is 400.
• An empty array removes all site access. Empty means no sites, never all.
• Blank entries are trimmed and duplicates removed; unknown site ids are rejected with 400, naming them.
• The response echoes the cleaned, applied list — read it back rather than assuming your input was applied verbatim.
• Applying this attempts to revoke the user's sessions, but that revocation is best-effort: if it fails after the assignments commit, the request still returns 200 and the sessions may survive until refresh or expiry. An already-issued access token survives in any case until it expires.

Persist: user_id and the applied site_ids.

7.4 Provision and test destination credentials

POST /api/v1/vault/credentials — createVaultCredential — settings.write POST /api/v1/vault/credentials/{id}/test — testStoredVaultCredential — settings.write

Store the credential the tenant's replication will use at the destination. Secrets are sent in the request body and encrypted at rest, and the response is a masked projection — for the destination types this workflow uses (cloudstack, vmware, nutanix) no secret is echoed back. Masking is applied per field name rather than to the whole payload, so treat the response as potentially sensitive and do not log it wholesale.

{
  "credential_type": "cloudstack",
  "credential_name": "acme-london-destination",
  "credential_data": {"api_host": "destination-platform.example.test", "api_key": "<redacted>", "secret_key": "<redacted>"},
  "scope": "site",
  "scope_id": "site-london"
}

• Scope must be site for a tenant destination. A global credential is not a tenant's.
• Returns 200, not 201.
• A duplicate name within the same (type, scope, scope id) is 409.
• A credential type that has no validation branch is refused with 400 even though the appliance advertises the type elsewhere. Test before you depend on it.

Then prove it works, before a pattern depends on it:

POST /api/v1/vault/credentials/cred-0000/test

The credential is loaded and decrypted server-side, the test result is persisted on the record, and no secret is echoed. Do this every time — a pattern created against an untested credential fails at provisioning, which is much later and much harder to diagnose.

Persist: credential_id.

7.5 Discover source VMs

Two operations start a discovery, and they are not interchangeable:

Unified discovery enumerates VMs from a source platform (vmware, cloudstack or nutanix) and returns them in the response. Supply credentials by reference only — vault_credential_id for VMware and Nutanix, cloudstack_source_id for CloudStack. The secret is injected server-side and never appears in the response. Legacy inline credential fields are ignored, and the deprecated credential_id field is actively rejected with 400. Never send a plaintext password here.

Set save_to_db: true to persist what it finds into the appliance's VM inventory. That write is not idempotent: VMs whose platform UUID is already owned by another credential are skipped, not errored, and are listed in skipped_vms. Read that array — a skipped VM is a VM you will not be able to attach.

Asynchronous discovery returns an execution_id you poll:

{"execution_id": "disc-0000", "status": "running", "execution_type": "manual", "triggered_by": "external-orchestrator"}

Note the shape: success is 200, not 202, and status is already "running". The 200 is an accept, not a completion.

The target site comes from the body site_id. Supply it. A concrete site runs against that site (a foreign or unclaimed site is 404); the literal "global" fans out across every site including other tenants' and requires estate-wide access (403 otherwise). Omitting it is accepted with a 200 and then fails asynchronously — the orchestrator requires a site, and the execution record lands in failed. triggered_by is required free text.

Poll GET /api/v1/discovery/executions/{id} — getDiscoveryExecution — inventory.read. Terminal states are success, failed and partial; pending and running are not. Treat partial as its own outcome and read vms_failed, vms_skipped and error_count before proceeding.

An unknown execution id answers 404 before the handler, in every scope mode. A foreign-site id also answers 404, but only under site-scope enforcement — the appliance this guide targets (§3). Either way a 404 here does not mean the run vanished; it may mean the id is not yours.

Persist: execution_id, and the terminal status you observed.

7.6 Pre-create a dormant replication pattern

POST /api/v1/dr/patterns — createDRPattern — dr.replication.write

Omit vm_context_ids and vm_assignments and you get a zero-VM dormant pattern. This is a supported, stable state, not a half-finished one: a zero-VM pattern owns no replication targets, so the sync scheduler, the manual-sync dispatcher and the failover orchestrator all no-op on it, and it consumes no licence admission. Set enabled: false as well to ship it switched off.

{
  "name": "gold-tier",
  "site_id": "site-london",
  "sync_interval_minutes": 60,
  "template_id": "tmpl-0000",
  "network_id": "net-0000",
  "disk_offering_id": "do-0000",
  "destination_vault_credential_id": "cred-0000",
  "enabled": false
}

• name, template_id and network_id are required. site_id must name an existing site and is the pattern's tenancy anchor.
• Pin destination_vault_credential_id whenever the site holds more than one destination credential. Unpinned falls back to a site-default ranking, which is ambiguous.
• Pattern names are unique per site, backed by a real database constraint. One tenant's pattern name never blocks another's.
• Licence admission does not run on an empty create. Supply vm_context_ids or vm_assignments here and create runs the same source-site authorisation, the same same-tenant rule and the same licence admission that attach does, with the same status codes (§7.7) — a create that names VMs can fail on licence. Create it empty, as above, and a provider pre-creating patterns will not discover a licence shortfall until workloads arrive at attach.
• Only a customizable service offering can provision, and it needs default_cpu_number and default_memory_mb. Provisioning sends a CPU count and a memory size with every controller deploy; CloudStack requires them on a custom (iscustomized) offering and refuses them on a static one — so a static offering can never provision (the appliance's own offering picker hides them for that reason), and a custom one cannot provision without both values. Either mistake used to surface per VM, hours later, in a job log; the create refuses first. The offering is looked up through the site's (pinned) credential when the create runs: static is 400 SERVICE_OFFERING_STATIC; custom without both values is 400 CUSTOM_OFFERING_REQUIRES_COMPUTE (missing[] names the fields); an id the destination does not have is 400 SERVICE_OFFERING_NOT_FOUND. When the destination cannot be asked, a pattern that carries both values is accepted unverified so a dormant pattern can be pre-created before its destination is reachable (a static offering would then fail at controller creation); one without them is 503 SERVICE_OFFERING_UNVERIFIED, and nothing is created. updateDRPattern applies the same rule to the merged result whenever it touches the offering, a value, the site or the pin: it cannot clear one value out from under a custom offering, cannot move the pattern onto a static one, and an explicit 0 is refused.

The infrastructure identifiers this body needs — templates, networks, offerings, zones, OS types and the site's destination defaults — are readable from the DR infrastructure lookups family (listDRTemplates, listDRNetworks, listDRServiceOfferings, listDRDiskOfferings, listDRZones, listDROSTypes, getDRSiteDestinationDefaults, and the VMware-specific lookups), all on dr.replication.read.

Persist: pattern_id.

7.7 Attach discovered VMs

Attachment is keyed on vm_context_id. Discovery hands back counts and platform records, not context identifiers — so read them from the inventory:

GET /api/v1/vm-contexts — listVMContexts — inventory.read

Each record carries context_id (the vm_context_id you need), vm_name, source_platform, the platform VM id and site_id. Filter with site_id, management_status, platform or search. Under site-scope enforcement the listing is filtered to the caller's scope and foreign rows are silently dropped, so an expected VM that is missing is more likely a scope or site-assignment problem than a discovery failure.

Then attach:

POST /api/v1/dr/patterns/{pattern_id}/vms — addDRPatternVMs — dr.replication.write

{"vm_context_ids": ["vmctx-0001", "vmctx-0002"]}

Supplying neither vm_context_ids nor vms is 400. The vms variant exists for per-VM overrides such as the destination network, but its element shape is not modelled in the contract, so this guide does not document it — set the pattern's defaults instead, or ask Sendense before depending on it.

Three things happen whenever VMs are named — here, and equally on a createDRPattern that supplies vm_context_ids. They do not happen on the empty create §7.6 recommends:

1. Source-site authorisation. The caller is authorised against both the pattern's recovery site and each VM's source site. They are allowed to differ by design, but you must be entitled to both. A site you may not use answers 404, not 403 — a denial never confirms the object exists.

2. The same-tenant rule. A replication pattern only carries its own tenant's workloads. Under enforced multi-tenancy the VM's source site and the pattern's site must belong to the same tenant: the same site, or two sites of one tenant (cross-site DR), are valid; a VM whose site belongs to another tenant is refused 409 CROSS_TENANT_ATTACHMENT — for every caller, the estate-wide provider credential included. Your credential reaches every tenant; that reach is administration, not permission to replicate one tenant's VM into another tenant's cloud. refused[] names each VM with both sites and both tenants, and nothing is attached. Ownership is read from the sites' tenant bindings (tenant_id, §7.11), never inferred from a name; an ungrouped site on either side is not a binding; and when the binding cannot be established the attachment fails closed (409 TENANCY_UNRESOLVED, 503 TENANCY_UNVERIFIED) rather than open. A pattern whose delete has been accepted (§17.3) refuses every attach with 409 PATTERN_DELETE_PENDING.

3. Licence admission. A licence shortfall surfaces here, not at pattern creation — and not as a 400. Two guards can refuse, and they answer differently:

   • The entitlement guard runs first, and on this operation it answers 403 only — an expired licence, a lapsed check-in, an insufficient edition, an unlicensed hypervisor pairing, or a workload released by reconciliation. Its body carries a machine-readable code in error, plus message and restore_only.
   • The protected-VM admission check runs second. The pool being at its limit is 409; licensing unavailable or invalid for the workload is 403. Its body is the ordinary {error, status, details} shape with no machine-readable code — only a message.

   So a 403 here is not necessarily a permission problem, and a 409 means the protected-VM pool is full. Read the body before you touch the credential's permission set.

Returns 202. Provisioning of each VM's replication target proceeds in the background.

Attachment is not naturally idempotent. The duplicate guard is an application-level check rather than a database uniqueness constraint, so it is racy under concurrency. Before retrying an attach whose response you lost, read the pattern's current assignments — GET /api/v1/dr/patterns/{pattern_id}/vms (listDRPatternVMs) — and attach only what is genuinely missing.

Poll assignments until each leaves provisioning:

Persist: each assignment_id, and each replication_target_id once the assignment reaches active. replication_target_id is null until then.

There is no filtered target lookup in this profile: resolve a target from its assignment (getDRAssignment, listDRPatternVMs) or from listDRVMReplicationStatus, which returns one posture row per live assignment with vm_context_id, assignment_id, target_id and target_status together.

7.8 Run the initial sync, then incremental syncs

POST /api/v1/dr/targets/{id}/sync — triggerDRTargetSync — dr.replication.write POST /api/v1/dr/patterns/{pattern_id}/sync — triggerDRPatternSync — dr.replication.write

{"sync_type": "full"}

sync_type is full or incremental; trigger_type is manual or scheduled. triggerDRPatternSync validates both and refuses anything else with 400. The single-target route does not validate them — it passes the values straight to enum columns, so a bad one is not caught at the API layer and surfaces later, typically as a 500. Either way it is not a 400 you can branch on. Send only the documented values. The body is optional on the single-target route.

The first sync is a full sync. Subsequent syncs are incremental, and the pattern's sync_interval_minutes drives them automatically — you do not have to trigger every one. Use these operations for the initial seed, for an out-of-band catch-up, and for a full resync where the pattern permits it (allow_manual_full_sync).

Single target returns 202 with the job:

{"message": "Sync triggered", "job": {"id": "sync-0000", "sync_type": "incremental", "status": "pending"}}

Whole pattern returns 202 with a dispatch handle:

{"trigger_job_id": "job-0000", "pattern_id": "pat-0000", "message": "pattern sync dispatch accepted; watch trigger_job_id or queue_event log entries for progress"}

Pattern dispatch runs server-side and serially through the admission gate, so a concurrency limit queues rather than fails.

Persist: job.id for a single target; trigger_job_id for a pattern.

7.9 Poll for completion

Sync job states: pending, running, then one of the terminal states completed, failed, cancelled. Progress reads carry progress_percent, bytes_transferred, total_bytes and transfer_speed_bps.

The pattern sync-dispatch read is scoped to the pattern in the path and carries terminal as an explicit boolean. Every way of failing to bind the job to that pattern — a job belonging to another pattern, a job of another type, an unresolvable record — answers 404, not 403, so a denial does not confirm existence.

GET /api/v1/dr/targets/{id}/sync/summary (getDRTargetSyncSummary) gives counts by state, bytes transferred and the last job over a period — the right read for "is this target healthy over time", and the read to consult before retrying a sync.

7.10 Confirm RPO and readiness

GET /api/v1/dr/patterns/{pattern_id}/rpo-status — getDRPatternRPOStatus GET /api/v1/dr/targets/{id}/failover/readiness — getDRTargetFailoverReadiness GET /api/v1/dr/targets/{id}/health — getDRTargetHealth

RPO status reports compliance across the pattern's targets:

{
  "success": true, "pattern_id": "pat-0000", "rpo_minutes": 60,
  "targets": [], "assignment_count": 0, "compliance_rate": 0, "average_rpo_minutes": 0,
  "summary": {"compliant": 0, "warning": 0, "breached": 0, "total": 0}
}

Read assignment_count alongside summary.total. summary counts materialised targets only, so total: 0 is ambiguous between "no VMs assigned" and "assigned but still provisioning". assignment_count disambiguates it. targets serialises as an empty array rather than null, so a correctly pre-created zero-VM pattern does not read as broken.

Readiness answers whether a target can fail over and why not if it cannot: {"ready": true, "reason": "Target is ready for failover"}. Health reports replica validation state per disk. Check both before any cutover; check readiness before every one.

Estate-wide posture reads: listDRVMReplicationStatus (one row per live assignment, with pattern, target, sync state and a display_status) and getDRSuccessRateMetrics (bucketed success-rate trend). Both are site-filtered under enforcement.

7.11 Reconcile after a restart or a lost response

A create whose response you never saw is indistinguishable, to you, from one that never ran; an orchestrator that restarts mid-workflow has the same problem for everything it had not yet persisted. Section 12.2 still asks you to persist every identifier — but you no longer have to trust that you did. The profile carries the reads that answer "what does this appliance actually hold?", so reconciliation is a read, not a guess:

GET /api/v1/tenants — listTenants — settings.read GET /api/v1/tenants/{id} — getTenant — settings.read GET /api/v1/sites — listSites — settings.read GET /api/v1/sites/{id} — getSite — settings.read GET /api/v1/users — listUsers — users.read GET /api/v1/users/{id}/sites — getUserSites — users.read

Listing tenants. listTenants returns every tenant on the appliance with its site and login counts — {"tenants": [{"id", "name", "status", "site_count", "user_count"}], "total"}. No filters, no pagination: the estate comes back in one page. Match your records by the tenant_id you persisted, or by name if you lost it (tenant names are unique). getTenant is the detail: the tenant's sites[] (with their SNA-pool attachment), its users[] with roles and site_ids, and its site-scoped credentials[] by identity only — id, site, type, name, never a secret. This is the inventory an offboarding plan is checked against (§17.1).

Listing sites and relating each to its tenant. listSites returns every site the caller can see — for the provider credential, the whole estate — and each carries tenant_id: the tenant that groups it, or null for an ungrouped site (a shared or provider-operated site, or one a tenant delete left behind, §17.6). getSite carries the same field. Relate a site to its tenant from this binding and nothing else: a site's name is a label you chose, the binding is what the appliance enforces grouping with. tenant_id is grouping and reporting data — it never authorises anything (§2.1), and the same-tenant rule of §7.7 reads it only to refuse.

Listing users. listUsers returns every login on the appliance with roles and permissions, unfiltered by design — users.read is a provider-side permission no tenant role carries, and there is no per-user detail read. For which of a tenant's sites a login is bound to, read the tenant (getTenant → users[].site_ids); for one login's bindings — the direct read-back of setUserSites (§7.3) after a lost response — read getUserSites, which answers {"user_id", "site_ids"}.

Who sees what. These are provider reads. Under enforced multi-tenancy a tenant-scoped login is refused on listTenants, getTenant, listUsers and getUserSites — 403 for lacking the permission, and 404 from the staff gate even if it held it — and on the two site reads it is projected: listSites returns only the caller's own sites (an empty binding list yields an empty list, never the estate) and another tenant's getSite answers 404, indistinguishable from a site that does not exist. inventory.read also admits the two site reads under enforcement, which is why a tenant operator can pick a site without holding settings.read.

Behaviour change in this release. The whole /api/v1/users family — listUsers, createUser, deleteUser, getUserSites, setUserSites, revokeUserSessions, and the user update routes — is now staff-only under enforced multi-tenancy. Any human login without the admin role now answers 404 on every one of them, whether or not it is bound to sites and even when it holds users.read or users.write (the seeded operator and viewer roles carry users.read, and an earlier release let them list users). Provider API credentials are unaffected (they are estate-wide by construction), and admin humans are unaffected. If you had automated user management through such a login, move it to a provider credential or an admin account.

Envelopes. listTenants and listSites answer {..., "total"}; listUsers answers {"users": [...]} with no total; getTenant and getSite answer the object directly. None of the four is wrapped in the {success, data} envelope the vault and DR-infrastructure lookups use.

Reconciling, concretely. After a restart: listTenants → for each tenant you expected, getTenant → compare its sites[], users[] and credentials[] with your records → for each site, getSite to confirm tenant_id → then the DR reads of §7.9 for patterns and targets. Anything the appliance holds that your records do not is something an earlier call created before its response was lost; anything your records hold that the appliance does not never ran. Recreate only the second kind.

8. Test Failover And Rollback

A test failover is a rehearsal. It brings the replica up at the destination, usually on a test network, without powering off the production source. It is fully reversible.

8.1 Run the rehearsal

POST /api/v1/dr/targets/{id}/failover — triggerDRTargetFailover Requires dr.replication.write and dr.failover.test.

{"failover_type": "test", "test_network_id": "net-0002"}

Returns 202 with the job. Typed confirmation is not required for a test failover — deliberately, because a confirmation demanded everywhere is a confirmation nowhere.

Pattern-wide: POST /api/v1/dr/patterns/{pattern_id}/failover (triggerDRPatternFailover) with failover_type: "test", parallel_limit to bound concurrency and excluded_vms for a selective run.

When a test failover completes, the failover job reaches completed and the target reaches failed_over. Both are stable; you can leave a rehearsal running while you validate it.

8.2 Clean it up

POST /api/v1/dr/targets/{id}/rollback — rollbackDRTargetFailover — dr.replication.write and dr.rollback.

{"confirm": "workload-01", "reason": "rehearsal complete"}

Rollback destroys the promoted destination copy and powers the source VM back on. There is no second rollback to undo it.

Rollback requires a typed confirmation. Under multi-tenant enforcement confirm and reason are both required and both checked before anything is destroyed. confirm is matched server-side against the resource's own authoritative name — the target's source_vm_name for rollbackDRTargetFailover, the pattern's name for rollbackDRPatternFailover — so echoing a value you supplied elsewhere in the request cannot satisfy it. A missing, blank or mismatched confirmation, or an empty reason, is refused with 400 and a body carrying code: CONFIRMATION_REQUIRED and confirm_with, which names exactly what to type. Nothing has run at that point. triggered_by is still accepted but no longer sets the audit actor; that is derived from your credential, and every entry point writes the audit trail — the refusal, and the initiation with your reason and the resolved actor. On the pattern route an unresolvable identity records as api rather than falling back to what you sent.

The single-target route returns 202; the pattern-wide route returns 200. The failover job moves to rolled_back and the target returns to ready, where it resumes ordinary replication.

If a rollback fails part-way, use its dedicated retry route — POST /api/v1/dr/failover/jobs/{job_id}/rollback/retry (retryDRRollbackJob) — rather than replaying the rollback trigger. Three things about it:

• It needs dr.rollback, exactly as the trigger does.
• It is eligible only when the failover job's status is rollback_failed. Any other status is refused with 409 ROLLBACK_RETRY_NOT_ELIGIBLE, whose body names the job's current status. A live failover sitting in pending_commit is refused: retrying there would start a first rollback rather than resume a failed one. Use rollbackDRTargetFailover for that.
• It re-runs an irreversible action rather than replaying a stored authorisation, so it requires its own confirm and reason, matched against the job's target source_vm_name.

A rollback that wedges without reaching a terminal state is moved to rollback_failed by the appliance, so it becomes retryable without operator intervention, and a retry whose own setup fails leaves the job retryable rather than stranding it.

The retry route does not work on a VMware destination. That path's rollback validation admits only completed and pending_commit, so the one status the retry route accepts is the one it rejects, and the call answers 500. On a VMware destination, roll a failover back through rollbackDRTargetFailover while it is still in pending_commit; there is no supported resumption of a VMware rollback that has already failed. .../rollback/cleanup (cleanupDRRollbackJob) is operator remediation for a partially applied rollback, not an automation step; it also requires dr.rollback, and it takes no confirmation.

8.3 Rehearsal boundaries

• A test failover can be repeated; a live one cannot.
• A rehearsal never reaches pending_commit and can never be committed. There is nothing to commit — commit exists only for a live or planned cutover.
• A rehearsal still consumes destination capacity while it is up. Roll it back.

9. Planned Cutover And Commit

A planned cutover is a live failover of a reachable source: Sendense powers off the production source VM, performs a final sync, and brings the replica up as the running workload. It is disruptive but still reversible until you commit.

9.1 Cut over

POST /api/v1/dr/targets/{id}/failover — triggerDRTargetFailover Requires dr.replication.write and dr.failover.live.

{"failover_type": "planned", "confirm": "workload-01", "reason": "cutover approved by change CR-0000"}

Live and planned failovers require the typed confirmation and reason. Returns 202.

Before you call it, and every time:

1. getDRTargetFailoverReadiness — ready must be true.
2. getDRTargetHealth — the replica must be validating cleanly.
3. getDRPatternRPOStatus or getDRTargetSyncSummary — the replica must be current enough that the final sync is short.
4. Your own approval record must exist.

When it completes, the failover job reaches pending_commit and the target reaches pending_commit. At that point the workload is running at the destination and rollback is still available.

9.2 Decide

This is the decision point, and it belongs to you, not to Sendense.

GET /api/v1/dr/targets/{id}/commit/readiness — getDRTargetCommitReadiness answers can_commit and, when false, reason. getDRFailoverJob gives the job's own status, current_step and any error_message, and getDRFailoverJournal gives the per-step audit trail of what ran and what compensation was applied — the record you use to explain the cutover afterwards.

Rollback remains possible while the target is in pending_commit and stops being possible the moment commit succeeds. There is no later undo, no grace period and no support path back. Once promoted_at is set and the target reads committed, the rollback path has been discarded.

9.3 Commit

POST /api/v1/dr/targets/{id}/commit — commitDRTargetFailover — dr.replication.write and dr.failover.commit.

POST /api/v1/dr/targets/tgt-0000/commit HTTP/1.1
Authorization: Bearer <commit credential>
Idempotency-Key: 7b3d1c88-2e4f-4a10-8c9b-5d6e7f801122
Content-Type: application/json

{"confirm": "workload-01", "reason": "cutover approved by change CR-0000"}

Send an Idempotency-Key, and persist it before you send the request. A key you cannot recover after a crash protects nothing. Section 14.3 is the full contract.

What commit does: the Sendense Controller VM that has been holding the replica is itself promoted. It is renamed to the source VM's name, its metadata is refreshed from the destination platform, the promoted_* fields are written, the target moves to committed, and controller ownership is retired — the target no longer owns a controller.

{
  "target_id": "tgt-0000",
  "job_id": "fo-0001",
  "promoted_vm_id": "vm-0000",
  "promoted_vm_name": "workload-01",
  "promoted_rename_status": "succeeded",
  "rollback_possible": false
}

Read these carefully:

• rollback_possible: false is the confirmation that the rollback path is gone.
• promoted_rename_status tells you whether the rename actually succeeded — succeeded, failed, skipped, or not_applicable on a VMware destination. It is the honest signal for "the production VM exists but is still named after the controller". Anything other than succeeded means the promotion stands but the name does not.
• cleanup_incomplete: true means the commit succeeded but some cleanup did not. The promotion still stands. Do not retry the commit; investigate the cleanup.

Pattern-wide commit (commitDRPatternFailover) promotes every VM in the pattern at once and returns pattern_id, job_id, committed (an array of target IDs) and failed (an array of per-VM failures). Read failed.length === 0 before treating a 200 as a finished cutover: fully_committed appears only on an Idempotency-Key replay of the commit (§14.3), not on the original 200. A partial pattern commit is terminal, so retrying it is refused rather than finishing the job, and the VMs already listed under committed are irreversibly promoted.

10. Unplanned Failover

An unplanned failover promotes the replica when the source side is unavailable or cannot be cleanly shut down. It is the same operation and the same permission as a planned cutover, with failover_type: "unplanned":

{"failover_type": "unplanned", "confirm": "workload-01", "reason": "source site unreachable, incident INC-0000"}

What differs is what you can rely on beforehand:

• There is no final sync, so the recovery point is whatever the last successful replication produced. Read last_sync_at on the target and getDRTargetSyncSummary and record the data loss you are accepting.
• Readiness may not be clean, because readiness partly reflects the source. Read it anyway and record what it said.
• The source is not powered off by Sendense, because it cannot be reached. Preventing a split brain when the source returns is your operational responsibility, not the appliance's.

Everything after the trigger is identical to a planned cutover: the job reaches pending_commit, rollback is available until commit, and commit is irreversible.

Rolling back an unplanned failover powers the source VM back on. If the source has been recovered independently in the meantime, that is a conflict you must resolve before calling rollback.

11. State And Identifier Lifecycle

11.1 The controller VM is not the promoted production VM

Before commit, controller_vm_id names a Sendense Controller VM: infrastructure that Sendense provisioned in the destination to hold and maintain the replica. It is not the customer's production workload, and it must never be published to an orchestrator, a CMDB or a tenant as such. Before commit the promoted_* identifiers are absent.

At commit, that same destination VM is promoted and renamed, and becomes production. The promoted VM is the controller VM that was holding the replica — that is correct by construction, not a conflation bug. But do not expect the two fields to match on a post-commit read: commit retires controller ownership and clears controller_vm_id, so a later getDRTarget returns a populated promoted_vm_id and an empty controller_vm_id. What must never be conflated is their meaning over time.

After commit:

• promoted_vm_id, promoted_vm_name and promoted_at are authoritative production identifiers.
• promoted_at is the reliable signal that commit completed.
• promoted_cloudstack_instance_name is best-effort. It is read back with a live call to the destination whose failure is logged and swallowed as non-fatal, so it can legitimately be empty after a fully successful commit — and on a VMware destination it is absent by construction. Its absence is not commit failure.
• The target no longer owns a controller.

11.2 Lifecycle diagram

This is the replication target's lifecycle. The failover job and the pattern assignment each carry their own separate status field — see the tables below. Every transition here is one the appliance actually writes.

stateDiagram-v2
    direction TB
    state "assignment: pending → provisioning → active" as attach
    [*] --> attach : addDRPatternVMs
    attach --> initializing : replication target created
    initializing --> ready : provisioning completed
    initializing --> error : provisioning failed
    initializing --> syncing : sync triggered before provisioning finished
    ready --> syncing : sync running
    syncing --> ready : sync completed
    syncing --> error : sync failed
    ready --> error : controller-identity or checkpoint failure
    error --> ready : acknowledge-error, or autonomous checkpoint recovery
    error --> initializing : retryDRTargetProvisioning
    ready --> failover_pending : failover triggered
    failover_pending --> failed_over : failover_type = test
    failover_pending --> pending_commit : failover_type = planned / unplanned
    failover_pending --> ready : failover failed, compensation succeeded
    failover_pending --> error : failover failed, compensation did not
    failed_over --> ready : rollback
    pending_commit --> ready : rollback — LAST chance
    pending_commit --> committed : COMMIT — irreversible
    committed --> [*] : record released, promoted VM KEPT
    error --> deleting : deleteDRTarget (operator)
    deleting --> delete_failed : destroy failed
    delete_failed --> deleting : retryDRTargetDestroy (operator)
    deleting --> [*] : target removed

Read the diagram with one rule in mind: pending_commit is the last state from which rollback is possible. Everything before it is reversible; committed is not.

committed is terminal for the replication lifecycle, and it is NOT a state that leads to a teardown. The VM the commit promoted is the tenant's production workload, so no Sendense path destroys it — see §17.1 Cutover safety state. A completed migration leaves the pattern by having its membership released, and its DR record is removed through the confirmed orphaned-target removal, which deletes records only.

The same lifecycle as a table, for when the diagram is not rendered:

17.1 Cutover safety state: what may be destroyed

Every replication target carries cutover_safety_state, and it — not status — is what every destruction path reads. It answers one question: may an ordinary Sendense cleanup path delete this target's destination VM and volumes?

reconciliation_required is TERMINAL, and deliberately so. It is the state Sendense uses whenever it cannot account for a destination, and in this release it is where every automatic recovery path ends — inline compensation, the failover job's cleanup, the rollback job's cleanup, force cleanup, the reaper, boot recovery and panic recovery. A target in it:

• cannot be destroyed;
• cannot be reprovisioned;
• cannot start another failover;
• is never automatically stale-cleaned;
• may have its pattern membership released, with zero provider-resource operations;
• may be removed from Sendense only through the confirmed record-only reconciliation workflow, which deletes Sendense records and states that destination resources may remain unmanaged.

There is no route back to ordinary. That is the first-release boundary, not an oversight: an operator-facing certify route is tracked for a future release.

ordinary is necessary but not sufficient. Three independent vetoes sit beside the state, and an ordinary target still answers 409 when any of them fires: it carries a promoted identity (code: promoted), its lifecycle status says a cutover happened (code: cutover_status), or another operation holds it (code: operation_lock_held). A target that does not exist answers 404, and a row that cannot be read answers 409 target_unreadable, which is retryable.

What the provider profile does NOT publish

Three per-target remediation operations were removed from the Multi-Tenant SHA provider profile for the first release, pending a separately accepted security review:

They are not required by the supported provider migration workflow, which is: create a pattern, add VMs, replicate, fail over, commit or roll back, release membership. They remain available to appliance operators and are unchanged in the full SHA API — they are simply not part of the published provider contract.

Several operator remediation routes were never in the profile and remain out of it: the failover job's cleanup, force cleanup, the rollback job's cleanup, sync-checkpoint recovery, force-clear-sync, and the abandon-failed-target route.

Teardown is unaffected. A provider still takes down what it created, through the pattern delete, the VM detach and the confirmed orphaned-target removal — and for a promoted or reconciliation_required target the first two release membership only, with zero provider-resource operations.

A stated limitation of the first release

Only two things return a target to ordinary once a cutover has started, and neither of them infers anything:

1. The cutover never began. The failover job row and its protection were created, but no execution goroutine and no platform side effect started. The owning job — and nothing else — clears its own protection.
2. An explicit platform rollback completed. The normal CloudStack or VMware rollback executor reached its actual successful terminal point, after every required platform operation returned success. That function's successful return is the proof; nothing is derived from the step journal, the compensation record or the job's status history.

Everything else leaves the target for reconciliation. A failover that entered execution and then relied on automatic compensation ends at reconciliation_required even when that compensation succeeded — inline compensation, the failover job's cleanup, force cleanup, the reaper, boot recovery, panic recovery and stale-lock recovery are all in this class.

What that costs you, plainly: such a target is never torn down by Sendense. You release its pattern membership, reconcile the destination VM and replica volumes yourself in your own cloud, and then remove the DR record through the confirmed orphaned-target removal — which deletes records only. To protect the same VM again afterwards you create a new target.

Why it is built this way. Five independent review rounds each produced a mechanism for deciding whether an automatic compensation had left the destination clean — a journal completeness scan, a compensation ledger, a liveness inference, a platform observation — and each one was shown to certify, in some reachable case, a destination that was still running the customer's workload as safe to delete. A state that over-refuses costs an operator a manual reconciliation. A state that under-refuses costs a running VM. The first release takes the former, and an operator-facing certify route is tracked for a future release.

Sync-checkpoint rollback is refused on a non-ordinary target. A checkpoint rollback stops the controller VM, reverts the replica volume snapshots and rewinds the CBT baseline — all of which target the same destination a cutover is using. Both routes that reach it (the autonomous checkpoint reaper and POST .../checkpoint/recover) now refuse for cutover_protected, promoted, reconciliation_required and any state the appliance cannot read, and make zero platform calls when they do. Finish or reconcile the cutover first; the checkpoint is left untouched and can be recovered afterwards.

A job recorded as in flight is never removed, and one class of it needs support. Where a target's failover history shows an operation still in flight, the confirmed record-only removal REFUSES it, in every safety state and whether or not an operation lock is still held. That refusal is deliberate: the appliance's stale-lock sweep clears a cutover lock by AGE without establishing that the worker stopped, so a missing lock is not evidence that anything ended, and removing the record could delete the target and its job rows out from under a running worker.

Most in-flight statuses settle themselves or are swept. Two groups are not: compensating and the four compensation_* statuses have no sweep behind them, so a cleanup whose process died leaves a job that nothing retires. That case requires Sendense support. It is not released through the provider API, and it is never resolved by destroying the target — a liveness-gated, support-only retirement is tracked for a future release.

One residual, stated rather than hidden. On CloudStack the source power-on at the end of a rollback is deliberately non-fatal — its failure message is "please manually power on source VM" — and it is not part of transition 2's proof. So an unplanned cutover whose rollback completed but whose source did not come back settles to ordinary, and an ordinary teardown of that target would then delete replica volumes while the source is still down. Confirm the source is running before tearing down a target that was rolled back from an unplanned cutover.

Three things follow that a provider integration has to plan for:

• DELETE /api/v1/dr/targets/{id}, POST /api/v1/dr/targets/{id}/destroy/retry and a POST /api/v1/dr/targets/{id}/provision/retry that would tear a stale target down first all answer 409 with code, cutover_safety_state and a remedy. Nothing is done to the target: no platform call, no status change, no lock left held. A target that does not exist answers 404, not 409 — a replay of a completed teardown is not a protected workload.
• A pattern delete and a VM detach release a promoted or reconciliation_required target instead of destroying it: the membership row goes, the target record and the destination stay. A release is not cleanup, and the response says so — a single-VM detach answers 200 with controller_vm_status: released_not_destroyed and released_targets, each entry carrying the remedy that says which of the two dispositions it is. A cutover_protected target is not released at all: the delete refuses it with the remedy to finish the cutover — and the remedy now says which routes return it to ordinary and which retire it for reconciliation.
• Removing the record through POST /api/v1/dr/orphaned-targets/{id}/remove deletes records only. For a reconciliation_required target the destination resources are left unmanaged, and the discard_provider_resources confirmation asks you to acknowledge exactly that. A promoted target is not asked for that confirmation: its VM is production and is kept, so the plan carries a disclosure saying so instead. An empty confirmations_required on a non-ordinary target therefore does not mean there is nothing to lose.

Upgrading: what the backfill does to your existing targets

Read this before upgrading. It changes what Sendense will tear down for you.

The rule is deliberately blunt, because the alternative was wrong. The multi-tenant SHA has had no production release, so this migration does not try to reconstruct what a pre-release build actually did to any destination. It refuses to guess:

Any failover history at all disqualifies a target from ordinary, whatever that history says about itself.

No pre-release job status is treated as proof that destination resources are absent — not completed, not failed_cleaned, not failed_recovered, not a completed rollback. Those values were written by builds that asserted more than they had done, and no migration can separate an honest row from a dishonest one.

What each of your targets becomes:

What that costs you. A target that has ever been failed over — including a single successful test rehearsal — keeps replicating and can still be failed over, but Sendense will never tear it down. Its exit is to release the pattern membership and remove the DR record through the confirmed orphaned-target removal, which deletes records only and leaves the controller VM and replica volumes in your cloud for you to remove. There is no route that returns a reconciliation_required target to ordinary; that is a deliberate decision for this first release, not an oversight.

The upgrade procedure, in order

Four steps. Do them in this order; step 4 is the one most estates will actually have to act on.

1. Quiesce the cutover plane. No failover, rollback or commit may be running, and no target may hold a failover, commit or rollback operation lock. Run the pre-upgrade check below; every count must be zero. If one is not, finish or roll back that operation and re-run the check. Do not force the upgrade past a non-zero count: the migration does not reconstruct a protection owner, so an operation that is running when you install becomes reconciliation_required and cannot be resumed.

2. Take a database backup, and verify it. The upgrade adds a column whose backfill is a one-way classification of your existing estate. A restore is the only way back to the pre-upgrade classification.

3. Audit what the upgrade will do. Run the audit query below before installing. It predicts the post-upgrade state of every target, and it has been verified to agree with the migration row for row. Read the reconciliation_required list: those are the targets Sendense will no longer tear down for you.

4. Inspect every replication pattern, and plan for its targets. A pattern whose targets are classified reconciliation_required still replicates and can still be managed, but it can no longer fail over and Sendense will never destroy its controller VMs.

   A test pattern is the common case, and it is the case to plan for. A replication target that has ever been failed over — including a single successful test rehearsal — is classified reconciliation_required, because a pre-release job status is not evidence about what is at the destination. If your estate is one test pattern used for rehearsals, expect all of its targets to be classified that way.

   The remedy is to release and recreate, and it is deliberately unexciting:

   • after the upgrade, GET /api/v1/dr/orphaned-targets lists every affected target with its state and its remedy;
   • delete the pattern, or detach its VMs. For reconciliation_required and promoted targets this releases membership only and makes zero StopVM, DetachVolume, DeleteVolume or DeleteVM calls — nothing in your cloud is touched;
   • remove each released record through POST /api/v1/dr/orphaned-targets/{id}/remove, which deletes Sendense records only and requires you to confirm that destination resources may remain unmanaged;
   • delete the leftover controller VMs and replica volumes in your own cloud;
   • recreate the pattern and add the VMs again. The new targets start ordinary and behave normally, including failover.

   Do this at a time of your choosing. Nothing expires, and a reconciliation_required target keeps replicating until you release it.

Before you upgrade

Do not upgrade while a failover, rollback or commit is running. This is a precondition, not advice: the migration does not reconstruct a protection owner for a pre-release job, so a cutover that is running when you install becomes reconciliation_required — never destroyed, but never torn down by Sendense either. If any count below is non-zero, stop and finish the operation first.

-- Pre-upgrade check. Every count must be zero before you install.
SELECT 'active failover jobs' AS check_name, COUNT(*) AS blocking
  FROM replication_failover_jobs
 WHERE status IN ('pending','syncing_final','snapshotting','stopping_vm',
                  'injecting_drivers','detaching_disks','reordering_disks',
                  'configuring_network','starting_vm','validating')
UNION ALL
SELECT 'jobs awaiting a commit or rollback decision', COUNT(*)
  FROM replication_failover_jobs
 WHERE status IN ('completed','pending_commit')
UNION ALL
SELECT 'active rollback jobs', COUNT(*)
  FROM replication_failover_jobs
 WHERE status IN ('rollback_pending','rolling_back')
UNION ALL
SELECT 'commit in progress', COUNT(*)
  FROM replication_failover_jobs
 WHERE status = 'committing'
UNION ALL
SELECT 'compensation in progress', COUNT(*)
  FROM replication_failover_jobs
 WHERE status IN ('compensating','compensation_stopping_vm','compensation_reverting',
                  'compensation_deleting','compensation_starting_vm')
UNION ALL
SELECT 'held cutover operation locks', COUNT(*)
  FROM replication_targets
 WHERE COALESCE(locked_by_job_id,'') <> ''
   AND COALESCE(locked_by_operation,'') IN ('failover','commit','rollback');

Resolve each non-zero row before installing: let a running failover finish, then commit or roll it back; let a running rollback or compensation complete; and clear a held lock only after confirming no operation is behind it. Do not reconstruct an interrupted operation by hand — if the check does not pass, stop the upgrade. Take a database backup first either way.

Audit what the upgrade will do to your estate

This reproduces the migration's own precedence, so the counts match what the upgrade will write:

SELECT CASE
         WHEN t.status = 'committed'
           OR COALESCE(t.promoted_vm_id,'') <> ''
           OR COALESCE(t.promoted_vm_name,'') <> ''
           OR t.promoted_at IS NOT NULL
           THEN 'promoted'
         WHEN EXISTS (SELECT 1 FROM replication_failover_jobs j
                       WHERE j.replication_target_id = t.id)
           OR t.status IN ('failover_pending','failed_over','pending_commit',
                           'rollback_pending','rolled_back')
           OR (COALESCE(t.locked_by_job_id,'') <> ''
               AND COALESCE(t.locked_by_operation,'') IN ('failover','commit','rollback'))
           THEN 'reconciliation_required'
         ELSE 'ordinary'
       END AS cutover_safety_state,
       COUNT(*) AS targets
  FROM replication_targets t
 GROUP BY 1
 ORDER BY 2 DESC;

A test pattern whose targets all become reconciliation_required is expected on an appliance that has been used for rehearsals. The supported way back to a clean estate is to release those memberships, remove the DR records through the orphaned-target inventory, delete the leftover controller VMs and volumes in your cloud, and recreate the pattern.

Choosing the right way out of error matters. Pick by what failed:

acknowledgeDRTargetError is itself refused with 409 and code: CHECKPOINT_RECOVERY_REQUIRED while a sync checkpoint still needs recovering; the body names the blocking checkpoint. The checkpoint reaper may also clear the error and return the target to ready on its own, so a target can leave error without your calling anything — poll, do not assume.

Four values in the schema are not part of this lifecycle. provisioning is the column's stored default and no code path writes it — a target you can read is already initializing or later. rollback_pending and rolled_back are statuses of the failover job, not of the target: rollback returns the target directly to ready, so do not wait for the target to show them. sync_paused is defined but unwritten.

The failover job carries its own status field, which is not the target's:

11.3 Identifier table

11.3.1 Reading the disks

GET /api/v1/dr/targets/{id}/disks (listDRTargetDisks, dr.replication.read) is the dedicated listing for the source-to-replica disk mapping; getDRTarget returns the same rows alongside root_disk_index. Each row carries disk_type — replica for a copy of a source disk, controller for a disk belonging to the Sendense Controller VM itself, so filter before you count — plus source_disk_id and source_disk_index on the source side and volume_id, volume_name and, once attached, destination_disk_id on the destination side.

Two cautions. Do not assume source_disk_index: 0 is the boot disk; read root_disk_index on the target instead. And last_change_id is the incremental-sync anchor — if it is lost, the next sync for that disk is a full, which matters for your RPO and your transfer budget.

addDRTargetDisk adds a replica disk, and it is the one operation in this profile that is outright unsafe to replay (§14.2): each call creates and attaches another volume, and nothing deduplicates by source disk id.

11.4 Commit-readiness vocabulary

getDRTargetCommitReadiness and the commit conflict body use a commit_state vocabulary that describes the commit lifecycle, not the failover's. This distinction has exactly one dangerous failure mode, so it gets its own rule:

Never read never_started as "my failover did not happen". It means the opposite: the failover ran and is awaiting commit. A caller that retries a live failover on seeing never_started starts a second cutover of a production VM. The only state that means nothing ran is no_failover_job.

12. Polling And Third-Party Orchestration

12.1 Sendense does not call you

Sendense provides no generic outbound webhook engine, no OAuth client-credentials workflow for third-party systems, no external asynchronous-operation poller and no customer-defined action-pipeline runner. None of these exists in the product today, and this guide does not describe a way to configure one.

The provider's external automation service owns:

• third-party token acquisition and third-party REST calls;
• external asynchronous polling and external health checks;
• the business decision that precedes an irreversible commit;
• the workflow state machine that ties all of it together.

Sendense's obligation is to expose state and identifiers precisely enough for you to make those decisions. Sections 7, 11 and 14 are that surface.

12.2 What your orchestrator must do

1. Hold the provider credential in a server-side secret manager, never in code, config in a repository, or anything a browser or a tenant can reach.
2. Persist every identifier in section 11.3 that your workflow will need again — tenant, site, user, credential, pattern, assignment, target, disk, job and promoted ids — and reconcile them against the appliance after a restart with the reads of §7.11 rather than trusting that every earlier create landed.
3. Persist idempotency keys before sending the protected operation, not after.
4. Poll the durable job and readiness routes. Do not infer completion from elapsed time.
5. Survive your own restart without losing workflow state. An orchestrator that forgets it started a commit is the exact failure the idempotency key exists for.
6. Keep test, live, rollback and commit authority separate — separate credentials, and ideally separate components. See section 6.
7. Require an explicit business decision before commit. Never let a retry loop, a timeout handler or a scheduler reach the commit route on its own.
8. Treat the generated profile as the contract. Do not scrape the appliance's web interface, and do not parse human-readable error strings — read status codes and the structured fields.

12.3 Polling guidance

• Poll progress routes, not trigger routes. Re-issuing a trigger to "check" is how second jobs get started.
• Back off. A sync of a large disk runs for a long time; transfer_speed_bps and eta_seconds on the progress read let you pick a sensible interval.
• Treat every non-terminal state as "keep waiting", not "retry".
• Record the terminal state you observed, with its timestamp, before moving your workflow on.
• A 404 on a polling read is not automatically "gone" — under site-scope enforcement it can also mean "not yours". Check whether the identifier is one you actually persisted from a successful call.

13. Licence, Consumption And Health

Two operations, both settings.read, both staff-only — a site-scoped caller is refused with 404 under site-scope enforcement, so these are provider-credential reads.

13.1 Licence status

GET /api/v1/licensing/status — getLicenseStatus

One read gives capacity, consumption, position and entitlements:

Never present vmp_pool_capacity and vmp_total_capacity under one label. They differ by what is licensed but not currently enforceable — currency-lapsed and otherwise unusable documents. A document in its grace period is not in that gap; it still admits new workloads.

An unreachable licence server is a degraded state, not an error. The route returns 200 with central_verification.status = UNREACHABLE and a failure_code; the upstream error text is never echoed. Poll it rather than treating the first UNREACHABLE as an outage.

13.2 Consumption attribution — read this before you build a report

Capacity is licensed to the appliance, not to a tenant. Nothing in pool carries a tenant key, and it is provider-wide with no tenant or site attribution. Do not divide it up.

insights is the only place the API attributes consumption to a tenant. insights.tenants carries the per-tenant position, and insights.holds[] carries a tenant_id on each hold — read both, because consumption is bound capacity plus held capacity. Three caveats you must honour:

1. insights is best-effort and may be absent. The whole object is omitted when its build fails, so that a reporting join can never take down the licence figures beside it. An absent insights means unknown — retry. It never means zero.
2. The counts on one row come from two different keys. workloads is grouped by resolving each live protected-VM ledger row through a keyed relational chain (protected-VM row → VM context → site → tenant). armed and release_pending are grouped by the tenant_id column carried on the assignment row, which is display and audit grouping and never an authorisation input. The two counts on one row can legitimately disagree. Do not treat them as one join.
3. Read the whole array before concluding anything. Where the resolution chain is absent, every workload lands in the untenanted "" group. An all-empty-tenant response means attribution is unavailable on this appliance — not that nothing is tenanted.

tenant_name is resolved for display only. It is never a join key and never an authorisation input. Never attribute consumption by name matching, free text or a naming convention.

13.3 Licence assignments

GET /api/v1/licensing/assignments — listLicenseAssignments

Workload licence assignments, provider-wide across every tenant, newest first. Unbounded by default: supply limit and offset to page a large estate and read total to know when to stop. The paging window never widens the state or lineage_id filters, and total counts the filtered set rather than the table.

Each assignment carries vm_context_id, workload_lineage_id, state, and the release timestamps — enough to reconcile consumption against your own records through a real key.

13.4 Replication health

For DR health, use the operations in section 7.10: getDRPatternRPOStatus, getDRTargetFailoverReadiness, getDRTargetHealth, getDRTargetSyncSummary, listDRVMReplicationStatus and getDRSuccessRateMetrics.

For appliance-level and cross-product monitoring — protection patterns, per-VM SLA, EBA repository capacity, and a Prometheus endpoint — use the dedicated monitoring API with its own scoped token, documented in External Monitoring. That is a separate credential class with its own scopes, and it is deliberately not part of this profile.

14. Error Handling And Retry Rules

14.1 Status codes and what they actually mean

Two more you will meet:

• 400 is used where you might expect 409 on several SHA routes: a duplicate user email, an unknown role and an unknown site id are all 400. A replayed provider-credential revoke is also 400.
• 503 means a capability is not configured on this appliance — the pattern service, the pattern sync dispatcher, site assignment, the provider-credential feature, or the idempotency key store. It is a deployment fact, not a transient fault. Do not retry in a tight loop.

Error bodies are not uniform. Middleware and site-gate denials use one envelope; several handler families use their own. The profile models the overlapping cases explicitly. Read the status code and the structured fields the operation documents; never parse a human-readable message.

14.2 Retry classes

Every DR-plane create and trigger states its retry class in its description. Do not infer retry safety from the HTTP method.

Most SHA-plane writes carry no retry class. Three of them state their replay behaviour in their own contract description — createSite, unifiedDiscoverVMs and revokeProviderToken — and the six below say nothing at all. One of those six matters a great deal:

Sharp edges worth knowing before you meet them in production:

• createDRPattern cannot create a duplicate — names are unique per site with a real database constraint — but the replay does not succeed either; it is refused with 409. And the create also attaches VMs and starts provisioning, so read the pattern list for the name before retrying.
• addDRPatternVMs is not naturally idempotent and the duplicate guard is application-level rather than a database constraint, so it is racy under concurrency. Read listDRPatternVMs before retrying.
• Cancels do not converge in their response. A second cancelDRSyncJob answers 409 SYNC_JOB_NOT_CANCELLABLE; a second cancelDRFailoverJob fails its own guard and answers 500, which a naive client reads as transient and retries in a loop — and the first call may already have run cleanup that destroyed provisioned resources. Read the job's status first, always.
• deleteDRPattern is one request, one deletion. A pattern that owns targets answers 202 and finishes on its own: it is disabled, its delete is recorded (delete_requested_at on the read model), its controller VMs are destroyed, and the pattern deletes itself when the last target is gone. A replay while cleanup runs is accepted again; a replay after completion — and any read — answers 404. There is no second delete to send (§17.3), and a 202 with deferred: true still completes on the appliance's own sweep (which leaves mid-cutover and locked targets to you — §17.3). The flag destroy_vms=true is accepted but not decisive: a pattern delete destroys the controller VMs of the targets it owns that are in a replication state — and never one a failover touched (§17.3).
• removeDROrphanedTarget is confirmed against a plan. Read getDROrphanedTarget immediately before, quote its plan_token, and send every confirmation it names as true; a stale token is 409 plan_changed with the current plan, a missing confirmation is 409 confirmations_required. A second remove answers 404 (§17.4).
• deleteDRTarget is the most dangerous replay on this plane. Destroy is asynchronous: inside the window a second call returns the running job with 202, which teaches a caller the replay is safe; afterwards it answers 404 or 500. Before commit the controller VM holds the only copy of the replicated data. Read the target's destroy state before retrying.
• acknowledgeDRTargetError converges in state but not in response: a second call finds the target no longer in error and answers 400.

14.3 The two irreversible commits

commitDRTargetFailover and commitDRPatternFailover are the only DR-plane operations that accept an Idempotency-Key, and they are the ones that most need it. createTenant accepts one too (§7.1); no other operation in this profile does.

With a key:

1. Persist the key before sending the request. This is the whole mechanism. A key generated in memory and lost to a crash protects nothing.
2. Send it as the Idempotency-Key request header.
3. A retry of the same request returns a bounded record of the original outcome with replayed: true and the idempotency_key you sent, instead of doing anything.
4. Reuse the same key only for a materially identical request. On the two DR commits the same key with different content is refused with 409 IDEMPOTENCY_KEY_REUSED. createTenant does NOT do this — its key predates the DR store and matches on the key alone, appliance-wide and unscoped to the caller, so a reused key replays the original tenant instead of refusing (§7.1). Treat the refusal as a DR-plane guarantee, not a general one.
5. A duplicate arriving while the original call is still running is refused with 409 IDEMPOTENCY_KEY_IN_FLIGHT. Wait and poll; do not spin.
6. If the appliance has no key store configured, a request carrying the header is refused with 503 rather than run unprotected — you would otherwise believe an irreversible operation was replay-safe when it was not.

The key is scoped to this operation, this exact resource and your own identity, so it can never replay across tenants, sites, targets or patterns. It authorises nothing: it is consulted only after authentication, the permission check, the site gate and the typed confirmation, so a replay must re-satisfy all four.

The replay payload is deliberately bounded. It does not reproduce every counter or internal handle from the original execution.

The omitted fields are execution counters that describe one run rather than the outcome, plus a credential-store handle that does not belong in a durable retry cache. Read them from the first response or from the resource itself.

The pattern replay is a different shape because its success response is: it carries pattern_id, job_id, committed, failed, fully_committed, replayed: true and the key, plus warning when the pattern is not fully committed. failed is the same length as the original's so a client that counts it keeps working, but each entry carries exactly target_id plus vm_name where known — the free-form per-VM error text is not stored.

Without a key, a commit against a target whose failover job already reached a terminal commit state answers 409 with a structured body, not a generic error: code, commit_state, already_committed, failover_job_id, terminal_status, and the promoted identifiers the original commit persisted.

already_committed is the only signal you may read as "my irreversible operation happened". A wedged commit — commit_state: failed — is also a 409, and treating it as success records an incomplete migration as finished.

14.4 After a timeout on a destructive operation

Never blindly replay. Follow this, in order:

15. Operational Checklist And Troubleshooting Boundaries

15.1 Before you go live

• The provider credential is in a server-side secret manager, and nowhere else.
• It holds only the permissions its component needs (section 6).
• Separate credentials for onboarding, replication, rehearsal, live cutover and commit.
• Its expiry is a deliberate choice, and you have a diarised rotation procedure.
• Your orchestrator persists every identifier in section 11.3.
• Idempotency keys are persisted before the request that uses them.
• Your orchestrator survives a restart mid-workflow, tested.
• Commit is gated behind an explicit, recorded business decision.
• No code path parses a human-readable error message.
• 404 is handled as "absent or not mine", not as "deleted".
• 423 is handled as "retry later", not as failure.
• Every destination credential has been tested with testStoredVaultCredential.
• Licence capacity has been checked, and your reporting honours section 13.2.

15.2 Common first-integration problems

15.3 Where support begins

Your automation owns everything up to and including the API call and its response handling. Escalate to Sendense support when:

• an operation returns 5xx repeatedly for the same request;
• a job sits in a non-terminal state well past any plausible duration;
• commit_state is failed — a wedged commit is operator-actionable and the promoted VM is retained, but cleanup may be incomplete;
• cleanup_incomplete is true and the leftover resources matter;
• licence figures disagree with your own records after honouring section 13.2.

Collect and quote: the operationId, the identifiers involved, the status code, the structured error fields, the timestamp, and the token_prefix of the credential used. Do not send the credential value.

16. The Generated Provider API Profile

SHA_MULTI_TENANT_CSP_OPENAPI.yaml is the machine-readable contract for everything on this page: 97 operations across 79 paths with 179 schemas. It is served from the appliance alongside the full API specification.

It is a projection of the appliance's canonical contract, not a separate API — same host, same /api/v1/... paths, same schemas, same behaviour. It is generated from the same sources by the same generator, so a route cannot drift between the two. If an operation is absent from it, that is deliberate: it is web-interface surface, a machine callback, support diagnostics or staff fabric, and it is not part of what a provider automates against.

Read it for: exact request and response schemas, every documented status code, per-operation permissions, the retry class of every DR-plane create and trigger — see §14.2 for the SHA-plane operations that carry none — and the examples this page draws on.

16.1 Credentials the profile accepts

The profile declares exactly two security schemes:

• providerApiToken — the provider service credential. This is what your automation holds.
• sessionBearer — a human session, for the operations in section 4.4.

A site API token is not accepted anywhere in this profile, and no operation in it declares one. Do not attempt to drive this workflow with one.

16.2 Known limitations

Recorded honestly, so you design around them rather than discovering them:

1. Discovery returns counts, not context identifiers. getDiscoveryExecution serialises the execution record — totals, timings, error counts — and unifiedDiscoverVMs returns platform-level VM records. Neither carries context_id. Read it from listVMContexts (section 7.7).
2. There is no filtered target lookup. listDRTargets takes no query parameters in this profile. Resolve a target from its assignment or from listDRVMReplicationStatus.
3. The reconciliation reads are unpaginated and unfiltered. listTenants, listSites and listUsers (§7.11) return the whole estate in one page and take no query parameters, and there is no per-user detail read (getUserSites and getTenant cover the bindings). listVaultCredentials is not in the profile: the per-tenant credential view is getTenant's credentials[], identity only.
4. Licence capacity has no tenant attribution, and consumption attribution is best-effort with the caveats in section 13.2.
5. Sendense does not call you. No outbound webhooks, no OAuth client-credentials flow, no action pipelines (section 12.1).
6. No automatic credential rotation (section 4.3).
7. Site creation has no idempotency protection, site names are unique appliance-wide, and a duplicate answers 500 rather than 409 (section 7.2). Prefer createTenant, which is idempotency-key protected.
8. Rollback confirmation is tied to multi-tenant enforcement. On an appliance running with site scoping enforced — the posture this guide is written for — confirm and reason are required on every rollback-initiating entry point: rollbackDRTargetFailover, rollbackDRPatternFailover and retryDRRollbackJob (section 8.2). cleanupDRRollbackJob is graded on dr.rollback but takes no confirmation. On a single-tenant appliance with scoping off they are accepted and not enforced, exactly as for commit and live failover, so your orchestrator should send them unconditionally rather than depending on the appliance to refuse.
9. Some response shapes are not modelled in the contract — the per-VM attach override (vms), the per-VM rows of listDRVMReplicationStatus, the pattern RPO targets[] and the per-disk health array. The fields this guide names are correct, but you cannot generate a typed client for them. (rollbackDRPatternFailover and commitDRPatternFailover were in this list and are now modelled: their rolled_back/committed and failed are ARRAYS, not counts, and both return job_id. The three cleanup routes are modelled on the compensation result they actually return, and the rollback refusals on the code/confirm_with and status/eligible_status bodies their descriptions name. Since 2026-08-22 the tenant delete plan, conflict and result, the orphaned-target plan and removal body, the pattern delete lifecycle, delete_requested_at, the pattern compute fields and the Site tenant_id are modelled too.)
10. A pattern delete — and a VM detach — destroys the controller VMs of the targets the pattern owns. There is no keep-controllers option; destroy_vms=true is accepted and does not change the outcome, and no profile operation preserves a controller for an owned target in a replication state. The one class neither ever destroys is a target a failover touched: a committed one is released (its VM is production; the record goes to the orphaned-target inventory), a failed-over, awaiting-commit or rolling-back one is named as blocked with its remedy and the pattern waits (§17.3). Only a target the teardown refused (§17.4) and a committed record it released outlive their pattern — the first as an unmanaged leftover, the second as the production VM's bookkeeping in the orphaned-target inventory.
11. The provider lifecycle rests on settings.write. Minting a credential, creating and deleting tenants, sites and stored credentials all sit on the same broad permission, which also reaches appliance administration (§5.2, Examples 1 and 7). A narrower permission for the lifecycle is tracked separately; until then, size the credential's controls for the permission it holds, not for the job it does.

17. Offboarding A Tenant

What provider automation creates it can also remove, from the same profile, in a documented order, with every refusal naming its remedy. Offboarding is destructive at two points — destroying a tenant's controller VMs, and deleting the tenant — and the appliance refuses to do the second while anything the tenant owns is still protected, so the order matters and the dry run is not optional. Example 7 is the credential.

The order, as observed end to end on a development appliance:

1. inspect the tenant, its sites and its users (§17.1);
2. tenant delete dry run — expect conflicts while workloads are protected (§17.2);
3. detach or destroy the protected workloads — the pattern teardown (§17.3);
4. resolve orphaned targets where the teardown refused (§17.4);
5. wait for the pattern and its targets to disappear (§17.3);
6. remove remaining credentials and logins only where separately required (§17.5);
7. tenant delete dry run again — expect no conflicts;
8. delete the tenant (§17.6);
9. decide whether to retain or delete the now-ungrouped sites (§17.7).

What this procedure does NOT say. An older refusal text asked you to "disable their protection (or move them to another site)" before deleting a tenant. There is no such operation. The way a workload stops being protected is §17.3 and §17.4: remove it from its pattern (destroying or resolving its target) and let its backups expire or delete them. The current refusal names those.

17.1 Inspect

GET /api/v1/tenants/{id} — getTenant — settings.read

Read the tenant (§7.11): its sites[], its users[] with their site_ids, and its credentials[] by identity. Read each site with getSite and note its tenant_id. Then the DR inventory for the tenant's sites: listDRPatterns and listDRPatternVMs for every pattern, listDRTargets, and listDROrphanedTargets — targets no pattern assignment claims are what block a site delete and what §17.4 resolves. Persist the pattern and target ids you find; you will poll them.

17.2 Dry-run the tenant delete

DELETE /api/v1/tenants/{id}?dry_run=true — deleteTenant — settings.write

The plan is returned and nothing is written. Send dry_run as one of 1, true, yes, on (or 0, false, no, off); any other value is refused 400 rather than guessed at — the appliance will not decide for you whether a preview or a real delete was meant.

While any of the tenant's sites still holds a protected VM, the plan carries conflicts[]:

{
  "tenant_id": "t_01H8ACME",
  "name": "Acme Corp",
  "ungroup_sites": ["site-acme-dr"],
  "detach_pool_site_ids": ["site-acme-dr"],
  "cascade_deletes": [{"site_id": "site-acme-dr", "label": "site-scoped vault credentials", "count": 1}],
  "delete_logins": [{"user_id": "u-0001", "email": "[email protected]"}],
  "keep_logins": [],
  "revoke_site_tokens": [{"id": "tok-0001", "name": "tenant-automation", "site_ids": ["site-acme-src"]}],
  "conflicts": [{
    "field": "remove_sites",
    "reason": "site \"site-acme-src\" still holds 3 protected virtual machines (app-01, app-02, app-03); stop protecting them first — remove them from their replication pattern (or delete the pattern), remove any orphaned replication targets, and delete or expire their backups — then delete the tenant"
  }]
}

A real delete with conflicts answers 409 with the same conflicts[] and changes nothing. Two other 409 shapes exist: a tenancy-preflight failure (field: tenancy or tenant_id — the appliance's multi-tenant entitlement is not licensed or not activated; offboarding runs through the same planner as editing, so restore the licence first), and an in-flight-work refusal {error, appliance_id, impact} when an appliance on the tenant's sites is still busy. "Protected" is judged from evidence — a replication target, membership in a live pattern, a backup or stored recovery point, a job, work in flight — never from a flag, so a VM that was discovered and never protected does not count, and one that is counts until its evidence is gone. Each conflict names the site, up to five VMs and the count, and the remedy.

Read the rest of the plan as the statement of what a successful delete will do: every owned site in ungroup_sites (they survive, §17.6), the pool delegations it detaches, what the cascade destroys per site (cascade_deletes, delete_credentials, delete_appliances), the logins it deletes (delete_logins — whose entire binding estate lies inside this tenant) and the logins it keeps (keep_logins — also bound elsewhere; they lose only their in-tenant bindings).

17.3 Tear down the replication pattern — one request, one deletion

DELETE /api/v1/dr/patterns/{pattern_id}?destroy_vms=true — deleteDRPattern — dr.replication.write

Deleting the pattern is how its VMs stop being protected. Send destroy_vms=true to record your consent explicitly; a pattern delete destroys the controller VMs of every target it owns that is in a replication state, regardless (there is no keep-controllers option — to keep one, use §17.4's route instead). It never destroys a target a failover touched: a failed-over controller is, or is about to be, the tenant's workload. A committed one is released (below); any other is named as blocked with its remedy and the pattern waits.

What happens, and what you poll:

• A pattern that owns no targets is deleted at once: 200, controller_vms_action: not_applicable — unless a membership is still being provisioned (its controller is being built and no target row exists yet). Then the delete answers 202 destroying with a message saying the provisioning run is being stopped; the pattern is disabled, the run unwinds, and the pattern deletes itself once it has. Poll getDRPattern to 404 as below.
• A pattern that owns targets answers 202 controller_vms_action: destroying. The delete is recorded on the pattern — getDRPattern now carries delete_requested_at and enabled: false — one controller destroy is queued per target, and the pattern is deleted automatically as soon as the last target it owns is gone. Poll getDRPattern until it answers 404; then poll listDRTargets for the tenant's sites until the targets are gone too (they normally go first). On the development appliance five controllers took about two and a half minutes. There is no second delete to send, and nothing else to do.
• A replay while cleanup runs is accepted again (202, the original delete_requested_at kept) — repeat the call after a lost response without fear. A replay after completion answers 404, as does every read: that is your terminal state.
• 202 with deferred: true means the delete was recorded but a destroy could not be queued on this request (details says why — a provider hiccup, a membership link that needed repair). The appliance re-offers the owned targets on its own sweep; nothing more is needed from you, and the pattern still ends at 404 — with one exception. The sweep never destroys a target locked by a running operation (a sync, a failover, a rollback, a commit, a cleanup, a controller update); it is offered again once the lock is released, which for a sync or a controller update means the next tick, and for a failover or a commit means the target has moved into a cutover state — the case below. The request path offers targets in turn and stops at the first it cannot queue (202 deferred, naming it), so a replay may be needed once a lock clears; the sweep does not stop.
• A target a failover touched is never destroyed by a pattern delete — on either path. A replication target whose status is not a replication state — failover_pending, failed_over, pending_commit, committed, rollback_pending or rolled_back (the target-side names; §11.2) — is handled in one of two ways. A committed target is released, not destroyed: its VM is production now, so the delete removes only its membership in the pattern (what a commit's own clean-up step does) and leaves the record to the orphaned-target inventory, where listDROrphanedTargets now lists it and §17.4's route removes it with its confirmations — the promoted VM is kept. Released targets appear under released_targets; when nothing else is owed, the pattern is deleted on that very request (200). When something else still keeps the pattern after a release — most often a second membership row of the same VM (a failed earlier attempt) that still claims the released record, which makes it a stranded orphan — the answer is 202 destroying with details saying exactly what and what to do (for the stranded record: remove it through §17.4; the pattern then completes on its own). Act on details before polling: that 202 does not reach 404 by itself. Every other cutover state is named under blocked_targets with its status and a remedy, and no retry_endpoint (a destroy retry is the wrong act); the pattern stays recorded-for-deletion and completes once they are gone. The remedies: a rehearsal (failed_over) — roll it back (§8.2), the target returns to ready and the next delete (or the sweep) destroys it; an uncommitted live failover (pending_commit) — commit it (it is then released) or roll it back; a failover or rollback in progress — wait for it. A target whose row the appliance could not read is named the same way with status: unknown and nothing done to it — replay the delete.
• 202 partially_destroying (or 409 blocked when nothing could be queued) names the targets whose cleanup could not start — a delete_failed target with last_error and its exact retry_endpoint (retryDRTargetDestroy), or a target a failover touched with its remedy (above); released_targets, when present, lists the committed ones that were released alongside. Retry or resolve them; the pattern completes when they are gone. A target whose cleanup cannot succeed at all is an operator judgement outside this profile: an administrator abandons its record from the appliance (the destination resources are then removed by hand), and the pattern completes after that too — so a teardown stuck on a dead destination ends at support, not in a retry loop.
• 409 stranded_targets refuses the whole delete: a target the pattern claims but does not name carries failover history, and a pattern delete is not consent to destroy the record of a cutover. Resolve each named target through §17.4, then send the delete again.
• While the delete is pending, attaching a VM or changing the pattern answers 409 PATTERN_DELETE_PENDING; the delete wins.

To keep the pattern and remove only some VMs, removeDRPatternVM / bulkRemoveDRPatternVMs detach them — destroying their targets the same way, under the same rule: a committed target is released (200, its VM untouched), a failed-over, awaiting-commit or rolling-back one is refused 409 with the target named and its remedy. A 409 can carry released_targets too — a VM with one committed target and one stuck target has the first released and the second refused on the same call; the membership is gone, the refusal is about the second target. There is no API path that keeps a controller VM for a target in a replication state that a pattern owns: every detach and every pattern delete destroys it. The orphaned-target route (§17.4) applies to targets the teardown refused (stranded ones with failover history) and to the committed records a delete or detach released — and discard_provider_resources there acknowledges that a controller is left behind unmanaged; it does not preserve it as something Sendense manages.

17.4 Resolve orphaned targets: plan tokens and confirmations

GET /api/v1/dr/orphaned-targets — listDROrphanedTargets — dr.replication.read GET /api/v1/dr/orphaned-targets/{id} — getDROrphanedTarget — dr.replication.read POST /api/v1/dr/orphaned-targets/{id}/remove — removeDROrphanedTarget — dr.replication.write

An orphaned target is a replication target no pattern assignment claims — typically one left by a committed or rolled-back failover, a removed VM, or a teardown that refused over failover history. They block a site delete, so an offboarding has to clear them.

Read the plan first and immediately before removing:

{
  "target_id": "tgt-0000",
  "status": "rolled_back",
  "failover_jobs": 1,
  "sync_jobs": 4,
  "pure_garbage": false,
  "destroys": ["1 failover job and its state journal", "4 sync job records"],
  "confirmations_required": [
    {"key": "discard_failover_history", "reason": "..."},
    {"key": "discard_provider_resources", "reason": "..."}
  ],
  "removable": true,
  "plan_token": "v2:9f2c..."
}

Then remove with a flat body: the plan_token the plan returned, plus true for every key in confirmations_required:

{"plan_token": "v2:9f2c...", "discard_failover_history": true, "discard_provider_resources": true}

What the confirmations mean, because they are the two things you are agreeing to lose:

• discard_failover_history — the target's failover jobs and their state journal are permanently deleted: who ran each cutover, when, and whether it was committed or rolled back. It cannot be recovered.
• discard_provider_resources — the removal does not touch the destination cloud. The controller VM and its replica volumes stay exactly as they are, unmanaged; this confirmation is your acknowledgement of that, not an instruction to delete them. Remove them by hand, or had already.

A target with nothing to lose (pure_garbage: true, no confirmations) removes with an empty body. Every refusal is 409: confirmations_required[] names what you did not confirm; plan_changed: true means the token is missing or stale — what the removal would destroy changed since you read the plan (a sync ran, a lock moved), so read it again and confirm against what it says now; the target stopped being an orphan while you confirmed; or removable: false with blockers[] (a live pattern still claims it — detach there instead). Every attempt is audited with the credential as actor; when the audit log cannot record the attempt the removal answers 503 and does not proceed. A second remove answers 404.

17.5 Credentials, tokens and logins — usually nothing to do

The tenant delete cascades the stored credentials scoped to the tenant's sites (cascade_deletes / credentials_deleted, each through the vault with its own audit row), revokes every site API token whose scope touches any of the tenant's sites (revoke_site_tokens in the plan, site_tokens_revoked in the result — a token left alive would quietly re-arm the moment one of the ungrouped sites was adopted by another tenant), and deletes the logins whose whole estate lay inside the tenant (logins_deleted, each with its own user_deleted audit row and session revocation). Two cases need a separate call:

• a login the delete will keep (keep_logins: also bound to sites outside the tenant) that should nevertheless go — DELETE /api/v1/users/{id} (deleteUser, users.write), which answers 200 {"success": true}, refuses the last administrator with 400, and answers 400, not 404, for an absent user — not idempotent;
• a stored credential scoped outside the tenant's sites, or one you want gone before the tenant — DELETE /api/v1/vault/credentials/{id} (deleteVaultCredential, settings.write), a hard delete with no in-use check, answering {"success": true, "data": {"id", "deleted": true}}, 404 for an unknown id (and 404 from the scope gate for one outside the caller's reach).

17.6 Delete the tenant

DELETE /api/v1/tenants/{id} — deleteTenant — settings.write

Dry-run once more (§17.2) and expect conflicts absent; the plan now lists everything the delete will do. Then delete. The result:

{
  "tenant_id": "t_01H8ACME",
  "name": "Acme Corp",
  "sites_ungrouped": ["site-acme-src", "site-acme-dr"],
  "pool_detached": ["site-acme-src", "site-acme-dr"],
  "cascaded_site_ids": ["site-acme-src", "site-acme-dr"],
  "cascade_deletes": [
    {"site_id": "site-acme-src", "label": "site-scoped vault credentials", "count": 1},
    {"site_id": "site-acme-dr", "label": "site-scoped vault credentials", "count": 1}
  ],
  "logins_deleted": 1,
  "logins_kept": 0,
  "appliances_deleted": 0,
  "credentials_deleted": 2,
  "site_tokens_revoked": 1
}

What a tenant delete retains — deliberately. The tenant's sites survive, ungrouped: their tenant_id becomes null, their user bindings are revoked and their SNA-pool delegation detached, but the site rows stay and so does their discovered inventory — every VM context discovered in them is still there after the tenant is gone. This is the safe default: a site may be regrouped under another tenant (createTenant can adopt ungrouped sites), and deleting discovered inventory is a separate, explicit act (§17.7). What goes with the tenant is its grouping, its site-scoped configuration (credentials, vCenter and CloudStack sources, appliances) and the logins that existed only for it.

The delete is audited as typed security_audit_log rows — tenant.site_ungrouped, tenant.site_cascaded (with what was destroyed) and tenant.deleted (with the summary) — with the calling credential as actor. A second delete answers 404; so does getTenant. The sites are not gone, and listSites shows them with tenant_id: null.

17.7 Decide about the sites

DELETE /api/v1/sites/{id} — deleteSite — settings.write

The now-ungrouped sites are yours to keep or remove. Deleting a site removes its discovered inventory — every VM context it holds — and is refused 409 while anything in it is still protected (the same evidence §17.2 judges by), while an orphaned replication target still names it (§17.4), while appliances, stored credentials, sources, physical machines or users are still bound to it, while another site delegates its SNA routing to it, and always for the reserved SNA-pool site. The plain-text body names the dependents. Answers 200 {"message": "Site deleted successfully"}; a replay answers 404 under enforced scoping. Keep the sites when the customer may return or another tenant will adopt them; delete them when the inventory must not linger on the appliance.

Persist: the terminal states you observed — the pattern's 404, each orphaned target's removal, the tenant delete result, each site delete — with timestamps, before closing the decommission. Then revoke the credential you minted for it (§4.5).