Backups and recovery

Provider uncertainty and recovery

SendGrid does not deduplicate a Mail Send request using Lamp's key. Lamp therefore stores a durable, envelope-hashed send claim before calling it. An accepted replay returns its original correlation ID without sending again. Explicit HTTP 429 rejections may retry. Network interruption, timeout, an uncertain 5xx or a Worker crash after claiming cannot safely resend: that notification becomes visibly failed/unconfirmed and retains correlation for a later signed receipt. The receipt can reconcile it to delivered, bounced or complained. Inspect provider evidence before any manual remediation; never clear the send ledger to force a resend. The ledger contains opaque hashes/IDs and state, not message bodies or credentials.

Resend keeps its existing frozen-envelope idempotency behavior. In either provider, queued means pending or accepted without a delivery receipt; it does not mean mailbox delivery. Retain the D1/R2 archive and the private deployment inputs for recovery. No remote restore or provider acceptance is implied by local verification.

Portable backups and restore

Export includes every application and owner sidecar table, original R2 bytes, HTTP metadata and custom metadata. The database reads run in one D1 transaction. Pause operator writes and scheduled work for a coordinated backup: D1 and R2 do not share a transaction, so the archive must not be treated as a cross-service point-in-time backup while data is changing. A failed export leaves no completed manifest; use a new empty output directory for the next attempt. Archive files are private because they include subscriber data, sessions and frozen mail.

pnpm run export:cloudflare -- --remote --environment production --out /private/backups/lamp-first
pnpm run export:cloudflare -- --persist-to /private/local-state --out /private/backups/lamp-local
pnpm run import:cloudflare -- --from /private/backups/lamp-local --persist-to /private/empty-state

Keep the owner's exact source revision alongside the backup. Import checks the archive checksums and core/owner migration hashes before any destination mutation. Owner migrations in migrations/ext/ use their own ledger and run after core migrations in development, setup and restore. Preserve the contents and identity of already-applied migrations; add and verify forward migrations for later schema changes, whether they change core or owner-specific data.

Restore into a new empty database and bucket. A partial or interrupted restore is never published by the import command, and rerunning against a populated location is rejected. Retain that location for diagnosis and retry into a new empty location using the unchanged archive. Do not erase an existing deployment to make it fit the import precondition. Keep the original token/signing secrets and verify export → import → export equality, native sign-in, public content and a publication before routing traffic to the restored deployment.

Dependency failure

Public reads use versioned R2 artifacts without querying D1. The Worker retains successful responses in its local edge cache for the one-hour outage window and marks that response as last known state when snapshot storage reads fail. A cold edge cache with inaccessible storage cannot recover bytes it has never seen; retain the original data and verify the tested deployment's warmed-cache path. /health checks D1, R2 and Queue reachability separately and must return503 during a dependency failure. A successful public response is not a healthy-backend proof. For managed installations (RER_INSTALLATION_ID), /health also completes pending public snapshot work before reporting readiness, including the initial migrated state in previews where cron is disabled. A failed snapshot build returns 503. This readiness step does not dispatch scheduled incidents, maintenance or mail. A Cloudflare-wide outage needs separately operated hosting; this release does not provision an independent standby or make that deployment claim.

For a remote restore, choose a new environment name and replacement host inputs, then run pnpm run setup:cloudflare -- --environment recovery --restore-from /private/backups/lamp-first. This provisions an empty target and imports before initialization; it never publishes a Worker. Inspect the restored archive and record equivalence before rer deploy --environment recovery. Keep original host routing active until the replacement has passed its application and provider checks. Passing --restore-from directly to deploy performs the same restore before verification and publication, and refuses a destination that already contains data.

Retained public snapshots

SNAPSHOT_CACHE binds the Worker-exported SnapshotCache SQLite Durable Object. Wrangler creates it through migration snapshot-cache-v1; no new secret, external account or manual namespace ID is required. Setup-generated and local configs carry the binding and migration. The object is a cache of one complete published R2 generation, replaced atomically and monotonically. It is necessary for PAGE-006/007: KV can negatively cache a missing value after first publication and cannot guarantee an immediate cold-edge read. Public serving never reads D1.

R2 remains authoritative during ordinary reads. On R2 failure the replica serves last-known HTML, Atom, widget and their generated assets with generation timestamps; /health independently reports dependency failures. A never-published page stays 503. A simultaneous Worker/replica outage is not claimed to be served by that unavailable runtime; the advertised downstream stale-if-error policy still applies.

Each R2 generation includes retained.json, so existing portable archives contain the complete replica input. Restore into a fresh environment as usual; the new Worker primes its empty replica from the restored R2 bundle. Legacy archives that predate the bundle are rebuilt by the minute scheduler from restored source. Deployment waits for X-Lamp-Snapshot-Retained: true on the public status response before declaring its smoke check complete, and fails visibly if readiness does not arrive within 90 seconds. The cache is derived, not separate owner data; do not copy a Durable Object namespace across owner installations.

Publication persists every immutable R2 artifact and its bundle, acknowledges the complete replica, then advances the R2 selector. A rejected replica write cannot expose a first or replacement R2 selector. Transient retries reuse the exact same rendered version and bytes, including when a replica acknowledgment is lost. If replica retention succeeds but the R2 selector write subsequently fails, the previous selector stays in place and the job remains failed/pending. During an R2 outage the replica may serve that newer complete, already R2-persisted generation as last-known state; it does not invent or partially compose a status. A later retry repairs publication. Source work is completed only for its selected version.