Harden backup and restore: recursive discovery, content fidelity, safe targets (card 71)

docs/FULL_BACKUP.md

@@ -1,16 +1,25 @@
-# Full top-level Firestore backup
+# Full recursive Firestore backup and recovery
 
-Existing operator tooling integrated from `932f730` (cards 64–65); no fresh live recovery is claimed by this integration.
+Operator tooling for recursive disaster-recovery backups, integrity validation, and safe content-fidelity restore across all Firestore collections. Distinct from [MEMORY_ARCHIVE.md](MEMORY_ARCHIVE.md), which archives only `memory_vectors` without vectors and re-embeds during restore.
 
-As-is clone of every top-level Firestore collection discovered by
-`listCollections()`. Distinct from [MEMORY_ARCHIVE.md](MEMORY_ARCHIVE.md),
-which archives only `memory_vectors` and strips embeddings.
+Prod `my-brain-88870` is read-only. Restore refuses that project id.
 
-Inventory is live discovery. Do not hardcode collection names.
+## Format and discovery
 
-Vectors and timestamps round-trip via `__fs` tags. No re-embed.
+Backups use manifest schema `metacortex-firestore-full-backup-v2`.
 
-Prod `my-brain-88870` is read-only. Restore refuses that project id.
+Discovery is recursive: it discovers top-level collections, subcollections, and subcollections under phantom parents (documents that have subcollections but no parent document payload).
+
+Each collection path is dumped to a deterministic NDJSON file named `encodeURIComponent(path) + ".ndjson"`. Each entry in `manifest.json` records:
+
+- Collection path (odd number of segments).
+- Document count.
+- NDJSON file name and size in bytes.
+- Exact SHA256 hex checksum of the NDJSON file bytes.
+- Vector dimensions observed across the collection documents.
+- Archive-level embedding space metadata (model and dimensions), or null when no vectors are present.
+
+The manifest is written last so an interrupted backup never leaves an incomplete manifest pointing to partial snapshots.
 
 ## Backup
 
@@ -20,22 +29,25 @@ npm --prefix functions run backup:firestore-full -- \
   --archive-dir /path/to/archive
 ```
 
-`--dry-run` lists collections and counts without writing.
+Pass `--dry-run` to discover paths and counts without writing any files.
 
-Each run calls `listCollections()` on the source, then rewrites every
-collection NDJSON file and `manifest.json`. This is a full snapshot, not an
-incremental delta. NDJSON files for collections that disappeared from the
-source are removed. Non-NDJSON files in the archive directory are left alone.
+Backup discovers collections recursively, verifies that dumped document counts match live counts, encodes typed Firestore values (Timestamps, VectorValues, GeoPoints, Buffers, DocumentReferences, and non-finite numbers), computes checksums, writes NDJSON files, writes `manifest.json`, and cleans up stale NDJSON files from prior runs.
 
-## Restore
+## Archive validation before writes
+
+Before any target mutation or live restoration, `validateArchive` verifies the archive on disk:
 
-Dry-run by default. `--write` is required to mutate the target.
+1. Manifest readability, valid schema (`metacortex-firestore-full-backup-v2` or legacy `v1`), and consistent path ordering.
+2. File existence, derived file names, and byte SHA256 checksum matches against manifest entries.
+3. NDJSON JSON syntax, non-empty string `_id` presence, id uniqueness, and document count equality.
+4. Structural decoding of all documents and tags via `decodeDocument`. Unknown `__fs` tags or malformed tag payloads are rejected immediately.
+5. Observed vector dimensions equal manifest entries, and observed vector dimensions match the manifest embedding space.
 
-`--write` converges each archived collection to the snapshot:
+Validation failures abort immediately with a complete list of detected problems before any write operations occur.
+
+## Restore
 
-- upsert every archive document by id (create or overwrite)
-- prune target documents whose ids are absent from the archive
-- create collections that exist in the archive but not yet on the target
+Restore defaults to dry-run mode. `--write` is required to mutate the target database.
 
 ```bash
 npm --prefix functions run restore:firestore-full -- \
@@ -44,34 +56,39 @@ npm --prefix functions run restore:firestore-full -- \
   --write
 ```
 
-`--clear-target-collections` is optional. It wipes each archived collection
-before the upsert. Converge does not need it; prune already removes extras.
+### Safety and target guards
 
-Restore does not delete target collections that are absent from the archive.
-The inventory gate checks source-to-target parity for source collections only.
+- **Separate-target default:** Restoring directly into the source project is refused unless `--target-prefix <prefix>` is provided or explicit replacement confirmation is passed.
+- **Upsert-only default:** Restore defaults to safe upserts (create or overwrite documents by id). Existing extra documents on the target are left untouched.
+- **Explicit replacement confirmation:** `--prune` (deleting target documents whose ids are absent from the archive) and `--clear-target-collections` (wiping collections before restore) require `--confirm-project <target-project-id>`.
+- **Target prefixing:** `--target-prefix <prefix>` prepends `<prefix>` to the root collection segment of each path (e.g. `memory_vectors/doc1/notes` becomes `test_memory_vectors/doc1/notes`), allowing safe testing inside an existing project.
+- **Vector compatibility guard:** If the archive contains vectors, restore checks that target embedding model and dimensions match the archive. On mismatch or unknown embedding space, restore aborts before any write and recommends portable import (`npm run restore:memories`), which re-embeds records with the current model into a separate collection. Passing `--accept-unknown-embedding-space` overrides this check for legacy v1 archives.
 
-## Re-run semantics
+### Interruption and recovery
 
-What happens if more docs (or collections) are added on prod and backup runs
-again:
+If a write failure occurs during restore, the command halts and prints `RESTORE INCOMPLETE: <underlying error>. re-run the same command to finish; upserts are idempotent`.
 
-- New collections on prod: the next backup includes them via a fresh
-  `listCollections()`. Restore creates them on QA when it upserts their docs.
-- New or updated docs: the backup rewrites the collection NDJSON. Restore
-  upserts those ids on QA.
-- Deleted docs on prod: they are absent from the new snapshot. Restore prunes
-  those ids from the matching QA collection.
-- Deleted collections on prod: their NDJSON is dropped from the archive.
-  Restore does not delete the leftover QA collection. The gate still passes
-  because it only requires source collections to exist on the target with
-  matching counts.
-- Always re-run the inventory gate after backup plus restore. Success only
-  when the gate exits 0.
+Because document writes are keyed by id, re-running the same restore command resumes and completes recovery safely.
+
+### Content-fidelity verification
+
+After writing, restore runs verification across every archived collection:
+
+- Reads back the stored documents from the target.
+- Verifies document equality and field content against the archive using stable JSON stringification.
+- For v2 archives, verifies that the target content SHA256 matches the archive SHA256 checksum.
+- When prune is not requested, extra target documents are reported as `extra_target_docs` rather than a failure. When prune is requested, extra target documents cause a verification failure.
+- Reports differing document IDs (missing, extra, changed, capped at 20 IDs each).
+
+A restore run is considered complete only when verification passes for all collections.
+
+## Legacy v1 archive compatibility
+
+The restore engine supports legacy v1 manifests (`metacortex-firestore-full-backup-v1`). Legacy manifests are normalized in memory, collection IDs are treated as paths, checksums and vector dimensions are computed during validation, and per-document equality verification is performed using canonical stable stringification.
 
 ## Inventory gate
 
-Exit 0 only when every collection from source `listCollections()` exists on
-the target with equal `count()`.
+The inventory gate verifies recursive collection discovery, matching document counts, and matching content SHA256 hashes across source and target projects:
 
 ```bash
 npm --prefix functions run gate:firestore-inventory -- \
@@ -80,13 +97,13 @@ npm --prefix functions run gate:firestore-inventory -- \
   --out /path/to/inventory-gate.json
 ```
 
-A backup is not complete until this gate exits 0. Prior #60/#62 runs hardcoded
-four collection names and missed live collections.
-
-## Current limits and future recovery work
-
-The implementation does not recursively discover subcollections, establish one consistent read timestamp, or compare content hashes. The inventory gate checks source collection presence and counts; it reports target-only collections without failing on them. Equal counts are not proof of content equality. The script name "full" denotes its current top-level inventory scope, not a complete recursive disaster-recovery guarantee.
+The gate succeeds only when all source collections exist on the target with equal document counts and identical content SHA256 checksums.
 
-Restore is dry-run unless `--write` is supplied. With `--write`, it prunes extra documents in archived target collections before upserting. That can leave a partial target if later writes fail. The hardcoded refusal of the maintainer's production project is not a general source/target protection for other installations. Always inspect the explicit target and restore to isolated QA first. Use a dedicated archive directory separate from portable memory archives.
+## Known limitations
 
-Retain historical inventory evidence from cards 64–65, but strengthen snapshot consistency, archive integrity, source/target guards, and content/lineage verification under RECOVERY in the [roadmap](../metacortexplan.md). These are future work, not changes introduced by this baseline. Physical pruning remains an operator recovery action, not an agent or product deletion feature.
+1. Firestore integer vs double distinction is not preserved by the Admin SDK read path.
+2. DocumentReference values are restored with their original absolute paths and are not remapped by `--target-prefix`.
+3. Field transforms, composite indexes, rules, TTL policies, and security rules are not part of the document archive.
+4. Verification covers Firestore documents, not Cloud Storage objects or Firebase Authentication users.
+5. Restore is not atomic across collections. In the event of an interruption, resume by re-running the restore command.
+6. Live production restore was not exercised.

docs/MEMORY_ARCHIVE.md

@@ -1,10 +1,10 @@
 # Portable memory archive
 
-Integrated from completed archive work on `worktree-memory-archive-task1` (`932f730`, cards 58–65). This is existing operator tooling, not the future owner dashboard. See [full backup](FULL_BACKUP.md) for top-level operational collections and [the roadmap](../metacortexplan.md) for recovery hardening.
+Integrated from completed archive work on `worktree-memory-archive-task1` (`932f730`, cards 58-65) and hardened under RECOVERY (card 71). Distinct from [FULL_BACKUP.md](FULL_BACKUP.md), which captures all collections with original vectors and types.
 
 ## Format and behavior
 
-Deterministic NDJSON preserves IDs, canonical/retrieval text, supported metadata/provenance, and media descriptors. It includes deprecated memories. It omits embeddings and re-embeds every record during restore. Raw image bytes are never in the memory store; separately stored artifacts are not copied. Audit/fingerprint/evaluation collections are outside this portable format.
+Deterministic NDJSON preserves IDs, canonical/retrieval text, supported metadata/provenance, and media descriptors. It includes deprecated memories. It omits embeddings and re-embeds every record during restore with the current model. Raw image bytes are never in the memory store; separately stored artifacts are not copied. Audit/fingerprint/evaluation collections are outside this portable format.
 
 Use a dedicated **private** Git archive repository. The archive is plaintext. Never point it at the source repository or a folder with unrelated staged changes. Use a different directory from a full Firestore backup: the manifest formats are distinct.
 
@@ -17,27 +17,40 @@ npm --prefix functions run backup:memories -- --project <project-id> --archive-d
 npm --prefix functions run backup:memories -- --project <project-id> --archive-dir <private-archive-path> --no-push
 ```
 
-Without `--dry-run`, this writes archive files and commits them. Without `--no-push`, it also pushes to the archive repository. No scheduling or upload is enabled by baseline integration. Operators can explicitly arrange a private schedule; cross-platform scheduling/setup is future work.
+Without `--dry-run`, this writes archive files and commits them. Without `--no-push`, it also pushes to the archive repository.
 
-The 10% document-count-drop guard rejects suspicious truncation; `--force` overrides it after operator investigation. `METACORTEX_ARCHIVE_DIR` can select the archive path. The deployment preflight warns about portable manifests more than three days old.
+The 10% document-count-drop guard rejects suspicious truncation; `--force` overrides it after operator investigation. `METACORTEX_ARCHIVE_DIR` can select the archive path.
 
-## Restore
+## Restore and validation
 
-The script requires `--collection` and defaults to dry-run. Use an explicit scratch project/collection:
+The restore script requires `--collection` and defaults to dry-run mode:
 
 ```bash
 npm --prefix functions run restore:memories -- --project <target-project-id> --archive-dir <private-archive-path> --collection memory_vectors_restore
 ```
 
-After inspecting the archive and target, add `--write` to re-embed/write. IDs and supported metadata are preserved. The script requires the current config environment, including API-key access for re-embedding; it does not automatically resolve Secret Manager. Provide secrets privately. Ensure the target collection has a matching vector index before querying it.
+### Pre-restore semantic validation
+
+Before any Firestore connection is opened or any writes occur (in both dry-run and write modes), the entire archive is validated via `validatePortableArchive`:
+
+- `schema_version` must equal `ARCHIVE_SCHEMA_VERSION`.
+- Document count in `memories.ndjson` must match `document_count` in `manifest.json`.
+- Every memory must have a non-empty string `id`, non-empty string `content` and `retrieval_text`, and a valid `metadata` object with string `module_name`, string `branch_state`, and finite numeric timestamps `created_at` and `updated_at`.
+- Document IDs must be unique across the archive.
+- Every `metadata.superseded_by` pointer must resolve to an ID present in the archive (dangling references fail validation).
+
+Any validation failure aborts execution and prints the detected issues.
+
+### Safety guards and live-collection protection
+
+- **Production project refusal:** Restoring to the production project `my-brain-88870` is refused (`assertWritableProject`).
+- **Live collection protection:** Restoring into the active live collection (`config.memoryCollection`, default `memory_vectors`) is refused by default. Operators must explicitly supply both `--replace-live` and `--confirm-collection <collection-name>` matching the collection to write into the live memory collection.
+- **Interruption behavior:** If an error occurs during re-embedding or writing, the script reports the number of documents restored so far, prints `RESTORE INCOMPLETE; re-run to finish (writes are idempotent by document id)`, and exits with code 1. Re-running resumes cleanly because document writes are keyed by ID.
+- **Embedding notice:** A model or dimension difference between the archive manifest and target configuration produces a warning noting that portable import re-embeds records with the current model into the target collection.
 
 ## Current limitations
 
-- Re-embedding requires model availability and does not promise bit-identical vectors.
-- A dimension/model mismatch and a manifest count mismatch currently warn rather than abort.
+- Re-embedding requires model availability and does not promise bit-identical vectors across model revisions.
 - Canonical serialization preserves known metadata fields, not arbitrary future fields.
-- Reading/writing the whole corpus is not a point-in-time or streaming recovery protocol.
-- The script requires a collection name but does **not** prohibit a live collection. Scratch-only restore is operator guidance, not enforced protection.
-- Environment loading includes `.env` and `.env.prod`; always explicitly review the target and model settings.
-
-RECOVERY tracks stronger validation, completeness/fidelity gates, compatibility checks, and safer targeting. Do not describe those controls as implemented. Verify restored IDs/content/lineage and retrieval independently; counts alone are insufficient.
+- Reading and writing the whole corpus is a batch operation, not a point-in-time snapshot or streaming recovery protocol.
+- Environment loading includes `.env` and `.env.prod`; always review target and model settings.

functions/package.json

@@ -23,6 +23,7 @@
     "inspect:prod": "node scripts/inspect-prod.mjs",
     "test": "vitest run --coverage",
     "test:watch": "vitest",
+    "test:emulator": "cd .. && firebase emulators:exec --only firestore --project demo-recovery \"npm --prefix functions test -- test/recovery.emulator.test.ts\"",
     "eval:files": "tsx scripts/provider-eval.ts",
     "backup:memories": "tsx scripts/backup-memories.ts",
     "backup:firestore-full": "tsx scripts/firestore-full-backup.ts",