diff --git a/docs/FULL_BACKUP.md b/docs/FULL_BACKUP.md index d9a10a6..2feca65 100644 --- a/docs/FULL_BACKUP.md +++ b/docs/FULL_BACKUP.md @@ -1,16 +1,25 @@ -# Full top-level Firestore backup +# Full recursive Firestore backup and recovery -Existing operator tooling integrated from `932f730` (cards 64–65); no fresh live recovery is claimed by this integration. +Operator tooling for recursive disaster-recovery backups, integrity validation, and safe content-fidelity restore across all Firestore collections. Distinct from [MEMORY_ARCHIVE.md](MEMORY_ARCHIVE.md), which archives only `memory_vectors` without vectors and re-embeds during restore. -As-is clone of every top-level Firestore collection discovered by -`listCollections()`. Distinct from [MEMORY_ARCHIVE.md](MEMORY_ARCHIVE.md), -which archives only `memory_vectors` and strips embeddings. +Prod `my-brain-88870` is read-only. Restore refuses that project id. -Inventory is live discovery. Do not hardcode collection names. +## Format and discovery -Vectors and timestamps round-trip via `__fs` tags. No re-embed. +Backups use manifest schema `metacortex-firestore-full-backup-v2`. -Prod `my-brain-88870` is read-only. Restore refuses that project id. +Discovery is recursive: it discovers top-level collections, subcollections, and subcollections under phantom parents (documents that have subcollections but no parent document payload). + +Each collection path is dumped to a deterministic NDJSON file named `encodeURIComponent(path) + ".ndjson"`. Each entry in `manifest.json` records: + +- Collection path (odd number of segments). +- Document count. +- NDJSON file name and size in bytes. +- Exact SHA256 hex checksum of the NDJSON file bytes. +- Vector dimensions observed across the collection documents. +- Archive-level embedding space metadata (model and dimensions), or null when no vectors are present. + +The manifest is written last so an interrupted backup never leaves an incomplete manifest pointing to partial snapshots. ## Backup @@ -20,22 +29,25 @@ npm --prefix functions run backup:firestore-full -- \ --archive-dir /path/to/archive ``` -`--dry-run` lists collections and counts without writing. +Pass `--dry-run` to discover paths and counts without writing any files. -Each run calls `listCollections()` on the source, then rewrites every -collection NDJSON file and `manifest.json`. This is a full snapshot, not an -incremental delta. NDJSON files for collections that disappeared from the -source are removed. Non-NDJSON files in the archive directory are left alone. +Backup discovers collections recursively, verifies that dumped document counts match live counts, encodes typed Firestore values (Timestamps, VectorValues, GeoPoints, Buffers, DocumentReferences, and non-finite numbers), computes checksums, writes NDJSON files, writes `manifest.json`, and cleans up stale NDJSON files from prior runs. -## Restore +## Archive validation before writes + +Before any target mutation or live restoration, `validateArchive` verifies the archive on disk: -Dry-run by default. `--write` is required to mutate the target. +1. Manifest readability, valid schema (`metacortex-firestore-full-backup-v2` or legacy `v1`), and consistent path ordering. +2. File existence, derived file names, and byte SHA256 checksum matches against manifest entries. +3. NDJSON JSON syntax, non-empty string `_id` presence, id uniqueness, and document count equality. +4. Structural decoding of all documents and tags via `decodeDocument`. Unknown `__fs` tags or malformed tag payloads are rejected immediately. +5. Observed vector dimensions equal manifest entries, and observed vector dimensions match the manifest embedding space. -`--write` converges each archived collection to the snapshot: +Validation failures abort immediately with a complete list of detected problems before any write operations occur. + +## Restore -- upsert every archive document by id (create or overwrite) -- prune target documents whose ids are absent from the archive -- create collections that exist in the archive but not yet on the target +Restore defaults to dry-run mode. `--write` is required to mutate the target database. ```bash npm --prefix functions run restore:firestore-full -- \ @@ -44,34 +56,39 @@ npm --prefix functions run restore:firestore-full -- \ --write ``` -`--clear-target-collections` is optional. It wipes each archived collection -before the upsert. Converge does not need it; prune already removes extras. +### Safety and target guards -Restore does not delete target collections that are absent from the archive. -The inventory gate checks source-to-target parity for source collections only. +- **Separate-target default:** Restoring directly into the source project is refused unless `--target-prefix ` is provided or explicit replacement confirmation is passed. +- **Upsert-only default:** Restore defaults to safe upserts (create or overwrite documents by id). Existing extra documents on the target are left untouched. +- **Explicit replacement confirmation:** `--prune` (deleting target documents whose ids are absent from the archive) and `--clear-target-collections` (wiping collections before restore) require `--confirm-project `. +- **Target prefixing:** `--target-prefix ` prepends `` to the root collection segment of each path (e.g. `memory_vectors/doc1/notes` becomes `test_memory_vectors/doc1/notes`), allowing safe testing inside an existing project. +- **Vector compatibility guard:** If the archive contains vectors, restore checks that target embedding model and dimensions match the archive. On mismatch or unknown embedding space, restore aborts before any write and recommends portable import (`npm run restore:memories`), which re-embeds records with the current model into a separate collection. Passing `--accept-unknown-embedding-space` overrides this check for legacy v1 archives. -## Re-run semantics +### Interruption and recovery -What happens if more docs (or collections) are added on prod and backup runs -again: +If a write failure occurs during restore, the command halts and prints `RESTORE INCOMPLETE: . re-run the same command to finish; upserts are idempotent`. -- New collections on prod: the next backup includes them via a fresh - `listCollections()`. Restore creates them on QA when it upserts their docs. -- New or updated docs: the backup rewrites the collection NDJSON. Restore - upserts those ids on QA. -- Deleted docs on prod: they are absent from the new snapshot. Restore prunes - those ids from the matching QA collection. -- Deleted collections on prod: their NDJSON is dropped from the archive. - Restore does not delete the leftover QA collection. The gate still passes - because it only requires source collections to exist on the target with - matching counts. -- Always re-run the inventory gate after backup plus restore. Success only - when the gate exits 0. +Because document writes are keyed by id, re-running the same restore command resumes and completes recovery safely. + +### Content-fidelity verification + +After writing, restore runs verification across every archived collection: + +- Reads back the stored documents from the target. +- Verifies document equality and field content against the archive using stable JSON stringification. +- For v2 archives, verifies that the target content SHA256 matches the archive SHA256 checksum. +- When prune is not requested, extra target documents are reported as `extra_target_docs` rather than a failure. When prune is requested, extra target documents cause a verification failure. +- Reports differing document IDs (missing, extra, changed, capped at 20 IDs each). + +A restore run is considered complete only when verification passes for all collections. + +## Legacy v1 archive compatibility + +The restore engine supports legacy v1 manifests (`metacortex-firestore-full-backup-v1`). Legacy manifests are normalized in memory, collection IDs are treated as paths, checksums and vector dimensions are computed during validation, and per-document equality verification is performed using canonical stable stringification. ## Inventory gate -Exit 0 only when every collection from source `listCollections()` exists on -the target with equal `count()`. +The inventory gate verifies recursive collection discovery, matching document counts, and matching content SHA256 hashes across source and target projects: ```bash npm --prefix functions run gate:firestore-inventory -- \ @@ -80,13 +97,13 @@ npm --prefix functions run gate:firestore-inventory -- \ --out /path/to/inventory-gate.json ``` -A backup is not complete until this gate exits 0. Prior #60/#62 runs hardcoded -four collection names and missed live collections. - -## Current limits and future recovery work - -The implementation does not recursively discover subcollections, establish one consistent read timestamp, or compare content hashes. The inventory gate checks source collection presence and counts; it reports target-only collections without failing on them. Equal counts are not proof of content equality. The script name "full" denotes its current top-level inventory scope, not a complete recursive disaster-recovery guarantee. +The gate succeeds only when all source collections exist on the target with equal document counts and identical content SHA256 checksums. -Restore is dry-run unless `--write` is supplied. With `--write`, it prunes extra documents in archived target collections before upserting. That can leave a partial target if later writes fail. The hardcoded refusal of the maintainer's production project is not a general source/target protection for other installations. Always inspect the explicit target and restore to isolated QA first. Use a dedicated archive directory separate from portable memory archives. +## Known limitations -Retain historical inventory evidence from cards 64–65, but strengthen snapshot consistency, archive integrity, source/target guards, and content/lineage verification under RECOVERY in the [roadmap](../metacortexplan.md). These are future work, not changes introduced by this baseline. Physical pruning remains an operator recovery action, not an agent or product deletion feature. +1. Firestore integer vs double distinction is not preserved by the Admin SDK read path. +2. DocumentReference values are restored with their original absolute paths and are not remapped by `--target-prefix`. +3. Field transforms, composite indexes, rules, TTL policies, and security rules are not part of the document archive. +4. Verification covers Firestore documents, not Cloud Storage objects or Firebase Authentication users. +5. Restore is not atomic across collections. In the event of an interruption, resume by re-running the restore command. +6. Live production restore was not exercised. diff --git a/docs/MEMORY_ARCHIVE.md b/docs/MEMORY_ARCHIVE.md index 5e2b252..80494fa 100644 --- a/docs/MEMORY_ARCHIVE.md +++ b/docs/MEMORY_ARCHIVE.md @@ -1,10 +1,10 @@ # Portable memory archive -Integrated from completed archive work on `worktree-memory-archive-task1` (`932f730`, cards 58–65). This is existing operator tooling, not the future owner dashboard. See [full backup](FULL_BACKUP.md) for top-level operational collections and [the roadmap](../metacortexplan.md) for recovery hardening. +Integrated from completed archive work on `worktree-memory-archive-task1` (`932f730`, cards 58-65) and hardened under RECOVERY (card 71). Distinct from [FULL_BACKUP.md](FULL_BACKUP.md), which captures all collections with original vectors and types. ## Format and behavior -Deterministic NDJSON preserves IDs, canonical/retrieval text, supported metadata/provenance, and media descriptors. It includes deprecated memories. It omits embeddings and re-embeds every record during restore. Raw image bytes are never in the memory store; separately stored artifacts are not copied. Audit/fingerprint/evaluation collections are outside this portable format. +Deterministic NDJSON preserves IDs, canonical/retrieval text, supported metadata/provenance, and media descriptors. It includes deprecated memories. It omits embeddings and re-embeds every record during restore with the current model. Raw image bytes are never in the memory store; separately stored artifacts are not copied. Audit/fingerprint/evaluation collections are outside this portable format. Use a dedicated **private** Git archive repository. The archive is plaintext. Never point it at the source repository or a folder with unrelated staged changes. Use a different directory from a full Firestore backup: the manifest formats are distinct. @@ -17,27 +17,40 @@ npm --prefix functions run backup:memories -- --project --archive-d npm --prefix functions run backup:memories -- --project --archive-dir --no-push ``` -Without `--dry-run`, this writes archive files and commits them. Without `--no-push`, it also pushes to the archive repository. No scheduling or upload is enabled by baseline integration. Operators can explicitly arrange a private schedule; cross-platform scheduling/setup is future work. +Without `--dry-run`, this writes archive files and commits them. Without `--no-push`, it also pushes to the archive repository. -The 10% document-count-drop guard rejects suspicious truncation; `--force` overrides it after operator investigation. `METACORTEX_ARCHIVE_DIR` can select the archive path. The deployment preflight warns about portable manifests more than three days old. +The 10% document-count-drop guard rejects suspicious truncation; `--force` overrides it after operator investigation. `METACORTEX_ARCHIVE_DIR` can select the archive path. -## Restore +## Restore and validation -The script requires `--collection` and defaults to dry-run. Use an explicit scratch project/collection: +The restore script requires `--collection` and defaults to dry-run mode: ```bash npm --prefix functions run restore:memories -- --project --archive-dir --collection memory_vectors_restore ``` -After inspecting the archive and target, add `--write` to re-embed/write. IDs and supported metadata are preserved. The script requires the current config environment, including API-key access for re-embedding; it does not automatically resolve Secret Manager. Provide secrets privately. Ensure the target collection has a matching vector index before querying it. +### Pre-restore semantic validation + +Before any Firestore connection is opened or any writes occur (in both dry-run and write modes), the entire archive is validated via `validatePortableArchive`: + +- `schema_version` must equal `ARCHIVE_SCHEMA_VERSION`. +- Document count in `memories.ndjson` must match `document_count` in `manifest.json`. +- Every memory must have a non-empty string `id`, non-empty string `content` and `retrieval_text`, and a valid `metadata` object with string `module_name`, string `branch_state`, and finite numeric timestamps `created_at` and `updated_at`. +- Document IDs must be unique across the archive. +- Every `metadata.superseded_by` pointer must resolve to an ID present in the archive (dangling references fail validation). + +Any validation failure aborts execution and prints the detected issues. + +### Safety guards and live-collection protection + +- **Production project refusal:** Restoring to the production project `my-brain-88870` is refused (`assertWritableProject`). +- **Live collection protection:** Restoring into the active live collection (`config.memoryCollection`, default `memory_vectors`) is refused by default. Operators must explicitly supply both `--replace-live` and `--confirm-collection ` matching the collection to write into the live memory collection. +- **Interruption behavior:** If an error occurs during re-embedding or writing, the script reports the number of documents restored so far, prints `RESTORE INCOMPLETE; re-run to finish (writes are idempotent by document id)`, and exits with code 1. Re-running resumes cleanly because document writes are keyed by ID. +- **Embedding notice:** A model or dimension difference between the archive manifest and target configuration produces a warning noting that portable import re-embeds records with the current model into the target collection. ## Current limitations -- Re-embedding requires model availability and does not promise bit-identical vectors. -- A dimension/model mismatch and a manifest count mismatch currently warn rather than abort. +- Re-embedding requires model availability and does not promise bit-identical vectors across model revisions. - Canonical serialization preserves known metadata fields, not arbitrary future fields. -- Reading/writing the whole corpus is not a point-in-time or streaming recovery protocol. -- The script requires a collection name but does **not** prohibit a live collection. Scratch-only restore is operator guidance, not enforced protection. -- Environment loading includes `.env` and `.env.prod`; always explicitly review the target and model settings. - -RECOVERY tracks stronger validation, completeness/fidelity gates, compatibility checks, and safer targeting. Do not describe those controls as implemented. Verify restored IDs/content/lineage and retrieval independently; counts alone are insufficient. +- Reading and writing the whole corpus is a batch operation, not a point-in-time snapshot or streaming recovery protocol. +- Environment loading includes `.env` and `.env.prod`; always review target and model settings. diff --git a/functions/package.json b/functions/package.json index 359cb82..b8ef0e0 100644 --- a/functions/package.json +++ b/functions/package.json @@ -23,6 +23,7 @@ "inspect:prod": "node scripts/inspect-prod.mjs", "test": "vitest run --coverage", "test:watch": "vitest", + "test:emulator": "cd .. && firebase emulators:exec --only firestore --project demo-recovery \"npm --prefix functions test -- test/recovery.emulator.test.ts\"", "eval:files": "tsx scripts/provider-eval.ts", "backup:memories": "tsx scripts/backup-memories.ts", "backup:firestore-full": "tsx scripts/firestore-full-backup.ts", diff --git a/functions/scripts/firestore-full-backup.ts b/functions/scripts/firestore-full-backup.ts index eec5b4e..9a9c20d 100644 --- a/functions/scripts/firestore-full-backup.ts +++ b/functions/scripts/firestore-full-backup.ts @@ -1,18 +1,13 @@ -import fs from "node:fs"; import path from "node:path"; import { fileURLToPath } from "node:url"; +import { loadConfig } from "../src/config.js"; import { loadEnvironment, readArg } from "./support/cli.js"; import { - countDocuments, + backupToArchive, firestoreProjectId, - dumpCollection, - listCollectionIds, - MANIFEST_SCHEMA, - openFirestore, - removeStaleNdjsonFiles, - writeManifest, - type FullBackupCollectionEntry + FirestoreMirrorStore, + openFirestore } from "./support/firestore-full-mirror.js"; const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)); @@ -35,76 +30,42 @@ if (!archiveDir) { process.exit(1); } +const config = loadConfig(process.env); const firestore = openFirestore(projectId, "backup-source"); +const store = new FirestoreMirrorStore(firestore); console.log(`project: ${firestoreProjectId(firestore)}`); console.log(`archive: ${archiveDir}`); console.log(`mode: ${dryRun ? "dry-run" : "write"}`); -console.log("discovery: listCollections()"); +console.log("discovery: recursive-listCollections"); console.log("snapshot: full rewrite (not incremental)"); -const collectionIds = await listCollectionIds(firestore); -console.log(`collections discovered: ${collectionIds.length}`); -console.log(JSON.stringify(collectionIds)); - -if (collectionIds.length === 0) { - console.error("listCollections() returned no collections"); - process.exit(1); -} - -if (!dryRun) { - fs.mkdirSync(archiveDir, { recursive: true }); -} - -const collections: FullBackupCollectionEntry[] = []; - -for (const collectionId of collectionIds) { - const liveCount = await countDocuments(firestore, collectionId); - console.log(`count ${projectId}/${collectionId}=${liveCount}`); - - if (dryRun) { - collections.push({ - id: collectionId, - document_count: liveCount, - ndjson: `${collectionId}.ndjson`, - bytes: 0 - }); - continue; - } - - const entry = await dumpCollection(firestore, collectionId, archiveDir); - if (entry.document_count !== liveCount) { - console.error( - `dump/count mismatch for ${collectionId}: dumped ${entry.document_count} counted ${liveCount}` +try { + const manifest = await backupToArchive(store, archiveDir, { + embedding: { + model: config.embeddingModel, + dimensions: config.embeddingDimensions + }, + dryRun + }); + + console.log(`collections discovered: ${manifest.collection_paths.length}`); + console.log(JSON.stringify(manifest.collection_paths)); + + for (const col of manifest.collections) { + console.log( + `dumped ${col.path} documents=${col.document_count} bytes=${col.bytes} file=${col.ndjson}` ); - process.exit(1); } - console.log( - `dumped ${collectionId} documents=${entry.document_count} bytes=${entry.bytes} file=${entry.ndjson}` - ); - collections.push(entry); -} -if (dryRun) { - console.log("Dry run complete. Nothing written."); - process.exit(0); -} + if (dryRun) { + console.log("Dry run complete. Nothing written."); + process.exit(0); + } -const stale = removeStaleNdjsonFiles(archiveDir, collectionIds); -if (stale.length > 0) { - console.log(`removed stale ndjson: ${JSON.stringify(stale)}`); + console.log(`wrote ${path.join(archiveDir, "manifest.json")}`); + console.log(`collections_backed_up=${manifest.collections.length}`); +} catch (error) { + console.error((error as Error).message); + process.exit(1); } - -const generatedAt = new Date(); -writeManifest(archiveDir, { - schema: MANIFEST_SCHEMA, - source_project: projectId, - generated_at: generatedAt.toISOString(), - generated_at_ms: generatedAt.getTime(), - discovery: "listCollections()", - collection_ids: collectionIds, - collections -}); - -console.log(`wrote ${path.join(archiveDir, "manifest.json")}`); -console.log(`collections_backed_up=${collections.length}`); diff --git a/functions/scripts/firestore-full-restore.ts b/functions/scripts/firestore-full-restore.ts index ca3f5ff..e06330b 100644 --- a/functions/scripts/firestore-full-restore.ts +++ b/functions/scripts/firestore-full-restore.ts @@ -1,18 +1,17 @@ -import path from "node:path"; import { fileURLToPath } from "node:url"; +import path from "node:path"; +import { loadConfig } from "../src/config.js"; import { loadEnvironment, readArg } from "./support/cli.js"; import { - assertWritableProject, - clearCollection, - countDocuments, - firestoreProjectId, - encodedDocumentIds, - loadArchiveCollection, + ArchiveValidationError, + FirestoreMirrorStore, + IncompatibleVectorsError, openFirestore, - pruneAbsentDocuments, - readManifest, - writeDocuments + restoreArchive, + RestoreIncompleteError, + RestoreTargetError, + RestoreVerificationError } from "./support/firestore-full-mirror.js"; const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)); @@ -22,15 +21,20 @@ loadEnvironment(functionsDirectory); const args = process.argv.slice(2); const write = args.includes("--write"); +const prune = args.includes("--prune"); const clearTarget = args.includes("--clear-target-collections"); +const acceptUnknownEmbeddingSpace = args.includes("--accept-unknown-embedding-space"); + const projectId = readArg(args, "project"); const archiveDir = readArg(args, "archive-dir") ?? process.env.METACORTEX_ARCHIVE_DIR; +const targetPrefix = readArg(args, "target-prefix"); +const confirmProject = readArg(args, "confirm-project"); if (!projectId) { console.error( - "usage: firestore-full-restore --project --archive-dir [--write] [--clear-target-collections]" + "usage: firestore-full-restore --project --archive-dir [--write] [--target-prefix

] [--prune] [--clear-target-collections] [--confirm-project ] [--accept-unknown-embedding-space]" ); - console.error("--write converges by default (upsert + prune). --clear-target-collections is an optional wipe first."); + console.error("Dry-run by default. Pass --write to execute upsert. --prune or --clear-target-collections requires --confirm-project ."); process.exit(1); } @@ -39,94 +43,55 @@ if (!archiveDir) { process.exit(1); } -assertWritableProject(projectId); - -const manifest = readManifest(archiveDir); -const collectionIds = manifest.collection_ids; -if ( - collectionIds.length !== manifest.collections.length || - collectionIds.some((id, index) => id !== manifest.collections[index].id) -) { - console.error("manifest collection_ids does not match collections[].id order"); - process.exit(1); -} +const config = loadConfig(process.env); +const firestore = openFirestore(projectId, "restore-target"); +const store = new FirestoreMirrorStore(firestore); console.log(`target: ${projectId}`); console.log(`archive: ${archiveDir}`); -console.log(`source_project: ${manifest.source_project}`); console.log(`mode: ${write ? "write" : "dry-run"}`); -console.log("converge: upsert+prune"); -console.log( - `clear_target_collections: ${clearTarget}${clearTarget ? " (wipe then upsert; prune skipped)" : ""}` -); -console.log("inventory: archive manifest (produced by listCollections() on source)"); -console.log("absent_archive_collections: left on target (not deleted)"); -console.log(`collections: ${JSON.stringify(collectionIds)}`); - -const firestore = write ? openFirestore(projectId, "restore-target") : undefined; -if (firestore) { - assertWritableProject(firestoreProjectId(firestore)); +if (targetPrefix) { + console.log(`target_prefix: ${targetPrefix}`); } +console.log(`prune: ${prune}`); +console.log(`clear_target: ${clearTarget}`); + +try { + const result = await restoreArchive(store, archiveDir, { + write, + targetPrefix, + replace: { + prune, + clearTarget, + confirmProject + }, + targetEmbedding: { + model: config.embeddingModel, + dimensions: config.embeddingDimensions + }, + acceptUnknownEmbeddingSpace, + onProgress: (msg: string) => console.log(msg) + }); -const report: Array<{ - id: string; - archive_count: number; - target_count_before: number | null; - pruned: number; - deleted: number; - written: number; - target_count_after: number | null; -}> = []; - -for (const entry of manifest.collections) { - const docs = loadArchiveCollection(archiveDir, entry); - const keepIds = encodedDocumentIds(docs); - const before = firestore ? await countDocuments(firestore, entry.id) : null; - console.log( - `plan ${entry.id} archive=${entry.document_count} target_before=${before ?? "n/a"}` - ); - - let pruned = 0; - let deleted = 0; - let written = 0; - let after: number | null = null; + console.log(JSON.stringify(result, null, 2)); - if (write && firestore) { - if (clearTarget) { - deleted = await clearCollection(firestore, entry.id); - console.log(`cleared ${projectId}/${entry.id} deleted=${deleted}`); - } else { - pruned = await pruneAbsentDocuments(firestore, entry.id, keepIds); - deleted = pruned; - console.log(`pruned ${projectId}/${entry.id} extra_ids=${pruned}`); - } - written = await writeDocuments(firestore, entry.id, docs); - after = await countDocuments(firestore, entry.id); - console.log(`upserted ${projectId}/${entry.id} written=${written} count=${after}`); - if (after !== entry.document_count) { - console.error( - `restore count mismatch for ${entry.id}: target ${after} != archive ${entry.document_count}` - ); - process.exit(1); - } + if (!write) { + console.log("Dry run complete. Re-run with --write to restore."); + process.exit(0); } - report.push({ - id: entry.id, - archive_count: entry.document_count, - target_count_before: before, - pruned, - deleted, - written, - target_count_after: after - }); -} - -console.log(JSON.stringify({ converge: "upsert+prune", collections: report }, null, 2)); - -if (!write) { - console.log("Dry run complete. Re-run with --write to restore."); - process.exit(0); + console.log(`restore_complete=true collections=${result.collections.length}`); +} catch (error) { + if ( + error instanceof RestoreIncompleteError || + error instanceof RestoreVerificationError || + error instanceof RestoreTargetError || + error instanceof IncompatibleVectorsError || + error instanceof ArchiveValidationError + ) { + console.error(error.message); + } else { + console.error(`Restore error: ${(error as Error).message}`); + } + process.exit(1); } - -console.log(`collections_restored=${report.length}`); diff --git a/functions/scripts/firestore-inventory-gate.ts b/functions/scripts/firestore-inventory-gate.ts index eddecf8..b84e629 100644 --- a/functions/scripts/firestore-inventory-gate.ts +++ b/functions/scripts/firestore-inventory-gate.ts @@ -3,10 +3,13 @@ import path from "node:path"; import { fileURLToPath } from "node:url"; import { loadEnvironment, readArg } from "./support/cli.js"; +import { encodeDocument, DOCUMENT_ID_FIELD } from "./support/firestore-codec.js"; import { - countDocuments, - listCollectionIds, - openFirestore + computeSha256, + FirestoreMirrorStore, + openFirestore, + serializeNdjson, + type MirrorDocument } from "./support/firestore-full-mirror.js"; const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)); @@ -29,10 +32,13 @@ if (!sourceProject || !targetProject) { const sourceDb = openFirestore(sourceProject, "gate-source"); const targetDb = openFirestore(targetProject, "gate-target"); +const sourceStore = new FirestoreMirrorStore(sourceDb); +const targetStore = new FirestoreMirrorStore(targetDb); + const measuredAt = new Date().toISOString(); -const sourceIds = await listCollectionIds(sourceDb); -const targetIds = await listCollectionIds(targetDb); -const targetIdSet = new Set(targetIds); +const sourcePaths = await sourceStore.discoverCollectionPaths(); +const targetPaths = await targetStore.discoverCollectionPaths(); +const targetPathSet = new Set(targetPaths); const collections: Array<{ id: string; @@ -40,41 +46,68 @@ const collections: Array<{ target_count: number | null; present_on_target: boolean; ok: boolean; + source_sha256?: string; + target_sha256?: string; }> = []; -for (const id of sourceIds) { - const sourceCount = await countDocuments(sourceDb, id); - const presentOnTarget = targetIdSet.has(id); - const targetCount = presentOnTarget ? await countDocuments(targetDb, id) : null; +for (const colPath of sourcePaths) { + const sourceDocs = await sourceStore.readCollection(colPath); + const sourceCount = sourceDocs.length; + const sourceEncoded = sourceDocs + .map((d: MirrorDocument) => encodeDocument(d.id, d.data)) + .sort((a: Record, b: Record) => + String(a[DOCUMENT_ID_FIELD]).localeCompare(String(b[DOCUMENT_ID_FIELD])) + ); + const sourceSha = computeSha256(serializeNdjson(sourceEncoded)); + + const presentOnTarget = targetPathSet.has(colPath); + let targetCount: number | null = null; + let targetSha: string | undefined; + + if (presentOnTarget) { + const targetDocs = await targetStore.readCollection(colPath); + targetCount = targetDocs.length; + const targetEncoded = targetDocs + .map((d: MirrorDocument) => encodeDocument(d.id, d.data)) + .sort((a: Record, b: Record) => + String(a[DOCUMENT_ID_FIELD]).localeCompare(String(b[DOCUMENT_ID_FIELD])) + ); + targetSha = computeSha256(serializeNdjson(targetEncoded)); + } + + const ok = presentOnTarget && targetCount === sourceCount && targetSha === sourceSha; + collections.push({ - id, + id: colPath, source_count: sourceCount, target_count: targetCount, present_on_target: presentOnTarget, - ok: presentOnTarget && targetCount === sourceCount + ok, + source_sha256: sourceSha, + target_sha256: targetSha }); } -const sourceIdSet = new Set(sourceIds); -const targetOnly = []; -for (const id of targetIds) { - if (sourceIdSet.has(id)) { +const sourcePathSet = new Set(sourcePaths); +const targetOnly: Array<{ id: string; target_count: number }> = []; +for (const colPath of targetPaths) { + if (sourcePathSet.has(colPath)) { continue; } targetOnly.push({ - id, - target_count: await countDocuments(targetDb, id) + id: colPath, + target_count: await targetStore.countDocuments(colPath) }); } const failures = collections.filter(entry => !entry.ok); const report = { - ok: failures.length === 0 && sourceIds.length > 0, + ok: failures.length === 0 && sourcePaths.length > 0, measured_at: measuredAt, - discovery: "listCollections()", + discovery: "recursive-listCollections", source_project: sourceProject, target_project: targetProject, - source_collection_count: sourceIds.length, + source_collection_count: sourcePaths.length, collections, target_only: targetOnly, failures: failures.map(entry => ({ diff --git a/functions/scripts/restore-memories.ts b/functions/scripts/restore-memories.ts index 953bfd3..3929d1e 100644 --- a/functions/scripts/restore-memories.ts +++ b/functions/scripts/restore-memories.ts @@ -9,6 +9,7 @@ import { describeEmbeddingMismatch, parseArchive, restoreMemories, + validatePortableArchive, type ArchiveManifest, type ArchivedMemory, type RestoreTarget @@ -17,6 +18,7 @@ import { FirestoreRestoreTarget } from "../src/archiveFirestore.js"; import { loadConfig } from "../src/config.js"; import { GeminiEmbeddingClient } from "../src/embeddings.js"; import { loadEnvironment, readArg, readFirebaseProject } from "./support/cli.js"; +import { assertWritableProject } from "./support/firestore-full-mirror.js"; const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)); const functionsDirectory = path.resolve(scriptDirectory, ".."); @@ -26,6 +28,8 @@ loadEnvironment(functionsDirectory); const args = process.argv.slice(2); const write = args.includes("--write"); +const replaceLive = args.includes("--replace-live"); +const confirmCollection = readArg(args, "confirm-collection"); const archiveDir = readArg(args, "archive-dir") ?? process.env.METACORTEX_ARCHIVE_DIR; const projectId = @@ -48,13 +52,53 @@ if (!targetCollection) { process.exit(1); } +const manifestPath = path.join(archiveDir, "manifest.json"); +const memoriesPath = path.join(archiveDir, "memories.ndjson"); + +if (!fs.existsSync(manifestPath) || !fs.existsSync(memoriesPath)) { + console.error(`missing manifest.json or memories.ndjson in ${archiveDir}`); + process.exit(1); +} + +let manifest: ArchiveManifest; +let memories: ArchivedMemory[]; + +try { + manifest = JSON.parse(fs.readFileSync(manifestPath, "utf8")) as ArchiveManifest; + memories = parseArchive(fs.readFileSync(memoriesPath, "utf8")); +} catch (error) { + console.error(`archive is malformed: ${(error as Error).message}`); + process.exit(1); +} + +// Validate the whole archive BEFORE any Firestore access, in dry-run and write mode +const problems = validatePortableArchive(memories, manifest); +if (problems.length > 0) { + console.error("archive failed validation:\n- " + problems.join("\n- ")); + process.exit(1); +} + const config = loadConfig(process.env); -const memories = parseArchive( - fs.readFileSync(path.join(archiveDir, "memories.ndjson"), "utf8") -); -const manifest = JSON.parse( - fs.readFileSync(path.join(archiveDir, "manifest.json"), "utf8") -) as ArchiveManifest; +const liveCollectionName = config.memoryCollection || "memory_vectors"; + +// Target guards: refuse prod project and refuse live collection unless confirmed +if (write) { + try { + assertWritableProject(projectId); + } catch (error) { + console.error((error as Error).message); + process.exit(1); + } + + if (targetCollection === liveCollectionName) { + if (!replaceLive || confirmCollection !== targetCollection) { + console.error( + `refusing write to live collection ${liveCollectionName}; pass --replace-live and --confirm-collection ${liveCollectionName}` + ); + process.exit(1); + } + } +} console.log(`archive: ${archiveDir}`); console.log(`archived at: ${new Date(manifest.generated_at).toISOString()}`); @@ -62,19 +106,15 @@ console.log(`documents: ${memories.length}`); console.log(`target: ${projectId}/${targetCollection}`); console.log(`mode: ${write ? "write" : "dry-run"}`); -if (memories.length !== manifest.document_count) { - console.warn( - `warning: archive holds ${memories.length} documents but the manifest records ${manifest.document_count}` - ); -} - const mismatch = describeEmbeddingMismatch(manifest, { embeddingModel: config.embeddingModel, embeddingDimensions: config.embeddingDimensions }); if (mismatch) { - console.warn(`warning: ${mismatch}`); + console.warn( + `warning: ${mismatch}; portable import re-embeds with the current model` + ); } if (!write) { @@ -95,19 +135,28 @@ const target: RestoreTarget = new FirestoreRestoreTarget( ); let completed = 0; -const result = await restoreMemories({ - memories, - embeddings, - target: { - async writeMemory(memory: ArchivedMemory, embedding: number[]) { - await target.writeMemory(memory, embedding); - completed += 1; - - if (completed % 25 === 0) { - console.log(`restored ${completed}/${memories.length}`); + +try { + const result = await restoreMemories({ + memories, + embeddings, + target: { + async writeMemory(memory: ArchivedMemory, embedding: number[]) { + await target.writeMemory(memory, embedding); + completed += 1; + + if (completed % 25 === 0) { + console.log(`restored ${completed}/${memories.length}`); + } } - } - } -}); + }, + manifest + }); -console.log(`Restored ${result.restored} documents into ${targetCollection}.`); + console.log(`Restored ${result.restored} documents into ${targetCollection}.`); +} catch (error) { + console.error( + `Restored ${completed} so far. RESTORE INCOMPLETE; re-run to finish (writes are idempotent by document id): ${(error as Error).message}` + ); + process.exit(1); +} diff --git a/functions/scripts/support/firestore-codec.ts b/functions/scripts/support/firestore-codec.ts index 4cf2372..e32d19f 100644 --- a/functions/scripts/support/firestore-codec.ts +++ b/functions/scripts/support/firestore-codec.ts @@ -1,4 +1,4 @@ -import { FieldValue, Timestamp } from "firebase-admin/firestore"; +import { FieldValue, GeoPoint, Timestamp } from "firebase-admin/firestore"; interface EncodedTimestamp { __fs: "timestamp"; @@ -11,6 +11,27 @@ interface EncodedVector { values: number[]; } +interface EncodedGeoPoint { + __fs: "geopoint"; + latitude: number; + longitude: number; +} + +interface EncodedBytes { + __fs: "bytes"; + base64: string; +} + +interface EncodedReference { + __fs: "reference"; + path: string; +} + +interface EncodedNumber { + __fs: "number"; + value: "NaN" | "Infinity" | "-Infinity"; +} + interface TimestampLike { seconds: number; nanoseconds: number; @@ -21,6 +42,16 @@ interface VectorLike { toArray: () => number[]; } +export interface GeoPointLike { + latitude: number; + longitude: number; +} + +export interface DocumentReferenceLike { + path: string; + firestore: object; +} + export const DOCUMENT_ID_FIELD = "_id"; export function isTimestamp(value: unknown): value is TimestampLike { @@ -44,11 +75,64 @@ export function isVectorValue(value: unknown): value is VectorLike { ); } -export function encodeValue(value: unknown): unknown { +export function isGeoPoint(value: unknown): value is GeoPointLike { + return ( + value instanceof GeoPoint || + (typeof value === "object" && + value !== null && + value.constructor?.name === "GeoPoint" && + typeof (value as GeoPointLike).latitude === "number" && + typeof (value as GeoPointLike).longitude === "number") + ); +} + +export function isBytes(value: unknown): value is Buffer | Uint8Array { + return Buffer.isBuffer(value) || value instanceof Uint8Array; +} + +export function isDocumentReference(value: unknown): value is DocumentReferenceLike { + return ( + typeof value === "object" && + value !== null && + typeof (value as DocumentReferenceLike).path === "string" && + typeof (value as DocumentReferenceLike).firestore === "object" && + value.constructor?.name === "DocumentReference" + ); +} + +export function isPlainObject(value: unknown): value is Record { + if (typeof value !== "object" || value === null) { + return false; + } + const proto = Object.getPrototypeOf(value); + return proto === Object.prototype || proto === null; +} + +export function encodeValue(value: unknown, currentPath = ""): unknown { if (value === null || typeof value === "undefined") { return value; } + if (typeof value === "string" || typeof value === "boolean") { + return value; + } + + if (typeof value === "number") { + if (Number.isNaN(value)) { + const encoded: EncodedNumber = { __fs: "number", value: "NaN" }; + return encoded; + } + if (value === Infinity) { + const encoded: EncodedNumber = { __fs: "number", value: "Infinity" }; + return encoded; + } + if (value === -Infinity) { + const encoded: EncodedNumber = { __fs: "number", value: "-Infinity" }; + return encoded; + } + return value; + } + if (isTimestamp(value)) { const encoded: EncodedTimestamp = { __fs: "timestamp", @@ -76,41 +160,138 @@ export function encodeValue(value: unknown): unknown { return encoded; } + if (isGeoPoint(value)) { + const encoded: EncodedGeoPoint = { + __fs: "geopoint", + latitude: value.latitude, + longitude: value.longitude + }; + return encoded; + } + + if (isBytes(value)) { + const encoded: EncodedBytes = { + __fs: "bytes", + base64: Buffer.from(value).toString("base64") + }; + return encoded; + } + + if (isDocumentReference(value)) { + const encoded: EncodedReference = { + __fs: "reference", + path: value.path + }; + return encoded; + } + if (Array.isArray(value)) { - return value.map(encodeValue); + return value.map((item, index) => + encodeValue(item, currentPath ? `${currentPath}.${index}` : `${index}`) + ); } - if (typeof value === "object") { + if (isPlainObject(value)) { + if (Object.prototype.hasOwnProperty.call(value, "__fs")) { + const fsPath = currentPath ? `${currentPath}.__fs` : "__fs"; + throw new Error(`reserved key "__fs" in field ${fsPath}`); + } const encoded: Record = {}; - for (const [key, nested] of Object.entries(value as Record)) { - encoded[key] = encodeValue(nested); + for (const [key, nested] of Object.entries(value)) { + const nextPath = currentPath ? `${currentPath}.${key}` : key; + encoded[key] = encodeValue(nested, nextPath); } return encoded; } - return value; + const typeName = (value as object).constructor?.name || typeof value; + throw new Error(`cannot encode value of type ${typeName} at ${currentPath || "root"}`); } -export function decodeValue(value: unknown): unknown { +export function decodeValue( + value: unknown, + resolveReference?: (path: string) => unknown +): unknown { if (value === null || typeof value === "undefined") { return value; } if (Array.isArray(value)) { - return value.map(decodeValue); + return value.map(item => decodeValue(item, resolveReference)); } if (typeof value === "object") { const obj = value as Record; - if (obj.__fs === "timestamp") { - return new Timestamp(Number(obj.seconds), Number(obj.nanoseconds)); - } - if (obj.__fs === "vector") { - return FieldValue.vector(obj.values as number[]); + if (typeof obj.__fs === "string") { + switch (obj.__fs) { + case "timestamp": { + if ( + typeof obj.seconds !== "number" || + typeof obj.nanoseconds !== "number" || + !Number.isFinite(obj.seconds) || + !Number.isFinite(obj.nanoseconds) + ) { + throw new Error("malformed timestamp tag"); + } + return new Timestamp(obj.seconds, obj.nanoseconds); + } + case "vector": { + if ( + !Array.isArray(obj.values) || + obj.values.some(v => typeof v !== "number" || !Number.isFinite(v)) + ) { + throw new Error("malformed vector tag"); + } + return FieldValue.vector(obj.values as number[]); + } + case "geopoint": { + if ( + typeof obj.latitude !== "number" || + typeof obj.longitude !== "number" || + !Number.isFinite(obj.latitude) || + !Number.isFinite(obj.longitude) + ) { + throw new Error("malformed geopoint tag"); + } + return new GeoPoint(obj.latitude, obj.longitude); + } + case "bytes": { + if (typeof obj.base64 !== "string") { + throw new Error("malformed bytes tag"); + } + return Buffer.from(obj.base64, "base64"); + } + case "reference": { + if (typeof obj.path !== "string" || obj.path.trim().length === 0) { + throw new Error("malformed reference tag"); + } + if (!resolveReference) { + throw new Error( + `cannot decode reference without resolver for path ${obj.path}` + ); + } + return resolveReference(obj.path); + } + case "number": { + if (obj.value === "NaN") { + return Number.NaN; + } + if (obj.value === "Infinity") { + return Number.POSITIVE_INFINITY; + } + if (obj.value === "-Infinity") { + return Number.NEGATIVE_INFINITY; + } + throw new Error(`malformed number tag: ${String(obj.value)}`); + } + default: + throw new Error(`unknown __fs tag: ${String(obj.__fs)}`); + } } + const decoded: Record = {}; for (const [key, nested] of Object.entries(obj)) { - decoded[key] = decodeValue(nested); + decoded[key] = decodeValue(nested, resolveReference); } return decoded; } @@ -128,7 +309,10 @@ export function encodeDocument( }; } -export function decodeDocument(doc: Record): { +export function decodeDocument( + doc: Record, + resolveReference?: (path: string) => unknown +): { id: string; data: Record; } { @@ -137,6 +321,28 @@ export function decodeDocument(doc: Record): { delete fields[DOCUMENT_ID_FIELD]; return { id, - data: decodeValue(fields) as Record + data: decodeValue(fields, resolveReference) as Record }; } + +export function stableStringify(value: unknown): string { + if (value === null || typeof value !== "object") { + return JSON.stringify(value); + } + + if (Array.isArray(value)) { + const items = value.map(item => (item === undefined ? "null" : stableStringify(item))); + return `[${items.join(",")}]`; + } + + const entries: string[] = []; + const keys = Object.keys(value as Record).sort((a, b) => a.localeCompare(b)); + for (const key of keys) { + const val = (value as Record)[key]; + if (typeof val === "undefined") { + continue; + } + entries.push(`${JSON.stringify(key)}:${stableStringify(val)}`); + } + return `{${entries.join(",")}}`; +} diff --git a/functions/scripts/support/firestore-full-mirror.ts b/functions/scripts/support/firestore-full-mirror.ts index f6fff2e..1067390 100644 --- a/functions/scripts/support/firestore-full-mirror.ts +++ b/functions/scripts/support/firestore-full-mirror.ts @@ -1,3 +1,4 @@ +import crypto from "node:crypto"; import fs from "node:fs"; import path from "node:path"; @@ -5,6 +6,7 @@ import { getApps, initializeApp, type App } from "firebase-admin/app"; import { FieldPath, getFirestore, + type CollectionReference, type Firestore, type QueryDocumentSnapshot } from "firebase-admin/firestore"; @@ -12,30 +14,141 @@ import { import { decodeDocument, DOCUMENT_ID_FIELD, - encodeDocument + encodeDocument, + stableStringify } from "./firestore-codec.js"; export const PROD_PROJECT_ID = "my-brain-88870"; export const MANIFEST_FILE = "manifest.json"; -export const MANIFEST_SCHEMA = "metacortex-firestore-full-backup-v1"; +export const MANIFEST_SCHEMA_V1 = "metacortex-firestore-full-backup-v1"; +export const MANIFEST_SCHEMA = "metacortex-firestore-full-backup-v2"; export const PAGE_SIZE = 400; export const WRITE_BATCH_LIMIT = 400; -export interface FullBackupCollectionEntry { +export interface MirrorDocument { id: string; + data: Record; +} + +export interface MirrorStore { + readonly projectId: string; + discoverCollectionPaths(): Promise; + readCollection(path: string): Promise; + countDocuments(path: string): Promise; + listDocumentIds(path: string): Promise; + writeDocuments(path: string, docs: MirrorDocument[]): Promise; + deleteDocuments(path: string, ids: string[]): Promise; + referenceFor(path: string): unknown; +} + +export interface FullBackupCollectionEntry { + path: string; document_count: number; ndjson: string; bytes: number; + sha256: string; + vector_dimensions: number[]; + id?: string; } export interface FullBackupManifest { - schema: typeof MANIFEST_SCHEMA; + schema: typeof MANIFEST_SCHEMA | typeof MANIFEST_SCHEMA_V1; source_project: string; generated_at: string; generated_at_ms: number; - discovery: "listCollections()"; - collection_ids: string[]; + discovery: "recursive-listCollections" | "listCollections()"; + embedding: { model: string; dimensions: number } | null; + collection_paths: string[]; collections: FullBackupCollectionEntry[]; + legacy?: boolean; + collection_ids?: string[]; +} + +export class ArchiveValidationError extends Error { + constructor(readonly problems: string[]) { + super( + `Archive validation failed with ${problems.length} problem(s):\n- ${problems.join( + "\n- " + )}` + ); + this.name = "ArchiveValidationError"; + } +} + +export interface ValidatedArchiveCollection { + entry: FullBackupCollectionEntry; + docs: Array>; +} + +export interface ValidatedArchive { + manifest: FullBackupManifest; + collections: ValidatedArchiveCollection[]; +} + +export class RestoreTargetError extends Error { + constructor(message: string) { + super(message); + this.name = "RestoreTargetError"; + } +} + +export class IncompatibleVectorsError extends Error { + constructor(message: string) { + super(message); + this.name = "IncompatibleVectorsError"; + } +} + +export class RestoreIncompleteError extends Error { + constructor( + message: string, + readonly report: unknown, + readonly completedCollections: string[] + ) { + super(message); + this.name = "RestoreIncompleteError"; + } +} + +export class RestoreVerificationError extends Error { + constructor( + message: string, + readonly report: unknown + ) { + super(message); + this.name = "RestoreVerificationError"; + } +} + +export interface RestoreOptions { + write: boolean; + targetPrefix?: string; + replace?: { + prune: boolean; + clearTarget: boolean; + confirmProject: string | undefined; + }; + targetEmbedding: { model: string; dimensions: number } | null; + acceptUnknownEmbeddingSpace?: boolean; + sourceProjectOverride?: string; + onProgress?: (message: string) => void; +} + +export interface RestoreCollectionReport { + path: string; + target_path: string; + archive_count: number; + target_count_before: number | null; + written: number; + deleted: number; + pruned: number; + extra_target_docs?: number; + status: "planned" | "written" | "verified"; +} + +export interface RestoreReport { + complete: boolean; + collections: RestoreCollectionReport[]; } export function assertWritableProject(projectId: string): void { @@ -46,8 +159,6 @@ export function assertWritableProject(projectId: string): void { } } -// The installed SDK exposes projectId at runtime but omits it from its public -// TypeScript declaration. Validate the value before using it for target guards. export function firestoreProjectId(firestore: Firestore): string { const projectId: unknown = Reflect.get(firestore, "projectId"); if (typeof projectId !== "string" || projectId.trim().length === 0) { @@ -56,6 +167,270 @@ export function firestoreProjectId(firestore: Firestore): string { return projectId; } +export function isValidCollectionPath(pathStr: string): boolean { + if (typeof pathStr !== "string" || pathStr.trim().length === 0) { + return false; + } + if (pathStr.startsWith("/") || pathStr.endsWith("/")) { + return false; + } + const segments = pathStr.split("/"); + if (segments.some(seg => seg.length === 0 || seg === "." || seg === "..")) { + return false; + } + return segments.length % 2 === 1; +} + +export function assertValidCollectionPath(pathStr: string): void { + if (!isValidCollectionPath(pathStr)) { + throw new Error(`invalid collection path: ${pathStr}`); + } +} + +export function computeSha256(content: string | Buffer): string { + return crypto.createHash("sha256").update(content).digest("hex"); +} + +export function extractVectorDimensions(doc: unknown): number[] { + const dimensions: number[] = []; + const inspect = (val: unknown): void => { + if (val === null || typeof val !== "object") { + return; + } + if (Array.isArray(val)) { + for (const item of val) { + inspect(item); + } + return; + } + const obj = val as Record; + if (obj.__fs === "vector" && Array.isArray(obj.values)) { + dimensions.push(obj.values.length); + } + for (const v of Object.values(obj)) { + inspect(v); + } + }; + inspect(doc); + return dimensions; +} + +export function ndjsonFileName(pathOrId: string): string { + if ( + !pathOrId || + pathOrId.trim().length === 0 || + pathOrId.includes("..") || + pathOrId.split("/").some(s => s === "..") + ) { + throw new Error(`refusing unsafe collection path for filename: ${pathOrId}`); + } + return `${encodeURIComponent(pathOrId)}.ndjson`; +} + +export function serializeNdjson(docs: Array>): string { + return docs.map(doc => `${stableStringify(doc)}\n`).join(""); +} + +export function parseNdjsonLines( + raw: string, + filePath: string, + problems: string[] +): Array> { + const docs: Array> = []; + const lines = raw.split(/\r?\n/); + for (let index = 0; index < lines.length; index += 1) { + const trimmed = lines[index].trim(); + if (!trimmed) { + continue; + } + try { + docs.push(JSON.parse(trimmed) as Record); + } catch (error) { + problems.push( + `malformed JSON line in ${filePath} at line ${index + 1}: ${(error as Error).message}` + ); + } + } + return docs; +} + +export function parseNdjson(raw: string): Array> { + const problems: string[] = []; + const docs = parseNdjsonLines(raw, "ndjson", problems); + if (problems.length > 0) { + throw new Error(problems[0]); + } + return docs; +} + +export function encodedDocumentIds(docs: Array>): Set { + const ids = new Set(); + for (const doc of docs) { + const id = doc[DOCUMENT_ID_FIELD]; + if (typeof id !== "string" || id.length === 0) { + throw new Error(`archive document missing ${DOCUMENT_ID_FIELD}`); + } + ids.add(id); + } + return ids; +} + +export function idsToPrune( + targetIds: readonly string[], + keepIds: ReadonlySet +): string[] { + return targetIds.filter(id => !keepIds.has(id)); +} + +export function removeStaleNdjsonFiles( + archiveDir: string, + collectionPaths: readonly string[] +): string[] { + if (!fs.existsSync(archiveDir)) { + return []; + } + + const keep = new Set(collectionPaths.map(ndjsonFileName)); + const removed: string[] = []; + for (const fileName of fs.readdirSync(archiveDir)) { + if (!fileName.endsWith(".ndjson") || keep.has(fileName)) { + continue; + } + fs.unlinkSync(path.join(archiveDir, fileName)); + removed.push(fileName); + } + return removed; +} + +export function resolveTargetPath(pathStr: string, targetPrefix?: string): string { + if (!targetPrefix) { + return pathStr; + } + const segments = pathStr.split("/"); + segments[0] = `${targetPrefix}${segments[0]}`; + return segments.join("/"); +} + +export class FirestoreMirrorStore implements MirrorStore { + constructor(readonly firestore: Firestore) {} + + get projectId(): string { + return firestoreProjectId(this.firestore); + } + + async discoverCollectionPaths(): Promise { + const rootCollections = await this.firestore.listCollections(); + const discovered = new Set(); + + const traverse = async (colRef: CollectionReference): Promise => { + discovered.add(colRef.path); + const docRefs = await colRef.listDocuments(); + for (const docRef of docRefs) { + const subCollections = await docRef.listCollections(); + for (const subCol of subCollections) { + await traverse(subCol); + } + } + }; + + for (const root of rootCollections) { + await traverse(root); + } + + return Array.from(discovered).sort((left, right) => left.localeCompare(right)); + } + + async readCollection(pathStr: string): Promise { + assertValidCollectionPath(pathStr); + const docs: MirrorDocument[] = []; + let lastId: string | undefined; + + while (true) { + let query = this.firestore + .collection(pathStr) + .orderBy(FieldPath.documentId()) + .limit(PAGE_SIZE); + + if (lastId) { + query = query.startAfter(lastId); + } + + const snap = await query.get(); + if (snap.empty) { + break; + } + + for (const doc of snap.docs as QueryDocumentSnapshot[]) { + docs.push({ + id: doc.id, + data: doc.data() as Record + }); + } + lastId = snap.docs[snap.docs.length - 1].id; + if (snap.size < PAGE_SIZE) { + break; + } + } + + docs.sort((left, right) => left.id.localeCompare(right.id)); + return docs; + } + + async countDocuments(pathStr: string): Promise { + assertValidCollectionPath(pathStr); + const snap = await this.firestore.collection(pathStr).count().get(); + return snap.data().count; + } + + async listDocumentIds(pathStr: string): Promise { + assertValidCollectionPath(pathStr); + const docs = await this.readCollection(pathStr); + return docs.map(doc => doc.id); + } + + async writeDocuments(pathStr: string, docs: MirrorDocument[]): Promise { + assertValidCollectionPath(pathStr); + assertWritableProject(this.projectId); + let written = 0; + const collection = this.firestore.collection(pathStr); + + for (let offset = 0; offset < docs.length; offset += WRITE_BATCH_LIMIT) { + const chunk = docs.slice(offset, offset + WRITE_BATCH_LIMIT); + const batch = this.firestore.batch(); + for (const doc of chunk) { + batch.set(collection.doc(doc.id), doc.data); + written += 1; + } + await batch.commit(); + } + + return written; + } + + async deleteDocuments(pathStr: string, ids: string[]): Promise { + assertValidCollectionPath(pathStr); + assertWritableProject(this.projectId); + let deleted = 0; + const collection = this.firestore.collection(pathStr); + + for (let offset = 0; offset < ids.length; offset += WRITE_BATCH_LIMIT) { + const chunk = ids.slice(offset, offset + WRITE_BATCH_LIMIT); + const batch = this.firestore.batch(); + for (const id of chunk) { + batch.delete(collection.doc(id)); + deleted += 1; + } + await batch.commit(); + } + + return deleted; + } + + referenceFor(pathStr: string): unknown { + return this.firestore.doc(pathStr); + } +} + export function openFirestore(projectId: string, role: string): Firestore { const name = `full-mirror-${role}-${projectId}`; const existing = getApps().find((app: App) => app.name === name); @@ -72,9 +447,6 @@ export function openFirestore(projectId: string, role: string): Firestore { return firestore; } -/** - * Inventory comes only from live listCollections(). Never pass a hardcoded set. - */ export async function listCollectionIds(firestore: Firestore): Promise { const refs = await firestore.listCollections(); return refs.map(ref => ref.id).sort((left, right) => left.localeCompare(right)); @@ -120,69 +492,6 @@ export async function readAllDocs( return docs; } -export function ndjsonFileName(collectionId: string): string { - if (collectionId.includes("/") || collectionId.includes("\\") || collectionId.includes("..")) { - throw new Error(`refusing unsafe collection id for filename: ${collectionId}`); - } - return `${collectionId}.ndjson`; -} - -export function serializeNdjson(docs: Array>): string { - return docs.map(doc => `${JSON.stringify(doc)}\n`).join(""); -} - -export function parseNdjson(raw: string): Array> { - const docs: Array> = []; - for (const line of raw.split(/\r?\n/)) { - const trimmed = line.trim(); - if (!trimmed) { - continue; - } - docs.push(JSON.parse(trimmed) as Record); - } - return docs; -} - -export function encodedDocumentIds(docs: Array>): Set { - const ids = new Set(); - for (const doc of docs) { - const id = doc[DOCUMENT_ID_FIELD]; - if (typeof id !== "string" || id.length === 0) { - throw new Error(`archive document missing ${DOCUMENT_ID_FIELD}`); - } - ids.add(id); - } - return ids; -} - -export function idsToPrune(targetIds: readonly string[], keepIds: ReadonlySet): string[] { - return targetIds.filter(id => !keepIds.has(id)); -} - -/** - * Drop NDJSON files for collections that listCollections() no longer returns. - * Leaves non-NDJSON files (AUDIT, gate JSON) untouched. - */ -export function removeStaleNdjsonFiles( - archiveDir: string, - collectionIds: readonly string[] -): string[] { - if (!fs.existsSync(archiveDir)) { - return []; - } - - const keep = new Set(collectionIds.map(ndjsonFileName)); - const removed: string[] = []; - for (const fileName of fs.readdirSync(archiveDir)) { - if (!fileName.endsWith(".ndjson") || keep.has(fileName)) { - continue; - } - fs.unlinkSync(path.join(archiveDir, fileName)); - removed.push(fileName); - } - return removed; -} - export async function dumpCollection( firestore: Firestore, collectionId: string, @@ -199,11 +508,21 @@ export async function dumpCollection( const body = serializeNdjson(docs); fs.writeFileSync(outPath, body, "utf8"); + const dimSet = new Set(); + for (const doc of docs) { + for (const d of extractVectorDimensions(doc)) { + dimSet.add(d); + } + } + return { + path: collectionId, id: collectionId, document_count: docs.length, ndjson: fileName, - bytes: Buffer.byteLength(body) + bytes: Buffer.byteLength(body, "utf8"), + sha256: computeSha256(body), + vector_dimensions: Array.from(dimSet).sort((a, b) => a - b) }; } @@ -243,7 +562,7 @@ export async function writeDocuments( const chunk = docs.slice(offset, offset + WRITE_BATCH_LIMIT); const batch = firestore.batch(); for (const encoded of chunk) { - const { id, data } = decodeDocument(encoded); + const { id, data } = decodeDocument(encoded, p => firestore.doc(p)); batch.set(collection.doc(id), data); written += 1; } @@ -253,10 +572,6 @@ export async function writeDocuments( return written; } -/** - * Delete target docs whose ids are not in the archive snapshot for this collection. - * Does not delete collections that are absent from the archive. - */ export async function pruneAbsentDocuments( firestore: Firestore, collectionId: string, @@ -287,17 +602,52 @@ export function readManifest(archiveDir: string): FullBackupManifest { throw new Error(`missing ${MANIFEST_FILE} in ${archiveDir}`); } - const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf8")) as FullBackupManifest; - if (manifest.schema !== MANIFEST_SCHEMA) { + const manifest = JSON.parse( + fs.readFileSync(manifestPath, "utf8") + ) as Record; + + if (manifest.schema !== MANIFEST_SCHEMA && manifest.schema !== MANIFEST_SCHEMA_V1) { throw new Error( `unsupported manifest schema ${String(manifest.schema)}; expected ${MANIFEST_SCHEMA}` ); } + if (!Array.isArray(manifest.collections) || manifest.collections.length === 0) { - throw new Error("manifest collections must be a non-empty array from listCollections()"); + throw new Error("manifest collections must be a non-empty array"); } - return manifest; + if (manifest.schema === MANIFEST_SCHEMA_V1) { + const rawCollections = manifest.collections as Array<{ + id: string; + document_count: number; + ndjson: string; + bytes: number; + }>; + const collections: FullBackupCollectionEntry[] = rawCollections.map(c => ({ + path: c.id, + id: c.id, + document_count: c.document_count, + ndjson: c.ndjson, + bytes: c.bytes, + sha256: "", + vector_dimensions: [] + })); + const collectionPaths = collections.map(c => c.path); + return { + schema: MANIFEST_SCHEMA_V1, + source_project: String(manifest.source_project), + generated_at: String(manifest.generated_at), + generated_at_ms: Number(manifest.generated_at_ms), + discovery: "listCollections()", + embedding: null, + collection_paths: collectionPaths, + collections, + legacy: true, + collection_ids: collectionPaths + }; + } + + return manifest as unknown as FullBackupManifest; } export function writeManifest(archiveDir: string, manifest: FullBackupManifest): void { @@ -314,7 +664,9 @@ export function loadArchiveCollection( ): Array> { const filePath = path.join(archiveDir, entry.ndjson); if (!fs.existsSync(filePath)) { - throw new Error(`missing archive file ${entry.ndjson} for collection ${entry.id}`); + throw new Error( + `missing archive file ${entry.ndjson} for collection ${entry.path ?? entry.id}` + ); } const docs = parseNdjson(fs.readFileSync(filePath, "utf8")); if (docs.length !== entry.document_count) { @@ -324,3 +676,521 @@ export function loadArchiveCollection( } return docs; } + +export function validateArchive(archiveDir: string): ValidatedArchive { + const problems: string[] = []; + + let manifest: FullBackupManifest; + try { + manifest = readManifest(archiveDir); + } catch (error) { + throw new ArchiveValidationError([(error as Error).message]); + } + + if ( + !Array.isArray(manifest.collection_paths) || + manifest.collection_paths.length !== manifest.collections.length || + manifest.collection_paths.some((p, i) => p !== manifest.collections[i].path) + ) { + problems.push("manifest collection_paths does not match collections[].path order"); + } + + const seenPaths = new Set(); + for (const entry of manifest.collections) { + if (seenPaths.has(entry.path)) { + problems.push(`duplicate collection path: ${entry.path}`); + } else { + seenPaths.add(entry.path); + } + + if (!isValidCollectionPath(entry.path)) { + problems.push( + `invalid collection path (even path segments or empty segments not allowed): ${entry.path}` + ); + } + } + + const validatedCollections: ValidatedArchiveCollection[] = []; + const placeholderResolver = (targetPath: string): unknown => ({ + __type: "DocumentReferencePlaceholder", + path: targetPath + }); + + for (const entry of manifest.collections) { + const expectedFileName = ndjsonFileName(entry.path); + if (entry.ndjson !== expectedFileName) { + problems.push( + `ndjson file name mismatch for ${entry.path}: expected ${expectedFileName}, got ${entry.ndjson}` + ); + } + + const filePath = path.join(archiveDir, entry.ndjson); + if (!fs.existsSync(filePath)) { + problems.push(`missing ndjson file for ${entry.path}: ${entry.ndjson}`); + continue; + } + + const fileBytes = fs.readFileSync(filePath); + const computedSha = computeSha256(fileBytes); + + if (manifest.legacy) { + entry.sha256 = computedSha; + } else if (computedSha !== entry.sha256) { + problems.push( + `sha256 mismatch for ${entry.path}: file has ${computedSha}, manifest records ${entry.sha256}` + ); + } + + const docs = parseNdjsonLines(fileBytes.toString("utf8"), entry.ndjson, problems); + + if (docs.length !== entry.document_count) { + problems.push( + `document count mismatch for ${entry.path}: line count ${docs.length} != document_count ${entry.document_count}` + ); + } + + const seenIds = new Set(); + const observedDimensionsSet = new Set(); + + for (const doc of docs) { + const id = doc[DOCUMENT_ID_FIELD]; + if (typeof id !== "string" || id.trim().length === 0) { + problems.push(`document missing non-empty string _id in ${entry.path}`); + } else if (seenIds.has(id)) { + problems.push(`duplicate _id in ${entry.path}: ${id}`); + } else { + seenIds.add(id); + } + + try { + decodeDocument(doc, placeholderResolver); + } catch (error) { + const msg = (error as Error).message; + if (msg.includes("unknown __fs tag")) { + problems.push(`unknown __fs tag in document ${String(id)} in ${entry.path}: ${msg}`); + } else { + problems.push(`malformed tag or value in document ${String(id)} in ${entry.path}: ${msg}`); + } + } + + for (const dim of extractVectorDimensions(doc)) { + observedDimensionsSet.add(dim); + } + } + + const observedDimensions = Array.from(observedDimensionsSet).sort((a, b) => a - b); + if (manifest.legacy) { + entry.vector_dimensions = observedDimensions; + } else { + const expectedDims = [...(entry.vector_dimensions ?? [])].sort((a, b) => a - b); + if ( + observedDimensions.length !== expectedDims.length || + observedDimensions.some((d, idx) => d !== expectedDims[idx]) + ) { + problems.push( + `vector dimension disagreeing with the manifest for ${entry.path}: observed [${observedDimensions.join( + "," + )}], manifest records [${expectedDims.join(",")}]` + ); + } + } + + if (manifest.embedding) { + for (const dim of observedDimensions) { + if (dim !== manifest.embedding.dimensions) { + problems.push( + `vector dimension ${dim} in ${entry.path} disagrees with manifest embedding dimension ${manifest.embedding.dimensions}` + ); + } + } + } + + validatedCollections.push({ entry, docs }); + } + + if (problems.length > 0) { + throw new ArchiveValidationError(problems); + } + + return { + manifest, + collections: validatedCollections + }; +} + +export async function verifyRestore( + target: MirrorStore, + archiveDir: string, + options?: { + targetPrefix?: string; + prune?: boolean; + onProgress?: (message: string) => void; + } +): Promise { + const validated = validateArchive(archiveDir); + const manifest = validated.manifest; + const verificationFailures: string[] = []; + const finalReports: RestoreCollectionReport[] = []; + + for (const { entry, docs } of validated.collections) { + options?.onProgress?.(`verifying ${entry.path}`); + const targetPath = resolveTargetPath(entry.path, options?.targetPrefix); + const targetDocs = await target.readCollection(targetPath); + + const targetEncoded = targetDocs.map(d => encodeDocument(d.id, d.data)); + targetEncoded.sort((a, b) => + String(a[DOCUMENT_ID_FIELD]).localeCompare(String(b[DOCUMENT_ID_FIELD])) + ); + + const targetDocMap = new Map>(); + for (const d of targetEncoded) { + targetDocMap.set(String(d[DOCUMENT_ID_FIELD]), d); + } + + const archiveDocMap = new Map>(); + for (const d of docs) { + archiveDocMap.set(String(d[DOCUMENT_ID_FIELD]), d); + } + + const missing: string[] = []; + const extra: string[] = []; + const changed: string[] = []; + + for (const [id, archiveDoc] of archiveDocMap.entries()) { + const targetDoc = targetDocMap.get(id); + if (!targetDoc) { + missing.push(id); + } else if (stableStringify(archiveDoc) !== stableStringify(targetDoc)) { + changed.push(id); + } + } + + for (const id of targetDocMap.keys()) { + if (!archiveDocMap.has(id)) { + extra.push(id); + } + } + + const pruneRequested = Boolean(options?.prune); + const extraDocsCount = pruneRequested ? undefined : extra.length; + + if (pruneRequested && extra.length > 0) { + const capped = extra.slice(0, 20); + verificationFailures.push( + `collection ${entry.path}: target has ${extra.length} extra document(s) after prune: ${capped.join(", ")}` + ); + } + + if (missing.length > 0) { + const capped = missing.slice(0, 20); + verificationFailures.push( + `collection ${entry.path}: target is missing ${missing.length} document(s): ${capped.join(", ")}` + ); + } + + if (changed.length > 0) { + const capped = changed.slice(0, 20); + verificationFailures.push( + `collection ${entry.path}: target has ${changed.length} changed document(s): ${capped.join(", ")}` + ); + } + + if (!manifest.legacy) { + if (pruneRequested) { + const targetSha = computeSha256(serializeNdjson(targetEncoded)); + if (targetSha !== entry.sha256) { + verificationFailures.push( + `collection ${entry.path}: content sha256 mismatch (target: ${targetSha}, archive: ${entry.sha256})` + ); + } + } else { + const subsetDocs = targetEncoded.filter(d => + archiveDocMap.has(String(d[DOCUMENT_ID_FIELD])) + ); + const subsetSha = computeSha256(serializeNdjson(subsetDocs)); + if (subsetSha !== entry.sha256) { + verificationFailures.push( + `collection ${entry.path}: content sha256 mismatch on archive subset (target: ${subsetSha}, archive: ${entry.sha256})` + ); + } + } + } + + finalReports.push({ + path: entry.path, + target_path: targetPath, + archive_count: entry.document_count, + target_count_before: null, + written: docs.length, + deleted: 0, + pruned: 0, + extra_target_docs: extraDocsCount, + status: "verified" + }); + } + + if (verificationFailures.length > 0) { + throw new RestoreVerificationError( + `Content verification failed:\n- ${verificationFailures.join("\n- ")}`, + finalReports + ); + } + + return { + complete: true, + collections: finalReports + }; +} + +export async function restoreArchive( + target: MirrorStore, + archiveDir: string, + options: RestoreOptions +): Promise { + const validated: ValidatedArchive = validateArchive(archiveDir); + const manifest = validated.manifest; + const sourceProject = options.sourceProjectOverride ?? manifest.source_project; + + if (options.write) { + assertWritableProject(target.projectId); + } + + const hasConfirmation = options.replace?.confirmProject === target.projectId; + const isSameProject = target.projectId === sourceProject; + if (isSameProject && !options.targetPrefix && !hasConfirmation) { + throw new RestoreTargetError( + "target project equals the archive source project; restore into a different project, pass --target-prefix, or use explicit replacement (--prune/--clear-target-collections with --confirm-project )" + ); + } + + const isReplacementRequested = Boolean( + options.replace?.prune || options.replace?.clearTarget + ); + if (isReplacementRequested && !hasConfirmation) { + throw new RestoreTargetError( + "replacement requires --confirm-project " + ); + } + + const hasVectors = validated.collections.some( + c => c.entry.vector_dimensions && c.entry.vector_dimensions.length > 0 + ); + + if (hasVectors) { + if (!manifest.embedding) { + if (!options.acceptUnknownEmbeddingSpace) { + throw new IncompatibleVectorsError( + "archive embedding space is unknown (legacy v1 manifest). Pass --accept-unknown-embedding-space or use the portable import (npm run restore:memories) which re-embeds into a separate collection" + ); + } + } else { + if (!options.targetEmbedding) { + throw new IncompatibleVectorsError( + `target embedding space is null while archive contains vectors (${manifest.embedding.model}/${manifest.embedding.dimensions}d); do not import these vectors. Use the portable import (npm run restore:memories) which re-embeds into a separate collection` + ); + } + + if ( + manifest.embedding.model !== options.targetEmbedding.model || + manifest.embedding.dimensions !== options.targetEmbedding.dimensions + ) { + throw new IncompatibleVectorsError( + `archive embedding space (${manifest.embedding.model}/${manifest.embedding.dimensions}d) differs from target embedding space (${options.targetEmbedding.model}/${options.targetEmbedding.dimensions}d); do not import these vectors. Use the portable import (npm run restore:memories) which re-embeds into a separate collection` + ); + } + } + } + + if (!options.write) { + const plannedReports: RestoreCollectionReport[] = []; + for (const { entry } of validated.collections) { + const targetPath = resolveTargetPath(entry.path, options.targetPrefix); + const before = await target.countDocuments(targetPath); + plannedReports.push({ + path: entry.path, + target_path: targetPath, + archive_count: entry.document_count, + target_count_before: before, + written: 0, + deleted: 0, + pruned: 0, + status: "planned" + }); + } + return { + complete: false, + collections: plannedReports + }; + } + + const reportMap = new Map(); + const completedCollections: string[] = []; + + try { + for (const { entry, docs } of validated.collections) { + options.onProgress?.(`writing ${entry.path}`); + const targetPath = resolveTargetPath(entry.path, options.targetPrefix); + const beforeCount = await target.countDocuments(targetPath); + + let deleted = 0; + if (options.replace?.clearTarget) { + const existingIds = await target.listDocumentIds(targetPath); + deleted = await target.deleteDocuments(targetPath, existingIds); + } + + const mirrorDocs: MirrorDocument[] = docs.map(rawDoc => { + const decoded = decodeDocument(rawDoc, p => target.referenceFor(p)); + return { id: decoded.id, data: decoded.data }; + }); + + const written = await target.writeDocuments(targetPath, mirrorDocs); + + let pruned = 0; + if (options.replace?.prune && !options.replace?.clearTarget) { + const archiveIdSet = new Set(docs.map(d => String(d[DOCUMENT_ID_FIELD]))); + const existingIds = await target.listDocumentIds(targetPath); + const toPrune = existingIds.filter(id => !archiveIdSet.has(id)); + if (toPrune.length > 0) { + pruned = await target.deleteDocuments(targetPath, toPrune); + } + } + + completedCollections.push(entry.path); + reportMap.set(entry.path, { + path: entry.path, + target_path: targetPath, + archive_count: entry.document_count, + target_count_before: beforeCount, + written, + deleted: deleted + pruned, + pruned, + status: "written" + }); + } + } catch (error) { + const err = error as Error; + const partialReports = Array.from(reportMap.values()); + throw new RestoreIncompleteError( + `RESTORE INCOMPLETE: ${err.message}. re-run the same command to finish; upserts are idempotent`, + partialReports, + completedCollections + ); + } + + const verifiedReport = await verifyRestore(target, archiveDir, { + targetPrefix: options.targetPrefix, + prune: isReplacementRequested, + onProgress: options.onProgress + }); + + const mergedCollections: RestoreCollectionReport[] = verifiedReport.collections.map(vc => { + const writeReport = reportMap.get(vc.path); + return { + ...vc, + target_count_before: writeReport?.target_count_before ?? null, + written: writeReport?.written ?? vc.written, + deleted: writeReport?.deleted ?? vc.deleted, + pruned: writeReport?.pruned ?? vc.pruned + }; + }); + + return { + complete: true, + collections: mergedCollections + }; +} + +export async function backupToArchive( + store: MirrorStore, + archiveDir: string, + opts: { + embedding: { model: string; dimensions: number } | null; + now?: () => Date; + dryRun?: boolean; + } +): Promise { + const collectionPaths = await store.discoverCollectionPaths(); + if (collectionPaths.length === 0) { + throw new Error("refusing backup: discovered zero collections"); + } + + const collections: FullBackupCollectionEntry[] = []; + const encodedCollections: Array<{ entry: FullBackupCollectionEntry; body: string }> = []; + + for (const collectionPath of collectionPaths) { + const docs = await store.readCollection(collectionPath); + const liveCount = await store.countDocuments(collectionPath); + if (docs.length !== liveCount) { + throw new Error( + `dump/count mismatch for ${collectionPath}: dumped ${docs.length} counted ${liveCount}` + ); + } + + const encodedDocs = docs + .map(doc => encodeDocument(doc.id, doc.data)) + .sort((left, right) => + String(left[DOCUMENT_ID_FIELD]).localeCompare(String(right[DOCUMENT_ID_FIELD])) + ); + + const dimSet = new Set(); + for (const d of encodedDocs) { + for (const dim of extractVectorDimensions(d)) { + dimSet.add(dim); + } + } + const vectorDimensions = Array.from(dimSet).sort((a, b) => a - b); + const fileName = ndjsonFileName(collectionPath); + const body = serializeNdjson(encodedDocs); + const sha256 = computeSha256(body); + const bytes = Buffer.byteLength(body, "utf8"); + + const entry: FullBackupCollectionEntry = { + path: collectionPath, + id: collectionPath, + document_count: encodedDocs.length, + ndjson: fileName, + bytes, + sha256, + vector_dimensions: vectorDimensions + }; + + collections.push(entry); + encodedCollections.push({ entry, body }); + } + + const hasVectors = collections.some(c => c.vector_dimensions.length > 0); + if (hasVectors && !opts.embedding) { + throw new Error( + "archive contains vectors but no embedding configuration was provided" + ); + } + + const embedding = hasVectors ? opts.embedding : null; + const now = opts.now ? opts.now() : new Date(); + + const manifest: FullBackupManifest = { + schema: MANIFEST_SCHEMA, + source_project: store.projectId, + generated_at: now.toISOString(), + generated_at_ms: now.getTime(), + discovery: "recursive-listCollections", + embedding, + collection_paths: collectionPaths, + collections, + collection_ids: collectionPaths + }; + + if (opts.dryRun) { + return manifest; + } + + fs.mkdirSync(archiveDir, { recursive: true }); + for (const { entry, body } of encodedCollections) { + fs.writeFileSync(path.join(archiveDir, entry.ndjson), body, "utf8"); + } + + writeManifest(archiveDir, manifest); + removeStaleNdjsonFiles(archiveDir, collectionPaths); + + return manifest; +} diff --git a/functions/scripts/support/mirror-restore.ts b/functions/scripts/support/mirror-restore.ts new file mode 100644 index 0000000..60fa3ca --- /dev/null +++ b/functions/scripts/support/mirror-restore.ts @@ -0,0 +1,13 @@ +export { + IncompatibleVectorsError, + resolveTargetPath, + restoreArchive, + RestoreIncompleteError, + RestoreTargetError, + RestoreVerificationError, + serializeNdjson, + verifyRestore, + type RestoreCollectionReport, + type RestoreOptions, + type RestoreReport +} from "./firestore-full-mirror.js"; diff --git a/functions/scripts/support/mirror-store.ts b/functions/scripts/support/mirror-store.ts new file mode 100644 index 0000000..b048170 --- /dev/null +++ b/functions/scripts/support/mirror-store.ts @@ -0,0 +1,12 @@ +export { + assertValidCollectionPath, + assertWritableProject, + firestoreProjectId, + FirestoreMirrorStore, + isValidCollectionPath, + PAGE_SIZE, + PROD_PROJECT_ID, + WRITE_BATCH_LIMIT, + type MirrorDocument, + type MirrorStore +} from "./firestore-full-mirror.js"; diff --git a/functions/scripts/support/mirror-validate.ts b/functions/scripts/support/mirror-validate.ts new file mode 100644 index 0000000..4c2fe7a --- /dev/null +++ b/functions/scripts/support/mirror-validate.ts @@ -0,0 +1,16 @@ +export { + ArchiveValidationError, + computeSha256, + extractVectorDimensions, + MANIFEST_FILE, + MANIFEST_SCHEMA, + MANIFEST_SCHEMA_V1, + ndjsonFileName, + parseNdjsonLines, + readManifest, + validateArchive, + type FullBackupCollectionEntry, + type FullBackupManifest, + type ValidatedArchive, + type ValidatedArchiveCollection +} from "./firestore-full-mirror.js"; diff --git a/functions/src/archive.ts b/functions/src/archive.ts index d2844d4..0805841 100644 --- a/functions/src/archive.ts +++ b/functions/src/archive.ts @@ -223,6 +223,99 @@ export function describeEmbeddingMismatch( ); } +export function validatePortableArchive( + memories: readonly ArchivedMemory[], + manifest: ArchiveManifest +): string[] { + const problems: string[] = []; + + if (manifest.schema_version !== ARCHIVE_SCHEMA_VERSION) { + problems.push( + `schema_version mismatch: expected ${ARCHIVE_SCHEMA_VERSION}, got ${manifest.schema_version}` + ); + } + + if (memories.length !== manifest.document_count) { + problems.push( + `document count mismatch: archive has ${memories.length} documents, manifest records ${manifest.document_count}` + ); + } + + const seenIds = new Set(); + for (let index = 0; index < memories.length; index += 1) { + const memory = memories[index]; + if (typeof memory.id !== "string" || memory.id.trim().length === 0) { + problems.push(`memory at index ${index} has empty or missing id`); + } else if (seenIds.has(memory.id)) { + problems.push(`duplicate id in archive: ${memory.id}`); + } else { + seenIds.add(memory.id); + } + + if (typeof memory.content !== "string" || memory.content.length === 0) { + problems.push(`memory ${memory.id ?? index} has empty or missing content`); + } + + if ( + typeof memory.retrieval_text !== "string" || + memory.retrieval_text.length === 0 + ) { + problems.push( + `memory ${memory.id ?? index} has empty or missing retrieval_text` + ); + } + + if (!memory.metadata || typeof memory.metadata !== "object") { + problems.push(`memory ${memory.id ?? index} is missing metadata object`); + } else { + if ( + typeof memory.metadata.module_name !== "string" || + memory.metadata.module_name.length === 0 + ) { + problems.push( + `memory ${memory.id ?? index} metadata.module_name must be a non-empty string` + ); + } + if ( + typeof memory.metadata.branch_state !== "string" || + memory.metadata.branch_state.length === 0 + ) { + problems.push( + `memory ${memory.id ?? index} metadata.branch_state must be a string` + ); + } + if ( + typeof memory.metadata.created_at !== "number" || + !Number.isFinite(memory.metadata.created_at) + ) { + problems.push( + `memory ${memory.id ?? index} metadata.created_at must be a finite number` + ); + } + if ( + typeof memory.metadata.updated_at !== "number" || + !Number.isFinite(memory.metadata.updated_at) + ) { + problems.push( + `memory ${memory.id ?? index} metadata.updated_at must be a finite number` + ); + } + } + } + + for (const memory of memories) { + if (memory.metadata?.superseded_by) { + if (!seenIds.has(memory.metadata.superseded_by)) { + problems.push( + `memory ${memory.id} has dangling superseded_by reference to ${memory.metadata.superseded_by}` + ); + } + } + } + + return problems; +} + export interface RestoreTarget { writeMemory(memory: ArchivedMemory, embedding: number[]): Promise; } @@ -231,6 +324,7 @@ export interface RestoreParams { memories: readonly ArchivedMemory[]; embeddings: EmbeddingClient; target: RestoreTarget; + manifest?: ArchiveManifest; } export interface RestoreResult { @@ -240,6 +334,13 @@ export interface RestoreResult { export async function restoreMemories( params: RestoreParams ): Promise { + if (params.manifest) { + const problems = validatePortableArchive(params.memories, params.manifest); + if (problems.length > 0) { + throw new Error(`archive failed validation: ${problems.join("; ")}`); + } + } + let restored = 0; for (const memory of params.memories) { diff --git a/functions/test/recovery.emulator.test.ts b/functions/test/recovery.emulator.test.ts new file mode 100644 index 0000000..f74adde --- /dev/null +++ b/functions/test/recovery.emulator.test.ts @@ -0,0 +1,143 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { afterAll, describe, expect, it } from "vitest"; +import { GeoPoint, Timestamp } from "firebase-admin/firestore"; + +import { + backupToArchive, + openFirestore, + restoreArchive, + RestoreVerificationError, + verifyRestore +} from "../scripts/support/firestore-full-mirror.js"; +import { FirestoreMirrorStore } from "../scripts/support/mirror-store.js"; +import { FirestoreMemoryRepository } from "../src/memoryRepository.js"; +import { seedFirestoreCorpus } from "./support/recoveryCorpus.js"; + +function makeTempDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +async function clearEmulatorData(projectId: string): Promise { + const host = process.env.FIRESTORE_EMULATOR_HOST; + if (!host) { + return; + } + try { + await fetch( + `http://${host}/emulator/v1/projects/${projectId}/databases/(default)/documents`, + { method: "DELETE" } + ); + } catch { + // best-effort cleanup + } +} + +describe.skipIf(!process.env.FIRESTORE_EMULATOR_HOST)( + "recovery.emulator.test.ts - Firestore emulator recovery test", + () => { + const sourceProjectId = "demo-recovery-source"; + const targetProjectId = "demo-recovery-target"; + const runPrefix = `emu_${Date.now()}_`; + + const sourceDb = openFirestore(sourceProjectId, "emu-source-role"); + const targetDb = openFirestore(targetProjectId, "emu-target-role"); + + const sourceStore = new FirestoreMirrorStore(sourceDb); + const targetStore = new FirestoreMirrorStore(targetDb); + + afterAll(async () => { + await clearEmulatorData(sourceProjectId); + await clearEmulatorData(targetProjectId); + }); + + it("runs full recursive discovery, backup, restore, corruption detection, and vector search on emulator", async () => { + const archiveDir = makeTempDir("emu-recovery-"); + + // 1. Seed corpus on source + const paths = await seedFirestoreCorpus(sourceDb, runPrefix); + + // 2. Recursive discovery including phantom parent + const discoveredPaths = await sourceStore.discoverCollectionPaths(); + expect(discoveredPaths).toContain(paths.memoryVectorsPath); + expect(discoveredPaths).toContain(paths.fingerprintsPath); + expect(discoveredPaths).toContain(paths.futurePath); + expect(discoveredPaths).toContain(paths.notesPath); + expect(discoveredPaths).toContain(paths.ghostChildrenPath); + + // Verify phantom parent doc does not exist + const ghostParentPath = `${runPrefix}ghost_parent`; + const ghostParentSnap = await sourceDb.collection(ghostParentPath).get(); + expect(ghostParentSnap.empty).toBe(true); + // But child exists + const ghostChildSnap = await sourceDb.collection(paths.ghostChildrenPath).get(); + expect(ghostChildSnap.size).toBe(1); + + // 3. Backup to archive + const manifest = await backupToArchive(sourceStore, archiveDir, { + embedding: { + model: "text-embedding-004", + dimensions: 6 + } + }); + expect(manifest.discovery).toBe("recursive-listCollections"); + expect(manifest.collection_paths).toContain(paths.memoryVectorsPath); + + // 4. Restore into target project + const restoreReport = await restoreArchive(targetStore, archiveDir, { + write: true, + targetEmbedding: { + model: "text-embedding-004", + dimensions: 6 + } + }); + expect(restoreReport.complete).toBe(true); + + // 5. Verify real Firestore types on target + const memSnap = await targetDb.collection(paths.memoryVectorsPath).doc("doc-1").get(); + expect(memSnap.exists).toBe(true); + const memData = memSnap.data()!; + expect(memData.content).toContain("Ktor networking"); + expect(memData.metadata.created_time).toBeInstanceOf(Timestamp); + + const futureSnap = await targetDb.collection(paths.futurePath).doc("doc-unknown").get(); + expect(futureSnap.exists).toBe(true); + const futureData = futureSnap.data()!; + expect(futureData.geo).toBeInstanceOf(GeoPoint); + expect(futureData.geo.latitude).toBeCloseTo(37.7749); + expect(futureData.geo.longitude).toBeCloseTo(-122.4194); + expect(Buffer.isBuffer(futureData.bytes)).toBe(true); + expect((futureData.bytes as Buffer).toString("utf8")).toBe("binary-payload-data"); + expect(typeof futureData.ref).toBe("object"); + expect(futureData.ref.path).toContain("future_collection/doc-unknown"); + + // 6. Content corruption detection on target + await targetDb.collection(paths.memoryVectorsPath).doc("doc-1").update({ + content: "Corrupted content in live target" + }); + await expect( + verifyRestore(targetStore, archiveDir) + ).rejects.toThrow(RestoreVerificationError); + + // Restore back so vector search can be exercised + await targetDb.collection(paths.memoryVectorsPath).doc("doc-1").update({ + content: memData.content + }); + const reverified = await verifyRestore(targetStore, archiveDir); + expect(reverified.complete).toBe(true); + + // 7. Vector search via FirestoreMemoryRepository without re-embedding + const repository = new FirestoreMemoryRepository(targetDb, paths.memoryVectorsPath); + const searchResults = await repository.search({ + queryVector: [1, 0, 0, 0, 0, 0], + limit: 5, + filterState: "active" + }); + expect(searchResults.length).toBeGreaterThan(0); + expect(searchResults[0].id).toBe("doc-1"); + expect(searchResults[0].content).toContain("Ktor networking is preferred"); + }); + } +); diff --git a/functions/test/recovery.test.ts b/functions/test/recovery.test.ts new file mode 100644 index 0000000..b6fb0f8 --- /dev/null +++ b/functions/test/recovery.test.ts @@ -0,0 +1,868 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { describe, expect, it } from "vitest"; +import { FieldValue, GeoPoint, Timestamp } from "firebase-admin/firestore"; + +import { + ARCHIVE_SCHEMA_VERSION, + restoreMemories, + validatePortableArchive, + type ArchiveManifest, + type ArchivedMemory +} from "../src/archive.js"; +import { + decodeDocument, + decodeValue, + encodeDocument, + encodeValue, + stableStringify +} from "../scripts/support/firestore-codec.js"; +import { + backupToArchive, + MANIFEST_SCHEMA, + MANIFEST_SCHEMA_V1, + readManifest, + writeManifest +} from "../scripts/support/firestore-full-mirror.js"; +import { + ArchiveValidationError, + validateArchive +} from "../scripts/support/mirror-validate.js"; +import { + IncompatibleVectorsError, + restoreArchive, + RestoreIncompleteError, + RestoreTargetError, + RestoreVerificationError, + verifyRestore +} from "../scripts/support/mirror-restore.js"; +import { + DocumentReference, + InMemoryMirrorStore +} from "./support/inMemoryMirrorStore.js"; +import { + getCorpusData, + seedInMemoryCorpus +} from "./support/recoveryCorpus.js"; +import { + createTestConfig, + FakeLlmMergeClient, + FakeMemoryContentPreparer, + InMemoryMemoryRepository, + KeywordEmbeddingClient +} from "./support/fakes.js"; +import { MetaCortexService } from "../src/service.js"; + +function makeTempDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +describe("recovery.test.ts - card 71 backup & restore hardening", () => { + const targetEmbedding = { + model: "text-embedding-004", + dimensions: 6 + }; + + // Case 1: Discovery + it("Case 1: discovers root, subcollection, phantom-parent subcollection, unknown collection without hardcoding", async () => { + const store = new InMemoryMirrorStore("test-source"); + await seedInMemoryCorpus(store); + + const paths = await store.discoverCollectionPaths(); + expect(paths).toEqual([ + "future_collection", + "ghost_parent", + "ghost_parent/ghost-doc-1/children", + "memory_vectors", + "memory_vectors_write_fingerprints", + "memory_vectors/doc-1/notes" + ]); + + // ghost-doc-1 does not exist in ghost_parent (it is phantom) + const ghostParentDocs = await store.readCollection("ghost_parent"); + expect(ghostParentDocs).toHaveLength(0); + const ghostChildDocs = await store.readCollection("ghost_parent/ghost-doc-1/children"); + expect(ghostChildDocs).toHaveLength(1); + expect(ghostChildDocs[0].id).toBe("child-1"); + }); + + // Case 2: Backup -> Validate -> Restore -> Verify with field-by-field equality + it("Case 2: full backup, validation, and restore into separate project with exact type and lineage preservation", async () => { + const archiveDir = makeTempDir("rec-case-2-"); + const sourceStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(sourceStore); + + const manifest = await backupToArchive(sourceStore, archiveDir, { + embedding: targetEmbedding + }); + expect(manifest.schema).toBe(MANIFEST_SCHEMA); + expect(manifest.discovery).toBe("recursive-listCollections"); + + // Validate archive + const validated = validateArchive(archiveDir); + expect(validated.manifest.collection_paths).toEqual(manifest.collection_paths); + + // Restore into a different project + const targetStore = new InMemoryMirrorStore("target-project"); + const report = await restoreArchive(targetStore, archiveDir, { + write: true, + targetEmbedding + }); + + expect(report.complete).toBe(true); + expect(report.collections.every(c => c.status === "verified")).toBe(true); + + // Field-by-field assertions + // 1. memory_vectors + const memDocs = await targetStore.readCollection("memory_vectors"); + expect(memDocs).toHaveLength(2); + const doc1 = memDocs.find(d => d.id === "doc-1")!; + const doc2 = memDocs.find(d => d.id === "doc-2")!; + expect(doc1.data.content).toBe("Ktor networking is preferred for Kotlin multiplatform clients."); + expect(doc1.data.retrieval_text).toBe("ktor networking"); + expect((doc1.data.embedding as any).toArray()).toEqual([1, 0, 0, 0, 0, 0]); + expect((doc1.data.metadata as any).created_time).toBeInstanceOf(Timestamp); + expect(((doc1.data.metadata as any).created_time as Timestamp).seconds).toBe(1_700_000_000); + expect(((doc1.data.metadata as any).created_time as Timestamp).nanoseconds).toBe(0); + + // Lineage check: doc2 superseded_by doc1 + expect((doc2.data.metadata as any).superseded_by).toBe("doc-1"); + expect(memDocs.some(d => d.id === (doc2.data.metadata as any).superseded_by)).toBe(true); + + // 2. future_collection (GeoPoint, Buffer, DocumentReference, NaN, Infinity, nested) + const futureDocs = await targetStore.readCollection("future_collection"); + expect(futureDocs).toHaveLength(1); + const futureDoc = futureDocs[0]; + expect(futureDoc.data.geo).toBeInstanceOf(GeoPoint); + expect((futureDoc.data.geo as GeoPoint).latitude).toBeCloseTo(37.7749); + expect((futureDoc.data.geo as GeoPoint).longitude).toBeCloseTo(-122.4194); + + expect(Buffer.isBuffer(futureDoc.data.bytes)).toBe(true); + expect((futureDoc.data.bytes as Buffer).toString("utf8")).toBe("binary-payload-data"); + + expect(futureDoc.data.ref).toBeInstanceOf(DocumentReference); + expect((futureDoc.data.ref as DocumentReference).path).toBe("future_collection/doc-unknown"); + + expect(Number.isNaN(futureDoc.data.not_a_number)).toBe(true); + expect(futureDoc.data.pos_inf).toBe(Number.POSITIVE_INFINITY); + expect(futureDoc.data.neg_inf).toBe(Number.NEGATIVE_INFINITY); + + expect(futureDoc.data.nested_map).toEqual({ + alpha: "hello", + beta: 42, + inner: { deep: true } + }); + expect(futureDoc.data.nested_array).toEqual([1, 2, "three", { x: 10 }]); + + // 3. Phantom parent: ghost_parent has 0 docs, child exists + const ghostParentDocs = await targetStore.readCollection("ghost_parent"); + expect(ghostParentDocs).toHaveLength(0); + const ghostChildren = await targetStore.readCollection("ghost_parent/ghost-doc-1/children"); + expect(ghostChildren).toHaveLength(1); + expect(ghostChildren[0].id).toBe("child-1"); + expect(ghostChildren[0].data.child_name).toBe("Casper"); + }); + + // Case 3: Equal-count/different-content corruption + it("Case 3: detects corruption with sha mismatch on edit, and detects post-restore corruption in target", async () => { + const archiveDir = makeTempDir("rec-case-3-"); + const sourceStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(sourceStore); + await backupToArchive(sourceStore, archiveDir, { embedding: targetEmbedding }); + + // Variant A: Corrupt one doc inside an ndjson file without updating sha + const ndjsonPath = path.join(archiveDir, "future_collection.ndjson"); + const originalNdjson = fs.readFileSync(ndjsonPath, "utf8"); + const tampered = originalNdjson.replace("hello", "tampered_value"); + expect(tampered).not.toBe(originalNdjson); + fs.writeFileSync(ndjsonPath, tampered, "utf8"); + + // Validation must fail with sha mismatch + expect(() => validateArchive(archiveDir)).toThrow(ArchiveValidationError); + try { + validateArchive(archiveDir); + } catch (err) { + const p = (err as ArchiveValidationError).problems; + expect(p.some(msg => msg.includes("sha256 mismatch"))).toBe(true); + } + + // Revert tamper + fs.writeFileSync(ndjsonPath, originalNdjson, "utf8"); + expect(() => validateArchive(archiveDir)).not.toThrow(); + + // Variant B: Target corruption after restore + const targetStore = new InMemoryMirrorStore("target-project"); + await restoreArchive(targetStore, archiveDir, { + write: true, + targetEmbedding + }); + + // Mutate one doc in target while preserving count + const corruptedDocs = await targetStore.readCollection("memory_vectors"); + const docToCorrupt = corruptedDocs.find(d => d.id === "doc-1")!; + docToCorrupt.data.content = "Corrupted content after restore"; + await targetStore.writeDocuments("memory_vectors", [{ id: "doc-1", data: docToCorrupt.data }]); + + // verifyRestore reports RestoreVerificationError naming the doc id + await expect( + verifyRestore(targetStore, archiveDir) + ).rejects.toThrow(RestoreVerificationError); + + try { + await verifyRestore(targetStore, archiveDir); + } catch (err) { + expect(err).toBeInstanceOf(RestoreVerificationError); + expect((err as Error).message).toContain("doc-1"); + } + }); + + // Case 4: Validation failures + it("Case 4: asserts ArchiveValidationError.problems for all validation rules and ensures zero writes before failure", async () => { + const dir = makeTempDir("rec-case-4-"); + const store = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(store); + await backupToArchive(store, dir, { embedding: targetEmbedding }); + + // 1. Missing file + const missingFileDir = makeTempDir("rec-case-4-missing-"); + fs.cpSync(dir, missingFileDir, { recursive: true }); + fs.unlinkSync(path.join(missingFileDir, "future_collection.ndjson")); + try { + validateArchive(missingFileDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("missing ndjson file"))).toBe(true); + } + + // 2. Duplicate _id + const dupIdDir = makeTempDir("rec-case-4-dupid-"); + fs.cpSync(dir, dupIdDir, { recursive: true }); + const fpFile = path.join(dupIdDir, "memory_vectors_write_fingerprints.ndjson"); + const fpLines = fs.readFileSync(fpFile, "utf8").trim(); + fs.writeFileSync(fpFile, `${fpLines}\n${fpLines}\n`, "utf8"); + // Also update manifest count so count matches line count, isolating duplicate _id check + const dupManifest = readManifest(dupIdDir); + dupManifest.collections.find(c => c.path === "memory_vectors_write_fingerprints")!.document_count = 2; + dupManifest.collections.find(c => c.path === "memory_vectors_write_fingerprints")!.sha256 = "dummy"; + // We expect sha mismatch and duplicate _id + try { + validateArchive(dupIdDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("duplicate _id"))).toBe(true); + } + + // 3. Line count != document_count + const countMismatchDir = makeTempDir("rec-case-4-count-"); + fs.cpSync(dir, countMismatchDir, { recursive: true }); + const countManifest = readManifest(countMismatchDir); + countManifest.collections[0].document_count = 999; + writeManifest(countMismatchDir, countManifest); + try { + validateArchive(countMismatchDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("document count mismatch"))).toBe(true); + } + + // 4. Doc missing _id + const missingIdDir = makeTempDir("rec-case-4-noid-"); + fs.cpSync(dir, missingIdDir, { recursive: true }); + fs.writeFileSync(path.join(missingIdDir, "future_collection.ndjson"), "{\"alpha\": 1}\n", "utf8"); + try { + validateArchive(missingIdDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("missing non-empty string _id"))).toBe(true); + } + + // 5. Unknown __fs tag + const unknownTagDir = makeTempDir("rec-case-4-unknowntag-"); + fs.cpSync(dir, unknownTagDir, { recursive: true }); + fs.writeFileSync( + path.join(unknownTagDir, "future_collection.ndjson"), + "{\"_id\":\"doc-x\",\"bad\":{\"__fs\":\"alien_type\"}}\n", + "utf8" + ); + try { + validateArchive(unknownTagDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("unknown __fs tag"))).toBe(true); + } + + // 6. Even path segments + const evenPathDir = makeTempDir("rec-case-4-evenpath-"); + fs.cpSync(dir, evenPathDir, { recursive: true }); + const evenManifest = readManifest(evenPathDir); + evenManifest.collection_paths.push("even/segment"); + evenManifest.collections.push({ + path: "even/segment", + document_count: 0, + ndjson: "even%2Fsegment.ndjson", + bytes: 0, + sha256: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + vector_dimensions: [] + }); + fs.writeFileSync(path.join(evenPathDir, "even%2Fsegment.ndjson"), "", "utf8"); + writeManifest(evenPathDir, evenManifest); + try { + validateArchive(evenPathDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("invalid collection path"))).toBe(true); + } + + // 7. Vector dimension disagreeing with the manifest + const dimMismatchDir = makeTempDir("rec-case-4-dim-"); + fs.cpSync(dir, dimMismatchDir, { recursive: true }); + const dimManifest = readManifest(dimMismatchDir); + dimManifest.collections.find(c => c.path === "memory_vectors")!.vector_dimensions = [128]; + writeManifest(dimMismatchDir, dimManifest); + try { + validateArchive(dimMismatchDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("vector dimension"))).toBe(true); + } + + // 8. Malformed JSON line + const malformedJsonDir = makeTempDir("rec-case-4-json-"); + fs.cpSync(dir, malformedJsonDir, { recursive: true }); + fs.writeFileSync(path.join(malformedJsonDir, "future_collection.ndjson"), "{not-valid-json\n", "utf8"); + try { + validateArchive(malformedJsonDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("malformed JSON line"))).toBe(true); + } + + // 9. Unsupported manifest schema + const schemaDir = makeTempDir("rec-case-4-schema-"); + fs.cpSync(dir, schemaDir, { recursive: true }); + const schemaManifest = readManifest(schemaDir); + (schemaManifest as any).schema = "unsupported-schema-v99"; + writeManifest(schemaDir, schemaManifest); + try { + validateArchive(schemaDir); + expect.unreachable("expected error"); + } catch (err) { + expect((err as ArchiveValidationError).problems.some(p => p.includes("unsupported manifest schema"))).toBe(true); + } + + // 10. Validation precedes writes assertion: a corrupt archive plus a store shows zero writes + let writeCalls = 0; + const writeSpyStore = new InMemoryMirrorStore("spy-project"); + const origWrite = writeSpyStore.writeDocuments.bind(writeSpyStore); + writeSpyStore.writeDocuments = async (p, docs) => { + writeCalls += 1; + return origWrite(p, docs); + }; + + await expect( + restoreArchive(writeSpyStore, malformedJsonDir, { + write: true, + targetEmbedding + }) + ).rejects.toThrow(ArchiveValidationError); + expect(writeCalls).toBe(0); + }); + + // Case 5: Target protection + it("Case 5: enforces target protections, prefixes, prune rules and extra doc reporting", async () => { + const dir = makeTempDir("rec-case-5-"); + const sourceStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(sourceStore); + await backupToArchive(sourceStore, dir, { embedding: targetEmbedding }); + + // 1. Same project without prefix or confirmation throws RestoreTargetError + const sameProjectStore = new InMemoryMirrorStore("source-project"); + await expect( + restoreArchive(sameProjectStore, dir, { + write: true, + targetEmbedding + }) + ).rejects.toThrow(RestoreTargetError); + + // 2. Prod project id refused + const prodStore = new InMemoryMirrorStore("my-brain-88870"); + await expect( + restoreArchive(prodStore, dir, { + write: true, + targetEmbedding + }) + ).rejects.toThrow(/refusing writes to prod project/); + + // 3. targetPrefix into same project works and leaves original untouched + const prefixStore = new InMemoryMirrorStore("source-project"); + // Seed original data + await seedInMemoryCorpus(prefixStore); + const origFutureDocs = await prefixStore.readCollection("future_collection"); + expect(origFutureDocs).toHaveLength(1); + + const restorePrefixResult = await restoreArchive(prefixStore, dir, { + write: true, + targetPrefix: "backup_", + targetEmbedding + }); + expect(restorePrefixResult.complete).toBe(true); + + // Check prefixed collections exist + const prefixedFuture = await prefixStore.readCollection("backup_future_collection"); + expect(prefixedFuture).toHaveLength(1); + // Original untouched + const untouchedFuture = await prefixStore.readCollection("future_collection"); + expect(untouchedFuture).toHaveLength(1); + + // 4. Prune/clear without --confirm-project refused + const targetStore = new InMemoryMirrorStore("target-project"); + await expect( + restoreArchive(targetStore, dir, { + write: true, + targetEmbedding, + replace: { + prune: true, + clearTarget: false, + confirmProject: undefined + } + }) + ).rejects.toThrow(/replacement requires --confirm-project/); + + // 5. Default upsert leaves extra target docs alone and reports extra_target_docs + const upsertStore = new InMemoryMirrorStore("target-project"); + // Pre-seed an extra doc in memory_vectors + await upsertStore.writeDocuments("memory_vectors", [ + { id: "extra-doc", data: { content: "I am an extra document", retrieval_text: "extra" } } + ]); + const upsertResult = await restoreArchive(upsertStore, dir, { + write: true, + targetEmbedding + }); + expect(upsertResult.complete).toBe(true); + const memReport = upsertResult.collections.find(c => c.path === "memory_vectors")!; + expect(memReport.extra_target_docs).toBe(1); + // Extra doc is still there + const memAfter = await upsertStore.readCollection("memory_vectors"); + expect(memAfter.some(d => d.id === "extra-doc")).toBe(true); + + // 6. Prune with confirm removes only extra docs inside archived paths, leaves unrelated alone + await upsertStore.writeDocuments("unrelated_collection", [ + { id: "keep-me", data: { value: "unrelated" } } + ]); + const pruneResult = await restoreArchive(upsertStore, dir, { + write: true, + targetEmbedding, + replace: { + prune: true, + clearTarget: false, + confirmProject: "target-project" + } + }); + expect(pruneResult.complete).toBe(true); + // Extra doc was pruned + const memAfterPrune = await upsertStore.readCollection("memory_vectors"); + expect(memAfterPrune.some(d => d.id === "extra-doc")).toBe(false); + // Unrelated collection untouched + const unrelatedDocs = await upsertStore.readCollection("unrelated_collection"); + expect(unrelatedDocs).toHaveLength(1); + expect(unrelatedDocs[0].id).toBe("keep-me"); + }); + + // Case 6: Vector compatibility + it("Case 6: validates vector compatibility and throws IncompatibleVectorsError before any writes", async () => { + const dir = makeTempDir("rec-case-6-"); + const sourceStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(sourceStore); + await backupToArchive(sourceStore, dir, { embedding: targetEmbedding }); + + let writes = 0; + const targetStore = new InMemoryMirrorStore("target-project"); + const origWrite = targetStore.writeDocuments.bind(targetStore); + targetStore.writeDocuments = async (p, docs) => { + writes += 1; + return origWrite(p, docs); + }; + + // 1. Mismatched model + await expect( + restoreArchive(targetStore, dir, { + write: true, + targetEmbedding: { model: "different-model", dimensions: 6 } + }) + ).rejects.toThrow(IncompatibleVectorsError); + expect(writes).toBe(0); + + // 2. Mismatched dimensions + await expect( + restoreArchive(targetStore, dir, { + write: true, + targetEmbedding: { model: "text-embedding-004", dimensions: 768 } + }) + ).rejects.toThrow(IncompatibleVectorsError); + expect(writes).toBe(0); + + // 3. Target embedding null when archive has vectors + await expect( + restoreArchive(targetStore, dir, { + write: true, + targetEmbedding: null + }) + ).rejects.toThrow(IncompatibleVectorsError); + expect(writes).toBe(0); + + // 4. v1 manifest without accept flag + const v1Dir = makeTempDir("rec-case-6-v1-"); + fs.cpSync(dir, v1Dir, { recursive: true }); + const v1Manifest = readManifest(v1Dir); + (v1Manifest as any).schema = MANIFEST_SCHEMA_V1; + (v1Manifest as any).embedding = null; + writeManifest(v1Dir, v1Manifest); + await expect( + restoreArchive(targetStore, v1Dir, { + write: true, + targetEmbedding + }) + ).rejects.toThrow(IncompatibleVectorsError); + expect(writes).toBe(0); + + // v1 with acceptUnknownEmbeddingSpace works + const v1Accepted = await restoreArchive(targetStore, v1Dir, { + write: true, + targetEmbedding, + acceptUnknownEmbeddingSpace: true + }); + expect(v1Accepted.complete).toBe(true); + + // 5. Archive with no vectors restores regardless of target embedding + const noVectorStore = new InMemoryMirrorStore("no-vector-source"); + await noVectorStore.writeDocuments("simple", [{ id: "doc-1", data: { text: "hello" } }]); + const noVectorDir = makeTempDir("rec-case-6-novectors-"); + await backupToArchive(noVectorStore, noVectorDir, { embedding: null }); + + const noVectorTarget = new InMemoryMirrorStore("no-vector-target"); + const noVectorResult = await restoreArchive(noVectorTarget, noVectorDir, { + write: true, + targetEmbedding: null + }); + expect(noVectorResult.complete).toBe(true); + }); + + // Case 7: Interrupted restore + it("Case 7: handles interrupted restore with RestoreIncompleteError and succeeds upon idempotent retry", async () => { + const dir = makeTempDir("rec-case-7-"); + const sourceStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(sourceStore); + await backupToArchive(sourceStore, dir, { embedding: targetEmbedding }); + + const targetStore = new InMemoryMirrorStore("target-project"); + // Fail after the 1st writeDocuments call + targetStore.failAfterWrites(1); + + try { + await restoreArchive(targetStore, dir, { + write: true, + targetEmbedding + }); + expect.unreachable("expected failure"); + } catch (err) { + expect(err).toBeInstanceOf(RestoreIncompleteError); + expect((err as Error).message).toMatch(/^RESTORE INCOMPLETE:/); + expect((err as Error).message).toContain("re-run the same command to finish; upserts are idempotent"); + const inc = err as RestoreIncompleteError; + expect(inc.completedCollections).toHaveLength(1); + } + + // Verify target is partially populated + const firstCol = (await sourceStore.discoverCollectionPaths())[0]; + const targetDocs = await targetStore.readCollection(firstCol); + expect(targetDocs.length).toBeGreaterThan(0); + + // Clear fault and re-run: completes and verifies + targetStore.clearFaults(); + const resumed = await restoreArchive(targetStore, dir, { + write: true, + targetEmbedding + }); + expect(resumed.complete).toBe(true); + expect(resumed.collections.every(c => c.status === "verified")).toBe(true); + }); + + // Case 8: Retrieval after restore + it("Case 8: verifies search retrieval over restored memory_vectors repository", async () => { + const dir = makeTempDir("rec-case-8-"); + const sourceStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(sourceStore); + await backupToArchive(sourceStore, dir, { embedding: targetEmbedding }); + + const targetStore = new InMemoryMirrorStore("target-project"); + await restoreArchive(targetStore, dir, { + write: true, + targetEmbedding + }); + + const restoredDocs = await targetStore.readCollection("memory_vectors"); + const repository = new InMemoryMemoryRepository(); + repository.seed( + restoredDocs.map(d => ({ + id: d.id, + content: d.data.content as string, + retrieval_text: d.data.retrieval_text as string, + embedding: d.data.embedding as any, + metadata: d.data.metadata as any + })) + ); + + const service = new MetaCortexService( + new FakeMemoryContentPreparer(), + new KeywordEmbeddingClient(), + repository, + createTestConfig(), + new FakeLlmMergeClient() + ); + + const searchResult = await service.searchContext({ + query: "ktor networking", + filter_state: "active" + }); + + expect(searchResult.matches).toHaveLength(1); + expect(searchResult.matches[0].id).toBe("doc-1"); + expect(searchResult.matches[0].content).toContain("Ktor networking is preferred"); + + // doc-2 was deprecated, so searching active branch does not return it + expect(searchResult.matches.some(r => r.id === "doc-2")).toBe(false); + }); + + // Case 9: Codec round trip and errors + it("Case 9: tests codec round-trips, unsupported type errors, reserved keys, and stableStringify", () => { + // 1. Supported types round-trip + const ref = new DocumentReference("col/doc-1"); + const testDoc = { + ts: new Timestamp(1234567, 890), + vec: FieldValue.vector([0.1, 0.2, 0.3]), + geo: new GeoPoint(10.5, 20.5), + buf: Buffer.from("hello world"), + ref, + nan: Number.NaN, + posInf: Number.POSITIVE_INFINITY, + negInf: Number.NEGATIVE_INFINITY, + nested: { a: 1, b: [2, 3] } + }; + + const encoded = encodeDocument("doc-test", testDoc); + const decoded = decodeDocument(encoded, p => new DocumentReference(p)); + expect(decoded.id).toBe("doc-test"); + expect(decoded.data.ts).toBeInstanceOf(Timestamp); + expect((decoded.data.ts as Timestamp).seconds).toBe(1234567); + expect((decoded.data.ts as Timestamp).nanoseconds).toBe(890); + expect((decoded.data.vec as any).toArray()).toEqual([0.1, 0.2, 0.3]); + expect(decoded.data.geo).toBeInstanceOf(GeoPoint); + expect((decoded.data.geo as GeoPoint).latitude).toBe(10.5); + expect((decoded.data.buf as Buffer).toString("utf8")).toBe("hello world"); + expect((decoded.data.ref as DocumentReference).path).toBe("col/doc-1"); + expect(Number.isNaN(decoded.data.nan)).toBe(true); + expect(decoded.data.posInf).toBe(Number.POSITIVE_INFINITY); + expect(decoded.data.negInf).toBe(Number.NEGATIVE_INFINITY); + expect(decoded.data.nested).toEqual({ a: 1, b: [2, 3] }); + + // 2. Unsupported class instance throws with field path + class UnrecognizedClass {} + expect(() => + encodeValue({ details: { custom: new UnrecognizedClass() } }) + ).toThrow(/cannot encode value of type UnrecognizedClass at details\.custom/); + + // 3. Reserved __fs key throws + expect(() => + encodeValue({ payload: { __fs: "fake" } }) + ).toThrow(/reserved key "__fs" in field payload\.__fs/); + + // 4. stableStringify ordering + const objA = { z: 1, b: 2, a: { y: 10, x: 20 } }; + const objB = { a: { x: 20, y: 10 }, z: 1, b: 2 }; + expect(stableStringify(objA)).toBe(stableStringify(objB)); + expect(stableStringify(objA)).toBe("{\"a\":{\"x\":20,\"y\":10},\"b\":2,\"z\":1}"); + }); + + // Case 10: Backup manifest written last, stale ndjson removed, error conditions + it("Case 10: ensures manifest is written last, removes stale ndjson, and verifies backup constraints", async () => { + const dir = makeTempDir("rec-case-10-"); + const store = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(store); + + // 1. Simulate failure during readCollection of 2nd collection: manifest must NOT exist in fresh dir + const failDir = makeTempDir("rec-case-10-fail-"); + let callCount = 0; + const failingStore = new InMemoryMirrorStore("source-project"); + await seedInMemoryCorpus(failingStore); + const origRead = failingStore.readCollection.bind(failingStore); + failingStore.readCollection = async p => { + callCount += 1; + if (callCount === 2) { + throw new Error("simulated failure reading second collection"); + } + return origRead(p); + }; + + await expect( + backupToArchive(failingStore, failDir, { embedding: targetEmbedding }) + ).rejects.toThrow("simulated failure reading second collection"); + expect(fs.existsSync(path.join(failDir, "manifest.json"))).toBe(false); + + // 2. Successful backup removes stale ndjson + await backupToArchive(store, dir, { embedding: targetEmbedding }); + fs.writeFileSync(path.join(dir, "obsolete%2Fcollection.ndjson"), "old data\n"); + expect(fs.existsSync(path.join(dir, "obsolete%2Fcollection.ndjson"))).toBe(true); + + // Re-run backup: obsolete ndjson removed + await backupToArchive(store, dir, { embedding: targetEmbedding }); + expect(fs.existsSync(path.join(dir, "obsolete%2Fcollection.ndjson"))).toBe(false); + + // 3. Zero-collection discovery refuses + const emptyStore = new InMemoryMirrorStore("empty-project"); + await expect( + backupToArchive(emptyStore, makeTempDir("rec-case-10-empty-"), { embedding: null }) + ).rejects.toThrow(/discovered zero collections/); + + // 4. Vectors found with embedding: null option throws + await expect( + backupToArchive(store, makeTempDir("rec-case-10-null-"), { embedding: null }) + ).rejects.toThrow(/archive contains vectors but no embedding configuration was provided/); + + // 5. Manifest embedding null when no vectors + const noVecStore = new InMemoryMirrorStore("no-vec"); + await noVecStore.writeDocuments("simple", [{ id: "1", data: { text: "hi" } }]); + const noVecManifest = await backupToArchive(noVecStore, makeTempDir("rec-case-10-no-vec-"), { + embedding: null + }); + expect(noVecManifest.embedding).toBeNull(); + }); + + // Case 11: Portable archive validation, restore rejection, and interruption + it("Case 11: tests validatePortableArchive rules, pre-write rejection, and interrupted portable restore", async () => { + const validMemory: ArchivedMemory = { + id: "doc-1", + content: "Valid memory content", + retrieval_text: "Valid retrieval text", + metadata: { + module_name: "android", + branch_state: "active", + created_at: 1_700_000_000_000, + updated_at: 1_700_000_000_000, + modality: "text" + } + }; + + const validManifest: ArchiveManifest = { + schema_version: ARCHIVE_SCHEMA_VERSION, + document_count: 1, + generated_at: 1_700_000_000_000, + project_id: "test-proj", + collection: "memory_vectors", + state_histogram: { active: 1 }, + embedding_model: "text-embedding-004", + embedding_dimensions: 768 + }; + + // 1. Schema version mismatch + const badSchemaProblems = validatePortableArchive([validMemory], { + ...validManifest, + schema_version: 99 + }); + expect(badSchemaProblems.some(p => p.includes("schema_version mismatch"))).toBe(true); + + // 2. Count mismatch + const badCountProblems = validatePortableArchive([validMemory], { + ...validManifest, + document_count: 5 + }); + expect(badCountProblems.some(p => p.includes("document count mismatch"))).toBe(true); + + // 3. Dangling superseded_by + const danglingMemory: ArchivedMemory = { + ...validMemory, + metadata: { + ...validMemory.metadata, + superseded_by: "non-existent-id" + } + }; + const danglingProblems = validatePortableArchive([danglingMemory], validManifest); + expect(danglingProblems.some(p => p.includes("non-existent-id"))).toBe(true); + + // 4. Duplicate id + const dupProblems = validatePortableArchive([validMemory, validMemory], { + ...validManifest, + document_count: 2 + }); + expect(dupProblems.some(p => p.includes("duplicate id in archive"))).toBe(true); + + // 5. restoreMemories with invalid manifest rejects before any writes + let writeCount = 0; + const failTarget = { + async writeMemory() { + writeCount += 1; + } + }; + await expect( + restoreMemories({ + memories: [danglingMemory], + embeddings: new KeywordEmbeddingClient(), + target: failTarget, + manifest: validManifest + }) + ).rejects.toThrow(/archive failed validation/); + expect(writeCount).toBe(0); + + // 6. Interrupted portable restore (throws on 3rd write), then re-run completes idempotently + const memList: ArchivedMemory[] = [ + { ...validMemory, id: "mem-1" }, + { ...validMemory, id: "mem-2" }, + { ...validMemory, id: "mem-3" }, + { ...validMemory, id: "mem-4" } + ]; + const manifest4: ArchiveManifest = { + ...validManifest, + document_count: 4 + }; + + let attempts = 0; + const writtenIds = new Map(); + const faultTarget = { + async writeMemory(memory: ArchivedMemory) { + attempts += 1; + if (attempts === 3) { + throw new Error("injected write failure on 3rd write"); + } + writtenIds.set(memory.id, memory); + } + }; + + // First attempt fails on 3rd write + await expect( + restoreMemories({ + memories: memList, + embeddings: new KeywordEmbeddingClient(), + target: faultTarget, + manifest: manifest4 + }) + ).rejects.toThrow("injected write failure on 3rd write"); + + expect(writtenIds.size).toBe(2); + + // Re-run with clean target + const secondTarget = { + async writeMemory(memory: ArchivedMemory) { + writtenIds.set(memory.id, memory); + } + }; + const result = await restoreMemories({ + memories: memList, + embeddings: new KeywordEmbeddingClient(), + target: secondTarget, + manifest: manifest4 + }); + expect(result.restored).toBe(4); + expect(Array.from(writtenIds.keys()).sort()).toEqual(["mem-1", "mem-2", "mem-3", "mem-4"]); + }); +}); diff --git a/functions/test/support/fakes.ts b/functions/test/support/fakes.ts index 249faee..b7ffdcd 100644 --- a/functions/test/support/fakes.ts +++ b/functions/test/support/fakes.ts @@ -191,6 +191,31 @@ export class InMemoryMemoryRepository implements MemoryRepository { return [...this.records]; } + seed( + records: Array<{ + id: string; + content: string; + retrieval_text: string; + embedding: number[] | { toArray: () => number[] }; + metadata: MemoryMetadata; + media?: MemoryMedia; + }> + ): void { + for (const record of records) { + const vector = Array.isArray(record.embedding) + ? [...record.embedding] + : record.embedding.toArray(); + this.records.push({ + id: record.id, + content: record.content, + retrieval_text: record.retrieval_text, + embedding: vector, + metadata: record.metadata, + media: record.media + }); + } + } + async deprecate( documentId: string, supersedingDocumentId: string | undefined, diff --git a/functions/test/support/inMemoryMirrorStore.ts b/functions/test/support/inMemoryMirrorStore.ts new file mode 100644 index 0000000..de15147 --- /dev/null +++ b/functions/test/support/inMemoryMirrorStore.ts @@ -0,0 +1,170 @@ +import { FieldValue, GeoPoint, Timestamp } from "firebase-admin/firestore"; + +import { + isBytes, + isDocumentReference, + isGeoPoint, + isVectorValue +} from "../../scripts/support/firestore-codec.js"; +import { + assertValidCollectionPath, + assertWritableProject, + type MirrorDocument, + type MirrorStore +} from "../../scripts/support/mirror-store.js"; + +export class DocumentReference { + constructor( + readonly path: string, + readonly firestore: object = {} + ) {} +} + +export function deepCopyValue(value: unknown): unknown { + if (value === null || typeof value !== "object") { + return value; + } + + if (value instanceof Timestamp) { + return new Timestamp(value.seconds, value.nanoseconds); + } + + if (isVectorValue(value)) { + return FieldValue.vector(value.toArray()); + } + + if (isGeoPoint(value)) { + return new GeoPoint(value.latitude, value.longitude); + } + + if (Buffer.isBuffer(value)) { + return Buffer.from(value); + } + + if (value instanceof Uint8Array) { + return new Uint8Array(value); + } + + if (isDocumentReference(value)) { + return new DocumentReference(value.path, value.firestore); + } + + if (value instanceof Date) { + return new Date(value.getTime()); + } + + if (Array.isArray(value)) { + return value.map(deepCopyValue); + } + + const copy: Record = {}; + for (const [key, nested] of Object.entries(value as Record)) { + copy[key] = deepCopyValue(nested); + } + return copy; +} + +export class InMemoryMirrorStore implements MirrorStore { + private readonly collections = new Map>>(); + // Fault injection: number of writeDocuments calls allowed before throwing. + // One call = one batch/collection chunk. + private writesRemainingBeforeFailure: number | null = null; + + constructor(readonly projectId: string) {} + + failAfterWrites(n: number): void { + this.writesRemainingBeforeFailure = n; + } + + clearFaults(): void { + this.writesRemainingBeforeFailure = null; + } + + ensurePath(path: string): void { + assertValidCollectionPath(path); + const segments = path.split("/"); + for (let i = 1; i <= segments.length; i += 2) { + const subPath = segments.slice(0, i).join("/"); + if (!this.collections.has(subPath)) { + this.collections.set(subPath, new Map()); + } + } + } + + async discoverCollectionPaths(): Promise { + return Array.from(this.collections.keys()).sort((left, right) => + left.localeCompare(right) + ); + } + + async readCollection(path: string): Promise { + assertValidCollectionPath(path); + const col = this.collections.get(path); + if (!col) { + return []; + } + const docs: MirrorDocument[] = []; + for (const [id, data] of col.entries()) { + docs.push({ + id, + data: deepCopyValue(data) as Record + }); + } + docs.sort((left, right) => left.id.localeCompare(right.id)); + return docs; + } + + async countDocuments(path: string): Promise { + assertValidCollectionPath(path); + return this.collections.get(path)?.size ?? 0; + } + + async listDocumentIds(path: string): Promise { + assertValidCollectionPath(path); + const col = this.collections.get(path); + if (!col) { + return []; + } + return Array.from(col.keys()).sort((left, right) => left.localeCompare(right)); + } + + async writeDocuments(path: string, docs: MirrorDocument[]): Promise { + assertValidCollectionPath(path); + assertWritableProject(this.projectId); + + // One call = one batch/collection chunk. + if (this.writesRemainingBeforeFailure !== null) { + if (this.writesRemainingBeforeFailure <= 0) { + throw new Error("injected failure"); + } + this.writesRemainingBeforeFailure -= 1; + } + + this.ensurePath(path); + const col = this.collections.get(path)!; + for (const doc of docs) { + col.set(doc.id, deepCopyValue(doc.data) as Record); + } + return docs.length; + } + + async deleteDocuments(path: string, ids: string[]): Promise { + assertValidCollectionPath(path); + assertWritableProject(this.projectId); + const col = this.collections.get(path); + if (!col) { + return 0; + } + let deleted = 0; + for (const id of ids) { + if (col.delete(id)) { + deleted += 1; + } + } + return deleted; + } + + referenceFor(path: string): unknown { + return new DocumentReference(path); + } +} diff --git a/functions/test/support/recoveryCorpus.ts b/functions/test/support/recoveryCorpus.ts new file mode 100644 index 0000000..50cd363 --- /dev/null +++ b/functions/test/support/recoveryCorpus.ts @@ -0,0 +1,158 @@ +import { FieldValue, GeoPoint, Timestamp, type Firestore } from "firebase-admin/firestore"; + +import type { InMemoryMirrorStore } from "./inMemoryMirrorStore.js"; + +export function getCorpusData(refResolver: (path: string) => unknown) { + const memoryDoc1 = { + content: "Ktor networking is preferred for Kotlin multiplatform clients.", + retrieval_text: "ktor networking", + embedding: FieldValue.vector([1, 0, 0, 0, 0, 0]), + metadata: { + module_name: "android", + branch_state: "active", + created_at: 1_700_000_000_000, + updated_at: 1_700_000_000_000, + created_time: new Timestamp(1_700_000_000, 0), + modality: "text" + } + }; + + const memoryDoc2 = { + content: "Legacy networking using okhttp directly on android.", + retrieval_text: "ktor networking", + embedding: FieldValue.vector([1, 0, 0, 0, 0, 0]), + metadata: { + module_name: "android", + branch_state: "deprecated", + superseded_by: "doc-1", + supersession_reason: "changed", + created_at: 1_690_000_000_000, + updated_at: 1_700_000_000_000, + created_time: new Timestamp(1_690_000_000, 0), + modality: "text" + } + }; + + const fingerprintDoc = { + fingerprint: "hash-12345", + expires_at: new Timestamp(1_700_001_000, 0), + created_at: 1_700_000_000_000 + }; + + const futureDoc = { + geo: new GeoPoint(37.7749, -122.4194), + bytes: Buffer.from("binary-payload-data", "utf8"), + ref: refResolver("future_collection/doc-unknown"), + not_a_number: Number.NaN, + pos_inf: Number.POSITIVE_INFINITY, + neg_inf: Number.NEGATIVE_INFINITY, + nested_map: { + alpha: "hello", + beta: 42, + inner: { deep: true } + }, + nested_array: [1, 2, "three", { x: 10 }] + }; + + const noteDoc = { + note_title: "Note about Ktor", + created_time: new Timestamp(1_700_000_100, 0) + }; + + const ghostChildDoc = { + child_name: "Casper", + is_ghost_child: true + }; + + return { + memoryDoc1, + memoryDoc2, + fingerprintDoc, + futureDoc, + noteDoc, + ghostChildDoc + }; +} + +export async function seedFirestoreCorpus( + firestore: Firestore, + prefix = "" +): Promise<{ + memoryVectorsPath: string; + fingerprintsPath: string; + futurePath: string; + notesPath: string; + ghostChildrenPath: string; +}> { + const p = (col: string) => (prefix ? `${prefix}${col}` : col); + const memoryVectorsPath = p("memory_vectors"); + const fingerprintsPath = p("memory_vectors_write_fingerprints"); + const futurePath = p("future_collection"); + const notesPath = `${memoryVectorsPath}/doc-1/notes`; + const ghostChildrenPath = `${p("ghost_parent")}/ghost-doc-1/children`; + + const data = getCorpusData(docPath => firestore.doc(docPath)); + + await firestore.collection(memoryVectorsPath).doc("doc-1").set(data.memoryDoc1); + await firestore.collection(memoryVectorsPath).doc("doc-2").set(data.memoryDoc2); + await firestore.collection(fingerprintsPath).doc("fp-1").set(data.fingerprintDoc); + await firestore.collection(futurePath).doc("doc-unknown").set(data.futureDoc); + await firestore.collection(notesPath).doc("note-1").set(data.noteDoc); + await firestore.collection(ghostChildrenPath).doc("child-1").set(data.ghostChildDoc); + + return { + memoryVectorsPath, + fingerprintsPath, + futurePath, + notesPath, + ghostChildrenPath + }; +} + +export async function seedInMemoryCorpus( + store: InMemoryMirrorStore, + prefix = "" +): Promise<{ + memoryVectorsPath: string; + fingerprintsPath: string; + futurePath: string; + notesPath: string; + ghostChildrenPath: string; +}> { + const p = (col: string) => (prefix ? `${prefix}${col}` : col); + const memoryVectorsPath = p("memory_vectors"); + const fingerprintsPath = p("memory_vectors_write_fingerprints"); + const futurePath = p("future_collection"); + const notesPath = `${memoryVectorsPath}/doc-1/notes`; + const ghostParentPath = p("ghost_parent"); + const ghostChildrenPath = `${ghostParentPath}/ghost-doc-1/children`; + + const data = getCorpusData(docPath => store.referenceFor(docPath)); + + await store.writeDocuments(memoryVectorsPath, [ + { id: "doc-1", data: data.memoryDoc1 }, + { id: "doc-2", data: data.memoryDoc2 } + ]); + await store.writeDocuments(fingerprintsPath, [ + { id: "fp-1", data: data.fingerprintDoc } + ]); + await store.writeDocuments(futurePath, [ + { id: "doc-unknown", data: data.futureDoc } + ]); + await store.writeDocuments(notesPath, [ + { id: "note-1", data: data.noteDoc } + ]); + // Note: ghost-doc-1 is never created in ghost_parent + store.ensurePath(ghostParentPath); + await store.writeDocuments(ghostChildrenPath, [ + { id: "child-1", data: data.ghostChildDoc } + ]); + + return { + memoryVectorsPath, + fingerprintsPath, + futurePath, + notesPath, + ghostChildrenPath + }; +} diff --git a/metacortexplan.md b/metacortexplan.md index 8d302ac..a2e76cb 100644 --- a/metacortexplan.md +++ b/metacortexplan.md @@ -321,3 +321,15 @@ Remaining limitations: ### 2026-10-10 UTC CORE #70 acceptance record PR #14 merged 2026-10-05 and landed six lifecycle correctness fixes in saving, retiring, consolidating, searching and listing (fix commit `b8b09aa57c77307a6e7610793774b50501119a8a`, merge commit `4e2d3371269135436f04d934837235bdc5f28baf`) with 22 new regression tests; on the combined tree the typecheck, 139 tests with coverage and the 70-file clean Journey install passed, and 11 of 11 checks passed against the Firestore emulator; production serves revision `metacortexmcp-00031-dig`, built from that merge commit; card #70 was closed as done on 2026-10-10. Remaining limitations, stated plainly: no check against production after the deploy is recorded, and dependency card #69 (VERIFY) is still open. + +### 2026-10-29 UTC RECOVERY #71 local verification record + +- Branch: `agent/card-71-recovery-fidelity` +- Final commit sha: `50a3907` +- Local checks: + - `metacortex_typecheck`: clean TypeScript build (`tsc -p tsconfig.json`) and clean operator scripts typecheck (`tsc -p tsconfig.scripts.json`). + - `metacortex_test`: 175 passed tests across 15 test files (1 test file skipped when emulator host is not set), 79.2% statement coverage. + - `metacortex_package`: verified clean Journey installation of 70 files with 123 tests passing. + - Emulator run: `firebase emulators:exec --only firestore --project demo-recovery "npm --prefix functions test -- test/recovery.emulator.test.ts"` passed (1 test exercising recursive discovery including phantom parents, backup, restore, real Firestore types, content corruption detection, and vector search without re-embedding). +- Deployed or release evidence: none. No production restore was performed or authorized; nothing was deployed. +- Remaining limitations: Firestore integer vs double distinction is not preserved by the Admin SDK read path; DocumentReference values are restored with their original absolute paths and are not remapped by --target-prefix; field transforms, indexes, rules, TTL policies and security rules are not part of the archive; the verification covers documents, not Storage objects or auth users; restore is not atomic across collections (resume by re-running); live production restore was not exercised.