diff --git a/docs/fly-guest-transport.md b/docs/fly-guest-transport.md index 906470a7..d2910d50 100644 --- a/docs/fly-guest-transport.md +++ b/docs/fly-guest-transport.md @@ -4,4 +4,57 @@ The execution host must have a supported `fly` CLI installed and access to the selected Fly app and Machine. The token must authorize Machine execution and SFTP access. Machines must be running. The guest image must support Fly's Machine exec and SFTP paths; images do not need exe.dev's SSH tooling or systemd. `run()` accepts a nonempty executable and argv plus optional stdin bytes, returns captured stdout/stderr and the process exit code, and enforces a deadline of at most ten minutes and a combined output cap of at most 16 MiB. File operations enforce a configurable transfer cap of at most 64 MiB and use normalized absolute guest paths. Timed out operations terminate the CLI process group. +Both execution backends install and invoke the same small `factorize-job` guest launcher as `exedev`. It requires Python 3, the `exedev` account, and a writable `/home/exedev/.factorize-job` directory. It accepts a bounded JSON launch payload over stdin, starts the code script or agent argv without a shell command assembled from the payload, writes structured process state, preserves stdout/stderr and the run result, and exposes `factorize-job status` for Fly scheduler polling. Fly images also need `runuser`, `/bin/sh`, `/usr/local/bin`, Node.js and OpenSSL for portable proxy access, and the provider's documented Machine exec/SFTP support. Codex images must provide the tested `codex-cli 0.167.1`; Fly checks this as the `exedev` user before launch and returns a bounded failure for a missing or unsupported version. Code jobs do not require Codex. Image verification checks Python 3 for exe.dev execution; Fly reports a bounded launch failure when guest prerequisites are absent. The helper and job Machine are not configured to restart and replay a failed process. Custom entrypoints can finish their own setup before the backend invokes the launcher. + The local protocol fixtures verify process result handling, timeout, output overflow, disconnect, safe argv construction, and file byte transfer. They do not validate live Fly authentication, Fly CLI compatibility, routing to a real Machine, or the guest's exec/SFTP capabilities. A separate opt-in validation requires a Fly account, an authorized app/Machine and token, an installed Fly CLI, and a custom image with Fly exec/SFTP support; no Fly network call is part of the required test suite. + +For portable integrations, the scheduler sends the scoped run credential over Fly's authenticated Machine exec channel directly to the dedicated credential proxy's private Unix socket. The proxy keeps it in process memory, bound to that run and provider VM identity, and removes it when durable revocation arrives. The credential is not part of Machine configuration, launch files, or the job VM payload. During guest launch, the VM generates its own private key and CSR; the proxy signs the CSR for the provider VM identity, and only the certificate and trust bundle return to the scheduler for installation. An internal-only Fly service exposes the proxy tunnel listener on port 9090. The guest starts a loopback CONNECT forwarder and sets HTTP(S)_PROXY for the job process. Each tunnel connection presents only its run ID over mTLS; the proxy retrieves its in-memory credential to perform fresh control-plane enrollment. + +## Credential-proxy package and local evidence + +`packages/credential-proxy` is a private npm workspace of native ES modules and +uses Node.js built-ins at runtime. Its package entry points are declared in +`package.json`; only `src/` is included in the package artifact, excluding the +test suite's synthetic TLS keys and fixtures. The Fly VM tunnel helper is generated from +`packages/api/src/fly-job-tunnel-helper.ts`. After `npm ci`, inspect the local +package contents without publishing with: + +```sh +npm pack --dry-run --workspace=@factorize/credential-proxy --json +npm run test:proxy +``` + +The provisioner accepts only an image pinned by digest and equal to the reviewed +image supplied by deployment configuration. Proxy control credentials enter +through the private bootstrap socket; run credentials enter through Fly's +authenticated Machine exec channel and remain in proxy memory. Neither user +unlock keys nor integration credentials belong in image build arguments, image +configuration, guest environment variables, or launch payloads. + +The minimal credential-proxy OCI recipe is `packages/credential-proxy/Dockerfile`. +It pins the Node 24.12.0 multi-platform manifest digest, installs the +authentication-only Codex CLI 0.167.1 from a checked-in lockfile, copies only +proxy runtime source, and runs as the unprivileged `node` user. Build locally +without publishing using `scripts/build-credential-proxy-image.sh`; Docker +Buildx emits an OCI archive with max-mode provenance and an SBOM under +`~/.factorize-job/credential-proxy-image/`, alongside a compact JSON summary +binding the archive digest to the source commit, base digest, and Codex version. +The OCI archive and generated provenance are inspectable/rebuildable artifacts; +this repository deliberately has no publishing workflow, and this local recipe +does not establish that any production digest has been published or approved. +The provisioner's reviewed-digest configuration must still be updated through +the separate release process after image review. Live Fly networking also +remains unverified. + +Local coverage is split across fixtures to avoid repeating full application +startup: `packages/credential-proxy/test/fly-guest-helper.e2e.test.mjs` executes +the generated guest helper against an mTLS proxy and local HTTPS upstream and +rejects a mismatched VM binding; `proxy-listener.e2e.test.mjs` checks per-VM +reflection, locked access, configured-origin forwarding, redirect rejection, +interrupted writes, and reconnect after lock/unlock. These synthetic fixtures do +not establish live provider or production behavior. The combined +`locked scoped run reconnects with its current grant while another VM stays denied` +fixture exercises the production grant resolver with a synthetic Factorize run +snapshot and local TLS upstream. The repository runbook +`packages/docs/self-hosting/credential-proxy-security.mdx` documents key handling, lifecycle recovery, +accepted provider/control-plane risks, and the independent Fly shutdown steps. diff --git a/package.json b/package.json index a2785aab..7b620b8a 100644 --- a/package.json +++ b/package.json @@ -43,5 +43,15 @@ "engines": { "node": ">=24.12.0", "npm": ">=11.19.0" + }, + "allowScripts": { + "esbuild@0.25.4": true, + "esbuild@0.18.20": true, + "esbuild@0.25.12": true, + "esbuild@0.28.2": true, + "esbuild@0.28.1": true, + "sharp@0.33.5": true, + "workerd@1.20251118.0": true, + "workerd@1.20260910.1": true } } diff --git a/packages/api-client/src/schema.d.ts b/packages/api-client/src/schema.d.ts index a14e7359..996739ce 100644 --- a/packages/api-client/src/schema.d.ts +++ b/packages/api-client/src/schema.d.ts @@ -1,5 +1,85 @@ // Generated from packages/docs/openapi.yaml. Do not edit. export interface paths { + "/api/v1/model-accounts": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * List model accounts available to the owner + * @description Requires an interactive owner browser session. Returns account metadata and the collection revision only; encrypted account caches and credentials are never returned. + * + * Requires the integrations:read scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + get: operations["get_api_v1_model_accounts"]; + put?: never; + /** + * Create a model account + * @description Requires an interactive owner browser session and the current collection revision. Accepts only user-encrypted initial account state; credentials are established directly with the credential proxy. + * + * Requires the integrations:write scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + post: operations["post_api_v1_model_accounts"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v1/model-accounts/{accountId}": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Delete a model account + * @description Requires an interactive owner browser session and the current collection revision. Deletion is recorded as a tombstone so future snapshots and unlock reconciliation cannot restore the account. + * + * Requires the integrations:write scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + delete: operations["delete_api_v1_model_accounts_accountId"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v1/model-accounts/{accountId}/browser-capability": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Issue a short-lived browser capability for a paired model account + * @description Requires an interactive owner browser session. Returns a single-use, 60-second capability scoped to the account, action, proxy boot and exact application origin. The browser sends it directly to the paired proxy; Factorize never receives model-account tokens. + * + * Requires the integrations:write scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + post: operations["post_api_v1_model_accounts_accountId_browser_capability"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/unlock-key/verification": { parameters: { query?: never; @@ -32,6 +112,54 @@ export interface paths { patch?: never; trace?: never; }; + "/api/v1/unlock-key/rotation-snapshot": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get encrypted state for owner key rotation + * @description Owner browser session only. Returns encrypted records, tombstones, and pending change history for local re-encryption; plaintext credentials and unlock keys are never returned. + * + * Requires the integrations:read scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + get: operations["get_api_v1_unlock_key_rotation_snapshot"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v1/unlock-key/rotation": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Atomically replace the unlock-key generation + * @description Owner browser session only. Commits a revision-checked encrypted generation and verification record, revokes grants, consumes enrollments, and locks the proxy. Fly connection and execution configuration are preserved. + * + * Requires the integrations:write scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + post: operations["post_api_v1_unlock_key_rotation"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/integrations/bearer": { parameters: { query?: never; @@ -1988,6 +2116,30 @@ export interface paths { patch?: never; trace?: never; }; + "/api/v1/runs/{runId}/retry-launch": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Manually retry a blocked launch + * @description Retries only a pre-launch proxy-gated run after its prerequisites have been resolved. The run remains blocked until an interactive owner explicitly retries it; unlock or status changes do not automatically launch it. + * + * Requires the runs:cancel scope. + * + * An interactive owner session is required; bearer tokens cannot call this operation. + */ + post: operations["post_api_v1_runs_runId_retry_launch"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/runs/{runId}/kill": { parameters: { query?: never; @@ -2683,6 +2835,176 @@ export interface components { } export type $defs = Record; export interface operations { + get_api_v1_model_accounts: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Model account metadata and collection revision */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + revision: number; + accounts: { + /** Format: uuid */ + id: string; + name: string; + description: string | null; + revision: number; + }[]; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + 409: components["responses"]["Error409"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; + post_api_v1_model_accounts: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody: { + content: { + "application/json": { + /** Format: uuid */ + id: string; + expectedRevision: number; + /** Format: uuid */ + idempotencyKey: string; + name: string; + description: string | null; + encryptedPayload: string; + }; + }; + }; + responses: { + /** @description Created model account */ + 201: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + revision: number; + duplicate: boolean; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + 409: components["responses"]["Error409"]; + 415: components["responses"]["Error415"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; + delete_api_v1_model_accounts_accountId: { + parameters: { + query?: never; + header?: never; + path: { + accountId: string; + }; + cookie?: never; + }; + requestBody: { + content: { + "application/json": { + expectedRevision: number; + /** Format: uuid */ + idempotencyKey: string; + }; + }; + }; + responses: { + /** @description Deleted model account */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + /** Format: uuid */ + id: string; + /** @constant */ + deleted: true; + revision: number; + duplicate: boolean; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + 409: components["responses"]["Error409"]; + 415: components["responses"]["Error415"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; + post_api_v1_model_accounts_accountId_browser_capability: { + parameters: { + query?: never; + header?: never; + path: { + accountId: string; + }; + cookie?: never; + }; + requestBody: { + content: { + "application/json": { + /** @enum {string} */ + action: "authorize" | "status" | "cancel"; + }; + }; + }; + responses: { + /** @description Paired proxy endpoint and one-use capability */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + /** Format: uri */ + endpoint: string; + capability: string; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + 409: components["responses"]["Error409"]; + 415: components["responses"]["Error415"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; get_api_v1_unlock_key_verification: { parameters: { query?: never; @@ -2758,6 +3080,140 @@ export interface operations { 503: components["responses"]["Error503"]; }; }; + get_api_v1_unlock_key_rotation_snapshot: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Encrypted rekey snapshot */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + ownerId: string; + /** Format: uuid */ + generationId: string; + revision: number; + records: { + /** Format: uuid */ + id: string; + /** @enum {string} */ + kind: "bearer" | "model"; + name: string; + origin: string | null; + description: string | null; + revision: number; + encryptedPayload: string | null; + deletedAt: string | null; + }[]; + changes: { + /** Format: uuid */ + id: string; + /** @enum {string} */ + kind: "bearer" | "model"; + name: string; + origin: string | null; + description: string | null; + revision: number; + /** Format: uuid */ + idempotencyKey: string; + /** @enum {string} */ + operation: "upsert" | "delete"; + encryptedPayload: string | null; + recordRevision?: number | null; + }[]; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + 409: components["responses"]["Error409"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; + post_api_v1_unlock_key_rotation: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody: { + content: { + "application/json": { + /** Format: uuid */ + expectedGenerationId: string; + /** Format: uuid */ + generationId: string; + expectedRevision: number; + encryptedVerification: string; + records: { + /** Format: uuid */ + id: string; + /** @enum {string} */ + kind: "bearer" | "model"; + name: string; + origin: string | null; + description: string | null; + revision: number; + encryptedPayload: string | null; + deletedAt: string | null; + }[]; + changes: { + /** Format: uuid */ + id: string; + /** @enum {string} */ + kind: "bearer" | "model"; + name: string; + origin: string | null; + description: string | null; + revision: number; + /** Format: uuid */ + idempotencyKey: string; + /** @enum {string} */ + operation: "upsert" | "delete"; + encryptedPayload: string | null; + recordRevision?: number | null; + }[]; + }; + }; + }; + responses: { + /** @description Key generation replaced */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + /** Format: uuid */ + generationId: string; + revision: number; + duplicate: boolean; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + 409: components["responses"]["Error409"]; + 415: components["responses"]["Error415"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; get_api_v1_integrations_bearer: { parameters: { query?: never; @@ -7956,6 +8412,43 @@ export interface operations { 503: components["responses"]["Error503"]; }; }; + post_api_v1_runs_runId_retry_launch: { + parameters: { + query?: never; + header?: never; + path: { + runId: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Manual retry queued */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": { + /** Format: uuid */ + runId: string; + /** @constant */ + retryRequested: true; + }; + }; + }; + 400: components["responses"]["Error400"]; + 401: components["responses"]["Error401"]; + /** @description Interactive owner session required */ + 403: components["responses"]["Error403"]; + 404: components["responses"]["Error404"]; + /** @description Run is not waiting for a manual launch retry */ + 409: components["responses"]["Error409"]; + 429: components["responses"]["Error429"]; + 500: components["responses"]["Error500"]; + 503: components["responses"]["Error503"]; + }; + }; post_api_v1_runs_runId_kill: { parameters: { query?: never; diff --git a/packages/api/migrations/0036_proxy_control_credentials.sql b/packages/api/migrations/0036_proxy_control_credentials.sql new file mode 100644 index 00000000..6c7d8b38 --- /dev/null +++ b/packages/api/migrations/0036_proxy_control_credentials.sql @@ -0,0 +1,10 @@ +CREATE TABLE app.credential_proxy_control_credentials ( + tenant_id uuid NOT NULL REFERENCES app.tenants(id) ON DELETE CASCADE, + proxy_identity text NOT NULL, + boot_epoch uuid NOT NULL, + token_digest bytea NOT NULL UNIQUE, + revoked_at timestamptz, + consumed_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (tenant_id, proxy_identity) +); diff --git a/packages/api/migrations/0037_proxy_control_credentials.sql b/packages/api/migrations/0037_proxy_control_credentials.sql new file mode 100644 index 00000000..b3c69e79 --- /dev/null +++ b/packages/api/migrations/0037_proxy_control_credentials.sql @@ -0,0 +1,2 @@ +ALTER TABLE app.credential_proxy_control_credentials + ADD COLUMN proxy_token_digest bytea UNIQUE; diff --git a/packages/api/migrations/0038_proxy_browser_control_pairings.sql b/packages/api/migrations/0038_proxy_browser_control_pairings.sql new file mode 100644 index 00000000..45a8eb2e --- /dev/null +++ b/packages/api/migrations/0038_proxy_browser_control_pairings.sql @@ -0,0 +1,8 @@ +CREATE TABLE app.credential_proxy_browser_pairings ( + tenant_id uuid PRIMARY KEY REFERENCES app.tenants(id) ON DELETE CASCADE, + proxy_identity text NOT NULL, + boot_epoch uuid NOT NULL, + browser_origin text NOT NULL, + updated_at timestamptz NOT NULL DEFAULT now(), + CHECK (browser_origin ~ '^https://[^/]+$') +); diff --git a/packages/api/migrations/0047_proxy_launch_status.sql b/packages/api/migrations/0047_proxy_launch_status.sql new file mode 100644 index 00000000..d7bf1f7a --- /dev/null +++ b/packages/api/migrations/0047_proxy_launch_status.sql @@ -0,0 +1,2 @@ +ALTER TABLE app.credential_proxy_states + ADD COLUMN unlocked boolean NOT NULL DEFAULT false; diff --git a/packages/api/src/api-contract.ts b/packages/api/src/api-contract.ts index db819a4c..29e97c7c 100644 --- a/packages/api/src/api-contract.ts +++ b/packages/api/src/api-contract.ts @@ -131,10 +131,23 @@ const bearerIntegrationCreate = z.object({ const bearerIntegrationUpdate = bearerIntegrationCreate.omit({ id: true }); const bearerIntegrationDelete = z.object({ expectedRevision: z.number().int().nonnegative(), idempotencyKey: z.uuid() }).strict(); const bearerIntegrationList = z.object({ revision: z.number().int().nonnegative(), integrations: z.array(bearerIntegrationMetadata) }); +const modelAccountMetadata = z.object({ id: z.uuid(), name: z.string(), description: z.string().nullable(), revision: z.number().int().nonnegative() }); +const modelAccountCreate = z.object({ + id: z.uuid(), expectedRevision: z.number().int().nonnegative(), idempotencyKey: z.uuid(), + name: z.string().trim().min(1).max(120), description: z.string().max(500).nullable(), + encryptedPayload: z.string().min(1).max(65_536), +}).strict(); +const modelAccountDelete = z.object({ expectedRevision: z.number().int().nonnegative(), idempotencyKey: z.uuid() }).strict(); +const modelAccountList = z.object({ revision: z.number().int().nonnegative(), accounts: z.array(modelAccountMetadata) }); const bearerIntegrationWriteResult = z.object({ revision: z.number().int().nonnegative(), duplicate: z.boolean() }); const bearerIntegrationDeleteResult = z.object({ id: z.uuid(), deleted: z.literal(true), revision: z.number().int().nonnegative(), duplicate: z.boolean() }); const unlockKeyVerification = z.object({ ownerId: z.string(), verification: z.object({ generationId: z.uuid(), encryptedVerification: z.string() }).strict().nullable() }).strict(); const unlockKeyVerificationInput = z.object({ generationId: z.uuid(), encryptedVerification: z.string().min(1).max(4096) }).strict(); +const rotationRecord = z.object({ id: z.uuid(), kind: z.enum(["bearer", "model"]), name: z.string().min(1).max(120), origin: z.string().nullable(), description: z.string().max(500).nullable(), revision: z.number().int().nonnegative(), encryptedPayload: z.string().max(65_536).nullable(), deletedAt: z.string().datetime({ offset: true }).nullable() }).strict(); +const rotationChange = z.object({ id: z.uuid(), kind: z.enum(["bearer", "model"]), name: z.string().min(1).max(120), origin: z.string().nullable(), description: z.string().max(500).nullable(), revision: z.number().int().positive(), idempotencyKey: z.uuid(), operation: z.enum(["upsert", "delete"]), encryptedPayload: z.string().max(65_536).nullable(), recordRevision: z.number().int().positive().nullable().optional() }).strict(); +const unlockKeyRotationSnapshot = z.object({ ownerId: z.string(), generationId: z.uuid(), revision: z.number().int().nonnegative(), records: z.array(rotationRecord).max(5000), changes: z.array(rotationChange).max(10000) }).strict(); +const unlockKeyGenerationReplacement = z.object({ expectedGenerationId: z.uuid(), generationId: z.uuid(), expectedRevision: z.number().int().nonnegative(), encryptedVerification: z.string().min(1).max(4096), records: z.array(rotationRecord).max(5000), changes: z.array(rotationChange).max(10000) }).strict(); +const modelAccountBrowserCapabilityInput = z.object({ action: z.enum(["authorize", "status", "cancel"]) }).strict(); function route( method: string, @@ -213,8 +226,34 @@ function authRoute(method: string, path: string, action: AuthAction, summary: st } export const API_ROUTES: RouteContract[] = [ + route("GET", "/api/v1/model-accounts", "integrations:read", { + summary: "List model accounts available to the owner", + description: "Requires an interactive owner browser session. Returns account metadata and the collection revision only; encrypted account caches and credentials are never returned.", + responses: { "200": { description: "Model account metadata and collection revision", content: { "application/json": { schema: jsonSchema(modelAccountList, "output") } } } }, + }, service => service.listModelAccounts(), { ownerSession: true }), + route("POST", "/api/v1/model-accounts", "integrations:write", { + summary: "Create a model account", + description: "Requires an interactive owner browser session and the current collection revision. Accepts only user-encrypted initial account state; credentials are established directly with the credential proxy.", + responses: { "201": { description: "Created model account", content: { "application/json": { schema: jsonSchema(bearerIntegrationWriteResult, "output") } } } }, + }, (service, { body }) => service.createModelAccount(body), { body: modelAccountCreate, maxBodyBytes: 70_000, ownerSession: true }), + route("DELETE", "/api/v1/model-accounts/{accountId}", "integrations:write", { + summary: "Delete a model account", + description: "Requires an interactive owner browser session and the current collection revision. Deletion is recorded as a tombstone so future snapshots and unlock reconciliation cannot restore the account.", + responses: { "200": { description: "Deleted model account", content: { "application/json": { schema: jsonSchema(bearerIntegrationDeleteResult, "output") } } } }, + }, (service, { params, body }) => service.deleteModelAccount(params.accountId, body), { body: modelAccountDelete, maxBodyBytes: 4096, ownerSession: true, parameters: z.object({ accountId: z.uuid() }) }), + route("POST", "/api/v1/model-accounts/{accountId}/browser-capability", "integrations:write", { + summary: "Issue a short-lived browser capability for a paired model account", + description: "Requires an interactive owner browser session. Returns a single-use, 60-second capability scoped to the account, action, proxy boot and exact application origin. The browser sends it directly to the paired proxy; Factorize never receives model-account tokens.", + responses: { "200": { description: "Paired proxy endpoint and one-use capability", content: { "application/json": { schema: jsonSchema(z.object({ endpoint: z.url(), capability: z.string() }), "output") } } } }, + }, (service, { params, body }) => service.issueModelAccountBrowserCapability(params.accountId, body.action), { body: modelAccountBrowserCapabilityInput, maxBodyBytes: 1024, ownerSession: true, parameters: z.object({ accountId: z.uuid() }) }), route("GET", "/api/v1/unlock-key/verification", "integrations:read", { summary: "Get the current encrypted unlock-key verification record", description: "Requires an interactive owner browser session. Returns user-encrypted verification ciphertext and its key generation; the unlock key is never accepted or returned.", responses: { "200": { description: "Encrypted verification record or empty setup state", content: { "application/json": { schema: jsonSchema(unlockKeyVerification, "output") } } } } }, service => service.getUnlockKeyVerification(), { ownerSession: true }), route("POST", "/api/v1/unlock-key/verification", "integrations:write", { summary: "Create the encrypted unlock-key verification record", description: "Requires an interactive owner browser session. Stores only browser-encrypted verification ciphertext and permits one setup per account.", responses: { "200": { description: "Created verification record", content: { "application/json": { schema: jsonSchema(z.object({ generationId: z.uuid() }), "output") } } } } }, (service, { body }) => service.createUnlockKeyVerification(body), { body: unlockKeyVerificationInput, maxBodyBytes: 8192, ownerSession: true }), + route("GET", "/api/v1/unlock-key/rotation-snapshot", "integrations:read", { summary: "Get encrypted state for owner key rotation", description: "Owner browser session only. Returns encrypted records, tombstones, and pending change history for local re-encryption; plaintext credentials and unlock keys are never returned.", responses: { "200": { description: "Encrypted rekey snapshot", content: { "application/json": { schema: jsonSchema(unlockKeyRotationSnapshot, "output") } } } } }, service => service.getUnlockKeyRotationSnapshot(), { ownerSession: true }), + route("POST", "/api/v1/unlock-key/rotation", "integrations:write", { + summary: "Atomically replace the unlock-key generation", + description: "Owner browser session only. Commits a revision-checked encrypted generation and verification record, revokes grants, consumes enrollments, and locks the proxy. Fly connection and execution configuration are preserved.", + responses: { "200": { description: "Key generation replaced", content: { "application/json": { schema: jsonSchema(z.object({ generationId: z.uuid(), revision: z.number().int().nonnegative(), duplicate: z.boolean() }), "output") } } } }, + }, (service, { body }) => service.replaceUnlockKeyGeneration(body), { body: unlockKeyGenerationReplacement, maxBodyBytes: 12_000_000, ownerSession: true }), route("GET", "/api/v1/integrations/bearer", "flows:read", { summary: "List encrypted bearer integrations", description: "Requires an interactive owner browser session. Returns safe metadata and user-encrypted ciphertext; credentials are never decrypted or revealed.", @@ -1098,6 +1137,15 @@ export const API_ROUTES: RouteContract[] = [ { summary: "Stop run", responses: { "200": { description: "Stopped" } } }, (service, { params }) => service.stopRun(params.runId), ), + route("POST", "/api/v1/runs/{runId}/retry-launch", "runs:cancel", { + summary: "Manually retry a blocked launch", + description: "Retries only a pre-launch proxy-gated run after its prerequisites have been resolved. The run remains blocked until an interactive owner explicitly retries it; unlock or status changes do not automatically launch it.", + responses: { + "200": { description: "Manual retry queued", content: { "application/json": { schema: { type: "object", required: ["runId", "retryRequested"], properties: { runId: { type: "string", format: "uuid" }, retryRequested: { type: "boolean", const: true } }, additionalProperties: false } } } }, + "403": { description: "Interactive owner session required" }, + "409": { description: "Run is not waiting for a manual launch retry" }, + }, + }, (service, { params }) => service.retryBlockedLaunch(params.runId), { ownerSession: true, parameters: z.object({ runId: z.uuid() }) }), route( "POST", "/api/v1/runs/{runId}/kill", diff --git a/packages/api/src/exe-vm-backend.ts b/packages/api/src/exe-vm-backend.ts index caeee884..98805a84 100644 --- a/packages/api/src/exe-vm-backend.ts +++ b/packages/api/src/exe-vm-backend.ts @@ -7,6 +7,7 @@ import { shellAtom, type ExeConnection, type ExeRunConnection } from "./exe"; import { HARNESS_LOG_LIMIT, safeDiagnosticText } from "./harness-diagnostics"; import { ExeRunProxy } from "./exe-run-proxy"; import { runProxyName } from "./run-credentials"; +import { FACTORIZE_JOB_LAUNCHER } from "./factorize-job-launcher"; const API = "https://exe.dev/exec"; const capabilities = ["output", "recovery", "stop"] as const; @@ -20,14 +21,6 @@ export function isOwnedVmName(vm: string): boolean { return /^factorize-[a-z0-9](?:[a-z0-9-]{0,47}[a-z0-9])?$/.test(vm); } -function harnessCommand(harness: NonNullable): string { - const environment = Object.entries(harness.env).map(([name, value]) => { - if (!/^[A-Z_][A-Z0-9_]*$/.test(name)) throw new Error("Invalid harness environment variable name"); - return shellAtom(`--setenv=${name}=${value}`); - }); - return [...environment, harness.executable, ...harness.args].map((value, index) => index < environment.length ? value : shellAtom(value)).join(" "); -} - export class ExeIntegrationError extends Error {} export class ExeImageVerificationError extends ExeIntegrationError { constructor(public kind: "image_pull" | "permission" | "capacity" | "timeout", message: string) { super(message); } @@ -142,9 +135,9 @@ export class ExeVmBackend implements ExecutionBackend { for (let attempt = 1; (!ssh || !ssh.ok) && attempt < 6; attempt++) ssh = await this.api(`ssh ${shellAtom(vm)} ${shellAtom("true")}`).catch(() => null); if (!ssh?.ok) throw new ExeIntegrationError(`ssh: VM was created from the image, but SSH access failed (${ssh?.status ?? "timeout"}). Ensure the image boots an exe.dev-compatible system with SSH enabled.`); await progress("checking_agent"); - const check = executionType === "code" ? "command -v bash >/dev/null 2>&1 && command -v curl >/dev/null 2>&1" : `command -v ${shellAtom(agentKind)} >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 && command -v jq >/dev/null 2>&1`; + const check = executionType === "code" ? "command -v bash >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1 && command -v curl >/dev/null 2>&1" : `command -v ${shellAtom(agentKind)} >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 && command -v jq >/dev/null 2>&1`; const agent = await this.api(`ssh ${shellAtom(vm)} ${shellAtom(check)}`).catch(() => null); - if (!agent?.ok) throw new ExeIntegrationError(executionType === "code" ? "agent: SSH works, but bash and curl prerequisites are missing. Use an exe.dev-compatible image with bash and curl installed." : `agent: SSH works, but the selected ${agentKind} harness prerequisites are missing. Use an image with ${agentKind}, curl, and jq installed.`); + if (!agent?.ok) throw new ExeIntegrationError(executionType === "code" ? "agent: SSH works, but bash, python3, or curl prerequisites are missing. Use an exe.dev-compatible image with bash, python3, and curl installed." : `agent: SSH works, but the selected ${agentKind} harness prerequisites are missing. Use an image with ${agentKind}, python3, curl, and jq installed.`); } finally { await progress("cleaning_up"); const removed = await this.deleteVm(vm); @@ -191,15 +184,18 @@ export class ExeVmBackend implements ExecutionBackend { } } - const prompt = base64(request.prompt), code = request.code, output = request.traceSources?.primary.formatVersion === "codex-exec-jsonl" ? request.traceSources.primary.path : "/tmp/factorize.log", unit = vm; + const code = request.code, output = request.traceSources?.primary.formatVersion === "codex-exec-jsonl" ? request.traceSources.primary.path : "/tmp/factorize.log", unit = vm; + const launchPayload = JSON.stringify({ version: 1, runId: request.runId, ...(code ? { code } : { prompt: request.prompt, harness: request.harness }), ...(request.traceSources ? { traceSources: request.traceSources } : {}) }); + const launchPayloadPath = "/home/exedev/.factorize-job/launch.json"; const work = [ "set -eu", "mkdir -p /home/exedev/workspace", "cd /home/exedev/workspace", ...(request.traceSources?.primary.kind === "execution_stream" ? [`mkdir -p -- ${shellAtom(request.traceSources.primary.path.slice(0, request.traceSources.primary.path.lastIndexOf("/")) || "/")}`, `touch -- ${shellAtom(request.traceSources.primary.path)}`] : []), "sudo install -d -o exedev -g exedev -m 0700 /home/exedev/.factorize-job", - ...(code ? [`printf '%s' ${shellAtom(base64(code.script))} | base64 -d > /tmp/factorize-code-job.sh`, `printf '%s' ${shellAtom(base64(JSON.stringify(code.context)))} | base64 -d > /home/exedev/.factorize-job/input.json`, "sudo chown exedev:exedev /home/exedev/.factorize-job/input.json"] : [`printf '%s' ${shellAtom(prompt)} | base64 -d > /tmp/factorize-prompt.md`]), - `if [ "$(sudo systemctl show ${shellAtom(unit)} --property=LoadState --value 2>/dev/null || true)" != loaded ]; then sudo systemd-run --quiet --uid=exedev --gid=exedev --unit=${shellAtom(unit)} --property=Type=exec --property=RemainAfterExit=yes --property=WorkingDirectory=/home/exedev/workspace --property=StandardInput=${code ? "/dev/null" : "file:/tmp/factorize-prompt.md"} --property=StandardOutput=${output === "/tmp/factorize.log" ? `append:${output}` : shellAtom(`append:${output}`)} --property=StandardError=append:/tmp/factorize.stderr ${code ? "env FACTORIZE_RESULT_PATH=/home/exedev/.factorize-job/results.json FACTORIZE_INPUT_PATH=/home/exedev/.factorize-job/input.json bash /tmp/factorize-code-job.sh" : harnessCommand(request.harness!)}; fi`, + `printf '%s' ${shellAtom(base64(FACTORIZE_JOB_LAUNCHER))} | sudo sh -c ${shellAtom("base64 -d > /usr/local/bin/factorize-job && chmod 0755 /usr/local/bin/factorize-job")}`, + `printf '%s' ${shellAtom(base64(launchPayload))} | base64 -d > ${shellAtom(launchPayloadPath)} && sudo chown exedev:exedev ${shellAtom(launchPayloadPath)}`, + `if [ "$(sudo systemctl show ${shellAtom(unit)} --property=LoadState --value 2>/dev/null || true)" != loaded ]; then sudo systemd-run --quiet --uid=exedev --gid=exedev --unit=${shellAtom(unit)} --property=Type=exec --property=RemainAfterExit=yes --property=WorkingDirectory=/home/exedev/workspace --property=StandardInput=/dev/null --property=StandardOutput=${output === "/tmp/factorize.log" ? `append:${output}` : shellAtom(`append:${output}`)} --property=StandardError=append:/tmp/factorize.stderr /bin/sh -c ${shellAtom(`exec /usr/local/bin/factorize-job launch --foreground < ${launchPayloadPath}`)}; fi`, "echo started", ].join("; "); if (request.traceSources?.primary.formatVersion === "codex-exec-jsonl") { diff --git a/packages/api/src/execution.ts b/packages/api/src/execution.ts index 4fa6d1ee..4e4e5f53 100644 --- a/packages/api/src/execution.ts +++ b/packages/api/src/execution.ts @@ -24,7 +24,7 @@ export interface LaunchRequest { } /** Opaque to orchestration. Only the backend that created it may interpret id. */ -export interface RunHandle { backendKind: string; id: string; runProxyName?: string; } +export interface RunHandle { backendKind: string; id: string; runProxyName?: string; /** Stable provider operation key retained across a scheduler restart. */ operationId?: string; } export interface ExecutionObservation { systemd?: import("./harness-diagnostics").ExecutionDiagnostics["systemd"]; diff --git a/packages/api/src/factorize-job-launcher.ts b/packages/api/src/factorize-job-launcher.ts new file mode 100644 index 00000000..5ff8673b --- /dev/null +++ b/packages/api/src/factorize-job-launcher.ts @@ -0,0 +1,153 @@ +/** Small guest-side supervisor shared by Fly and exe.dev execution backends. */ +export const FACTORIZE_JOB_LAUNCHER = String.raw`#!/usr/bin/env python3 +import json +import os +import pathlib +import re +import subprocess +import sys +import tempfile + +ROOT = pathlib.Path('/home/exedev/.factorize-job') +PAYLOAD = ROOT / 'launch.json' +STATE = ROOT / 'process.json' +RESULT = ROOT / 'results.json' +LOG = pathlib.Path('/tmp/factorize.log') +ERR = pathlib.Path('/tmp/factorize.stderr') + +def save(path, value): + fd, temp = tempfile.mkstemp(prefix='.factorize-', dir=str(path.parent)) + try: + with os.fdopen(fd, 'w', encoding='utf-8') as stream: + json.dump(value, stream, separators=(',', ':')) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temp, path) + finally: + try: os.unlink(temp) + except FileNotFoundError: pass + +def load_payload(): + raw = sys.stdin.buffer.read(32 * 1024 * 1024 + 1) + if len(raw) > 32 * 1024 * 1024: + raise ValueError('launch payload exceeds 32 MiB') + value = json.loads(raw) + if not isinstance(value, dict) or value.get('version') != 1: + raise ValueError('unsupported launch payload') + if ('code' in value) == ('harness' in value): + raise ValueError('launch payload must contain code or harness') + return value + +def child_command(payload): + env = os.environ.copy() + env['FACTORIZE_RESULT_PATH'] = str(RESULT) + if 'code' in payload: + proxy_url = env.get('FACTORIZE_PROXY_URL') + if proxy_url: + for name in ('HTTP_PROXY', 'HTTPS_PROXY', 'http_proxy', 'https_proxy'): + env[name] = proxy_url + code = payload['code'] + if not isinstance(code, dict) or not isinstance(code.get('script'), str) or not isinstance(code.get('context'), dict): + raise ValueError('invalid code launch payload') + script = ROOT / 'job.sh' + script.write_text(code['script'], encoding='utf-8') + script.chmod(0o700) + (ROOT / 'input.json').write_text(json.dumps(code['context']), encoding='utf-8') + env['FACTORIZE_INPUT_PATH'] = str(ROOT / 'input.json') + return ['/bin/bash', str(script)], b'', env + harness = payload['harness'] + if not isinstance(harness, dict) or not isinstance(harness.get('executable'), str) or not harness['executable'] or '\x00' in harness['executable'] or not isinstance(harness.get('args'), list): + raise ValueError('invalid agent launch payload') + if not all(isinstance(arg, str) and '\x00' not in arg for arg in harness['args']): + raise ValueError('invalid agent arguments') + for name, value in harness.get('env', {}).items(): + if not isinstance(name, str) or not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_]*', name) or not isinstance(value, str) or '\x00' in value: + raise ValueError('invalid agent environment') + env[name] = value + proxy_url = os.environ.get('FACTORIZE_PROXY_URL') + if proxy_url: + env['FACTORIZE_PROXY_URL'] = proxy_url + for name in ('HTTP_PROXY', 'HTTPS_PROXY', 'http_proxy', 'https_proxy'): + env[name] = proxy_url + env['FACTORIZE_RESULT_PATH'] = str(RESULT) + return [harness['executable'], *harness['args']], str(payload.get('prompt', '')).encode(), env + +def launch(foreground=False): + ROOT.mkdir(mode=0o700, parents=True, exist_ok=True) + if STATE.exists(): + current = json.loads(STATE.read_text()) + pid = current.get('pid') + if isinstance(pid, int): + try: os.kill(pid, 0) + except ProcessLookupError: pass + else: raise RuntimeError('a Factorize job is already active') + payload = load_payload() + source = payload.get('traceSources', {}).get('primary', {}) + trace_path = source.get('path') if source.get('kind') == 'execution_stream' else None + if trace_path: + trace = pathlib.PurePosixPath(trace_path) + if not trace.is_absolute() or '..' in trace.parts: + raise ValueError('invalid execution trace path') + pathlib.Path(trace).parent.mkdir(mode=0o700, parents=True, exist_ok=True) + pathlib.Path(trace).touch(mode=0o600, exist_ok=True) + command, stdin, env = child_command(payload) + PAYLOAD.write_text(json.dumps({key: value for key, value in payload.items() if key != 'prompt'}), encoding='utf-8') + input_path = ROOT / 'prompt.txt' + input_path.write_bytes(stdin) + save(STATE, {'state': 'starting'}) + stdout_path = pathlib.Path(trace_path) if trace_path else LOG + if foreground: + save(STATE, {'state': 'running', 'pid': os.getpid()}) + try: + with input_path.open('rb') as input_stream: + result = subprocess.run(command, stdin=input_stream, env=env, check=False) + save(STATE, {'state': 'exited', 'exitCode': result.returncode}) + raise SystemExit(result.returncode) + except BaseException as error: + if isinstance(error, SystemExit): raise + save(STATE, {'state': 'exited', 'exitCode': 127, 'error': str(error)[:500]}) + raise + with stdout_path.open('ab', buffering=0) as stdout, ERR.open('ab', buffering=0) as stderr: + process = subprocess.Popen([sys.executable, __file__, '_run', json.dumps(command), str(input_path)], stdin=subprocess.PIPE, stdout=stdout, stderr=stderr, env=env, start_new_session=True, close_fds=True) + save(STATE, {'state': 'running', 'pid': process.pid}) + process.stdin.write(b'go\n') + process.stdin.close() + print('started') + +def run(command, input_path): + try: + if not sys.stdin.buffer.readline(): + raise RuntimeError('launcher did not confirm process ownership') + with open(input_path, 'rb') as input_stream: + result = subprocess.run(command, stdin=input_stream, check=False) + save(STATE, {'state': 'exited', 'exitCode': result.returncode}) + raise SystemExit(result.returncode) + except BaseException as error: + if isinstance(error, SystemExit): raise + save(STATE, {'state': 'exited', 'exitCode': 127, 'error': str(error)[:500]}) + raise + +def status(): + try: value = json.loads(STATE.read_text()) + except FileNotFoundError: + print('missing') + return 2 + if value.get('state') == 'running' and isinstance(value.get('pid'), int): + try: os.kill(value['pid'], 0) + except ProcessLookupError: + value = {'state': 'exited', 'exitCode': 127, 'error': 'job supervisor exited without recording a result'} + save(STATE, value) + print(json.dumps(value, separators=(',', ':'))) + return 0 + +if __name__ == '__main__': + try: + if sys.argv[1:] == ['launch']: launch() + elif sys.argv[1:] == ['launch', '--foreground']: launch(foreground=True) + elif len(sys.argv) == 4 and sys.argv[1] == '_run': run(json.loads(sys.argv[2]), sys.argv[3]) + elif sys.argv[1:] == ['status']: raise SystemExit(status()) + else: raise ValueError('usage: factorize-job launch [--foreground]|status') + except Exception as error: + print('factorize-job: ' + str(error)[:500], file=sys.stderr) + raise SystemExit(1) +`; diff --git a/packages/api/src/flow-service.ts b/packages/api/src/flow-service.ts index ab12641d..66c8b0de 100644 --- a/packages/api/src/flow-service.ts +++ b/packages/api/src/flow-service.ts @@ -17,6 +17,8 @@ import type { JobPageQuery } from "./job-contracts"; import { PostgresJobRepository, StaleJobEdit } from "./postgres/job-repository"; import { ConnectionRepository } from "./postgres/connection-repository"; import { EncryptedIntegrationRepository, IntegrationIdempotencyConflictError, IntegrationValidationError, StaleIntegrationRevisionError } from "./postgres/encrypted-integration-repository"; +import { ProxyControlRepository } from "./postgres/proxy-control-repository"; +import { issueModelAccountCapability } from "./proxy-browser-capability"; import { decrypt, encrypt } from "./crypto"; import { InvocationService, type Job, type Trigger } from "./job-domain"; import { RunQueryRepository } from "./postgres/run-query-repository"; @@ -320,6 +322,12 @@ export class ApiService { catch (error) { if (error instanceof TraceReplayError) throw new ServiceError(error.status, error.code, error.message); throw error; } } async stopRun(runId: string) { await this.authorize("runs:cancel"); const stopped = await new RunQueryRepository(databaseFor(this.env), this.auth.tenantId).stop(runId); if (!stopped) throw new ServiceError(409, "operation_failed", "Run is not active"); if (this.env.SCHEDULER) await this.env.SCHEDULER.get(this.env.SCHEDULER.idFromName("global")).fetch("https://scheduler/wake", { method: "POST" }); return stopped; } + async retryBlockedLaunch(runId: string) { + await this.authorizeOwnerSession("runs:cancel"); + if (!await new RunRepository(databaseFor(this.env)).retryBlockedLaunch(this.auth.tenantId, runId)) throw new ServiceError(409, "operation_failed", "Run is not waiting for a manual launch retry"); + if (this.env.SCHEDULER) await this.env.SCHEDULER.get(this.env.SCHEDULER.idFromName("global")).fetch("https://scheduler/wake", { method: "POST" }); + return { runId, retryRequested: true }; + } killRun(runId: string) { return this.stopRun(runId); } async listJobSummaries(input: JobPageQuery, selector = false) { await this.authorize("jobs:read"); try { return await new JobSummaryRepository(databaseFor(this.env), this.auth.tenantId).page(input, selector); } catch (error) { if (error instanceof InvalidJobSummaryCursor) throw new ServiceError(400, "invalid_request", error.message); throw error; } } async listJobs() { await this.authorize("jobs:read"); const jobs = await this.jobs().list(), stats = await this.jobStatistics(jobs.map(job => job.id)); return Promise.all(jobs.map(job => this.presentJob(job, stats.get(job.id) ?? { running_count: "0", last_run_state: null }))); } @@ -576,6 +584,115 @@ export class ApiService { if (!rows.length) throw new ServiceError(409, "conflict", "An unlock key is already set up for this account."); return { generationId: input.generationId }; } + async getUnlockKeyRotationSnapshot() { + await this.authorizeOwnerSession("integrations:read"); + const repository = this.encryptedIntegrations(); + const [snapshot, verification] = await Promise.all([ + repository.rotationSnapshot(), + databaseFor(this.env).pool.query<{ generation_id: string }>("SELECT generation_id FROM app.unlock_key_verifications WHERE tenant_id=$1", [this.auth.tenantId]), + ]); + const generationId = verification.rows[0]?.generation_id; + if (!generationId) throw new ServiceError(409, "conflict", "Set up an unlock key before rotating or resetting integrations."); + const modelRevisions = new Map(); + const changes = snapshot.changes.map(change => { + let recordRevision: number | null = null; + if (change.kind === "model" && change.operation === "delete") modelRevisions.delete(change.id); + if (change.kind === "model" && change.operation === "upsert" && change.encryptedPayload !== null) { + recordRevision = modelRevisions.has(change.id) ? modelRevisions.get(change.id)! + 1 : change.revision; + modelRevisions.set(change.id, recordRevision); + } + return { ...change, recordRevision }; + }); + return { ownerId: this.auth.userId, generationId, ...snapshot, changes }; + } + async replaceUnlockKeyGeneration(input: { + expectedGenerationId: string; generationId: string; expectedRevision: number; encryptedVerification: string; + records: Array<{ id: string; kind: "bearer" | "model"; name: string; origin: string | null; description: string | null; revision: number; encryptedPayload: string | null; deletedAt: string | null }>; + changes: Array<{ id: string; kind: "bearer" | "model"; name: string; origin: string | null; description: string | null; revision: number; idempotencyKey: string; operation: "upsert" | "delete"; encryptedPayload: string | null; recordRevision?: number | null }>; + }) { + await this.authorizeOwnerSession("integrations:write"); + if (input.expectedGenerationId === input.generationId) throw new ServiceError(400, "invalid_request", "A replacement key generation must be new."); + const repository = this.encryptedIntegrations(); + const records = input.records.map(record => ({ ...record, deletedAt: record.deletedAt === null ? null : new Date(record.deletedAt) })); + const changes = input.changes.map(({ recordRevision: _recordRevision, ...change }) => change); + try { + await repository.stageGeneration({ generationId: input.generationId, expectedRevision: input.expectedRevision, records, changes }); + const committed = await repository.commitGeneration(input.generationId, input.expectedRevision, { + expectedVerificationGenerationId: input.expectedGenerationId, + verificationGenerationId: input.generationId, + encryptedVerification: input.encryptedVerification, + }); + return { generationId: input.generationId, ...committed }; + } catch (error) { + if (error instanceof StaleIntegrationRevisionError || (error as Error)?.name === "StaleIntegrationGenerationError") throw new ServiceError(409, "stale_revision", "Integration records changed during key replacement. Reload and retry with the current unlock key."); + if (error instanceof IntegrationIdempotencyConflictError) throw new ServiceError(409, "conflict", "This key replacement ID was already used for different encrypted data."); + if (error instanceof IntegrationValidationError) throw new ServiceError(400, "invalid_request", error.message); + throw error; + } + } + async issueModelAccountBrowserCapability(accountId: string, action: "authorize" | "status" | "cancel") { + await this.authorizeOwnerSession("integrations:write"); + if (!this.env.MODEL_ACCOUNT_CAPABILITY_PRIVATE_KEY) throw new ServiceError(503, "operation_failed", "Model-account browser authorization is unavailable."); + const [pairing, accounts] = await Promise.all([ + new ProxyControlRepository(databaseFor(this.env)).browserPairing(this.auth.tenantId), + this.encryptedIntegrations().records("model"), + ]); + if (!pairing) throw new ServiceError(503, "operation_failed", "The credential proxy is not paired for browser authorization."); + if (!accounts.some(account => account.id === accountId)) throw new ServiceError(404, "not_found", "Model account not found."); + const capability = await issueModelAccountCapability({ + privateKeyPem: this.env.MODEL_ACCOUNT_CAPABILITY_PRIVATE_KEY, + proxyIdentity: pairing.proxyIdentity, ownerId: this.auth.tenantId, accountId, action, + appOrigin: this.env.APP_ORIGIN, generation: pairing.bootEpoch, + }); + return { endpoint: `${pairing.browserOrigin}/model-accounts/${accountId}/${action}`, capability }; + } + async listModelAccounts() { + await this.authorizeOwnerSession("integrations:read"); + const repository = this.encryptedIntegrations(); + const [revision, accounts] = await Promise.all([repository.revision(), repository.records("model")]); + return { revision, accounts: accounts.map(({ id, name, description, revision }) => ({ id, name, description, revision })) }; + } + private async modelAccountWrite(input: { id: string; expectedRevision: number; idempotencyKey: string; name: string; description: string | null; encryptedPayload?: string; operation: "upsert" | "delete" }) { + try { return await this.encryptedIntegrations().write({ ...input, kind: "model", origin: null }); } + catch (error) { + if (error instanceof StaleIntegrationRevisionError) throw new ServiceError(409, "stale_revision", `The model account collection changed. Current revision is ${error.currentRevision}.`); + if (error instanceof IntegrationIdempotencyConflictError) throw new ServiceError(409, "conflict", "The idempotency key was already used for a different model account change."); + if (error instanceof IntegrationValidationError) throw new ServiceError(400, "invalid_request", error.message); + throw error; + } + } + async createModelAccount(input: { id: string; expectedRevision: number; idempotencyKey: string; name: string; description: string | null; encryptedPayload: string }) { + await this.authorizeOwnerSession("integrations:write"); + const repository = this.encryptedIntegrations(); + try { + const replay = await repository.replay({ ...input, kind: "model", origin: null, operation: "upsert" }); + if (replay) return replay; + } catch (error) { + if (error instanceof IntegrationIdempotencyConflictError) throw new ServiceError(409, "conflict", "The idempotency key was already used for a different model account change."); + if (error instanceof IntegrationValidationError) throw new ServiceError(400, "invalid_request", error.message); + throw error; + } + if ((await repository.records()).some(record => record.id === input.id)) throw new ServiceError(409, "conflict", "An integration with this id already exists."); + return this.modelAccountWrite({ ...input, operation: "upsert" }); + } + async deleteModelAccount(id: string, input: { expectedRevision: number; idempotencyKey: string }) { + await this.authorizeOwnerSession("integrations:write"); + const repository = this.encryptedIntegrations(); + const existing = (await repository.records("model")).find(record => record.id === id); + if (!existing) { + try { + const replay = await repository.replayDelete(id, "model", input.idempotencyKey); + if (replay) return { id, deleted: true as const, ...replay }; + } catch (error) { + if (error instanceof IntegrationIdempotencyConflictError) throw new ServiceError(409, "conflict", "The idempotency key was already used for a different model account change."); + if (error instanceof IntegrationValidationError) throw new ServiceError(400, "invalid_request", error.message); + throw error; + } + throw new ServiceError(404, "not_found", "Model account not found."); + } + const result = await this.modelAccountWrite({ id, name: existing.name, description: existing.description, ...input, operation: "delete" }); + return { id, deleted: true as const, ...result }; + } async removeTailIntegration(id: string) { await this.authorize("integrations:write"); if (!await this.connections().delete(`cloudflare-tail:${id}`)) throw new ServiceError(404, "not_found", "Integration not found"); return { id, deleted: true }; } async linearProjects() { await this.authorize("integrations:read"); return publicValue(z.array(namedOption), await this.providers().linearProjects()); } async linearOptions() { await this.authorize("integrations:read"); return publicValue(providerOptions, await this.providers().linearOptions()); } diff --git a/packages/api/src/fly-credential-proxy-provisioner.ts b/packages/api/src/fly-credential-proxy-provisioner.ts new file mode 100644 index 00000000..a6d5dac8 --- /dev/null +++ b/packages/api/src/fly-credential-proxy-provisioner.ts @@ -0,0 +1,186 @@ +import { provisionCredentialProxy, type ProxyBootstrapProvisioning } from "./proxy-provisioner"; +import type { ProxyControlRepository } from "./postgres/proxy-control-repository"; +import type { FlyMachine, FlyMachineConfig } from "./fly-machines-client"; +import { createFlyGuestTransport } from "./fly-guest-transport"; + +const IMAGE_DIGEST = /@sha256:[a-f0-9]{64}$/i; +const UUID_V4 = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +export type FlyCredentialProxyConfig = { + tenantId: string; + app: string; + /** Only reviewed credential-proxy images pinned by digest are accepted. */ + image: string; + region: string; + size: string; + browserCapabilityPublicKey: string; + memoryMb?: number; + controlOrigin: string; + /** Stable across retries of one provisioning operation. */ + operationId: string; +}; + +export type FlyCredentialProxyMachineClient = { + createMachine(app: string, config: FlyMachineConfig, operationId: string): Promise; + startMachine(app: string, machine: string): Promise; + destroyMachine(app: string, machine: string): Promise; +}; + +export type FlyCredentialProxyGuest = { + run(command: readonly string[], input?: string | Uint8Array): Promise<{ ok: boolean; exitCode: number; stdout: string; stderr: string }>; + /** The reviewed image startup command consumes this payload from stdin only. */ + startProxy(input: ProxyBootstrapProvisioning & { credential: string }): Promise; + /** Signs a VM-generated CSR over the provider's private guest channel. Only + * the public certificate and CA trust bundle return to the caller. */ + issueVmCertificate(input: { bootEpoch: string; vmIdentity: string; csr: string }): Promise<{ certificate: string; trustBundle: string }>; +}; + +/** Provider exec adapter: the bootstrap record is the command's stdin payload. */ +export function createFlyCredentialProxyGuest(options: { + app: string; + machine: string; + token: string; + cliPath?: string; + spawnProcess?: typeof import("node:child_process").spawn; +}): FlyCredentialProxyGuest { + const transport = createFlyGuestTransport({ ...options, timeoutMs: 30_000, outputLimitBytes: 16 * 1024 }); + return { + run: (command, input) => transport.run(command, input), + async startProxy(input) { + const result = await transport.run(["/usr/local/bin/factorize-proxy-startup"], `${JSON.stringify({ + type: "register-boot", + controlOrigin: input.controlOrigin, + browserOrigin: input.browserOrigin, + bootEpoch: input.bootEpoch, + credential: input.credential, + })}\n`); + if (!result.ok) throw new Error("Credential proxy startup command failed"); + if (result.stdout.trim() !== "Credential proxy started") throw new Error("Credential proxy startup was not confirmed"); + }, + async issueVmCertificate(input) { + if (!input || !UUID_V4.test(input.bootEpoch) || !/^[A-Za-z0-9:._-]{1,255}$/.test(input.vmIdentity) + || typeof input.csr !== "string" || input.csr.length < 100 || input.csr.length > 16_384) throw new TypeError("Invalid VM certificate provisioning request"); + const result = await transport.run(["/usr/local/bin/factorize-proxy-certify-vm"], `${JSON.stringify(input)}\n`); + if (!result.ok) throw new Error("Credential proxy VM certificate provisioning failed"); + let response: unknown; + try { response = JSON.parse(result.stdout); } catch { throw new Error("Credential proxy returned an invalid VM certificate response"); } + if (!response || typeof response !== "object" || Array.isArray(response) + || Object.keys(response).sort().join(",") !== "bootEpoch,certificate,trustBundle,vmIdentity" + || (response as any).bootEpoch !== input.bootEpoch || (response as any).vmIdentity !== input.vmIdentity + || typeof (response as any).certificate !== "string" || !(response as any).certificate.startsWith("-----BEGIN CERTIFICATE-----") + || typeof (response as any).trustBundle !== "string" || !(response as any).trustBundle.startsWith("-----BEGIN CERTIFICATE-----")) { + throw new Error("Credential proxy returned an invalid VM certificate response"); + } + return { certificate: (response as any).certificate, trustBundle: (response as any).trustBundle }; + }, + }; +} + +export type FlyCredentialProxyProvisionerOptions = { + control: Pick; + machines: FlyCredentialProxyMachineClient; + guest: (machine: FlyMachine) => FlyCredentialProxyGuest; + /** Exact reviewed proxy artifact selected by deployment configuration. */ + reviewedImage: string; + /** Injectable to keep boot identity deterministic in boundary tests. */ + newBootEpoch?: () => string; +}; + +function validate(input: FlyCredentialProxyConfig, reviewedImage: string): void { + if (!input.tenantId || !input.app || !input.operationId || input.operationId.length > 200) throw new TypeError("Invalid credential proxy provisioning identity"); + if (!/^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/.test(input.app)) throw new TypeError("Invalid credential proxy Fly app name"); + if (!IMAGE_DIGEST.test(input.image)) throw new TypeError("Credential proxy image must be pinned by sha256 digest"); + if (!IMAGE_DIGEST.test(reviewedImage) || input.image !== reviewedImage) throw new TypeError("Credential proxy image must match the configured reviewed image"); + if (!/^[a-z0-9]{3}$/.test(input.region)) throw new TypeError("Invalid credential proxy region"); + if (!/^(shared-cpu|performance)-(1|2|4)x$/.test(input.size)) throw new TypeError("Invalid credential proxy size"); + if (!input.browserCapabilityPublicKey.startsWith("-----BEGIN PUBLIC KEY-----")) throw new TypeError("Proxy browser capability public key is required"); + if (input.memoryMb !== undefined && (!Number.isInteger(input.memoryMb) || input.memoryMb < 256 || input.memoryMb > 8192)) throw new TypeError("Credential proxy memory must be between 256 and 8192 MiB"); +} + +/** + * Provisions one dedicated Fly Machine for the credential proxy and wires its + * startup through the one-use bootstrap boundary. The bootstrap credential is + * never part of the Machine config; the guest adapter must deliver it over its + * private stdin startup channel. + */ +export function createFlyCredentialProxyProvisioner(options: FlyCredentialProxyProvisionerOptions) { + return { + async start(input: FlyCredentialProxyConfig): Promise<{ proxyIdentity: string; bootEpoch: string; machine: FlyMachine }> { + validate(input, options.reviewedImage); + const origin = new URL(input.controlOrigin); + if (origin.protocol !== "https:" || origin.username || origin.password || origin.pathname !== "/" || origin.search || origin.hash) { + throw new TypeError("controlOrigin must be an HTTPS origin"); + } + const candidateBootEpoch = (options.newBootEpoch ?? (() => crypto.randomUUID()))(); + if (!UUID_V4.test(candidateBootEpoch)) throw new TypeError("boot epoch must be a UUID v4"); + const size = input.size.match(/^(shared-cpu|performance)-(1|2|4)x$/)!; + const config: FlyMachineConfig = { + image: input.image, + region: input.region, + cmd: ["/usr/local/bin/factorize-credential-proxy", "--boot-epoch", candidateBootEpoch], + guest: { cpu_kind: size[1] === "shared-cpu" ? "shared" : "performance", cpus: Number(size[2]), memory_mb: input.memoryMb ?? 512 }, + env: { + FACTORIZE_PROXY_TUNNEL_SERVER_NAME: `${input.app}.internal`, + FACTORIZE_PROXY_TUNNEL_PORT: "9090", + FACTORIZE_PROXY_BROWSER_CAPABILITY_PUBLIC_KEY: input.browserCapabilityPublicKey, + }, + services: [ + { protocol: "tcp", internal_port: 8080, ports: [{ port: 443, handlers: ["tls", "http"] }], force_instance_key: true }, + // Private 6PN service for job VMs. Tunnel TLS terminates in the + // credential proxy, where the per-VM client certificate is checked. + { protocol: "tcp", internal_port: 9090, ports: [] }, + ], + metadata: { + factorize_role: "credential-proxy", + factorize_tenant_id: input.tenantId, + factorize_operation_id: input.operationId, + factorize_boot_epoch: candidateBootEpoch, + }, + // The proxy must be explicitly started and must not be restarted after + // process failure with a stale one-use bootstrap credential. + auto_start: false, + auto_stop: false, + restart: { policy: "no" }, + }; + const machine = await options.machines.createMachine(input.app, config, input.operationId); + if (!machine.id) throw new Error("Fly returned a credential proxy Machine without an id"); + try { + const metadata = machine.config?.metadata; + const bootEpoch = metadata?.factorize_boot_epoch; + // The create response is the identity boundary for this Machine. Do not + // infer missing identity from the request: Fly may have reconciled an + // older or foreign Machine for the same operation key, and bootstrap + // authority must only be issued after all returned bindings match. + if (!metadata || !UUID_V4.test(bootEpoch ?? "") || bootEpoch !== candidateBootEpoch + || metadata.factorize_role !== "credential-proxy" + || metadata.factorize_tenant_id !== input.tenantId + || metadata.factorize_operation_id !== input.operationId) { + throw new Error("Fly returned a credential proxy Machine with mismatched provisioning identity"); + } + const proxyIdentity = `fly:${input.app}:${machine.id}`; + const bootstrap = { + tenantId: input.tenantId, + proxyIdentity, + bootEpoch, + controlOrigin: origin.origin, + browserOrigin: `https://${input.app}.fly.dev`, + }; + await provisionCredentialProxy(bootstrap, { + control: options.control, + startProxy: async (request) => { + // Fly Machine startup is explicit. Machine config and metadata only + // hold public identity/configuration; the token crosses stdin. + await options.machines.startMachine(input.app, machine.id); + await options.guest(machine).startProxy(request); + }, + }); + return { proxyIdentity, bootEpoch, machine }; + } catch (error) { + // Preserve the sanitized original provisioning failure while making a + // best-effort cleanup of a machine that never completed startup. + await options.machines.destroyMachine(input.app, machine.id).catch(() => undefined); + throw error; + } + }, + }; +} diff --git a/packages/api/src/fly-job-execution-backend.ts b/packages/api/src/fly-job-execution-backend.ts new file mode 100644 index 00000000..d2f4066b --- /dev/null +++ b/packages/api/src/fly-job-execution-backend.ts @@ -0,0 +1,279 @@ +import type { ExecutionBackend, ExecutionObservation, LaunchReceipt, LaunchRequest, RunHandle } from "./execution"; +import type { FlyMachine, FlyMachineConfig } from "./fly-machines-client"; +import { flyJobMachineConfig } from "./fly-job-machine"; +import { createFlyGuestTransport } from "./fly-guest-transport"; +import { createHash } from "node:crypto"; +import { safeDiagnosticText } from "./harness-diagnostics"; +import type { ArtifactCollectionRequest, ArtifactCollector, ExecutionTerminator, HarnessLogReader, JobFileCollectionRequest, JobFileCollector, JobResultReader } from "./execution"; +import { sourceCommand } from "./trace-source"; +import { FACTORIZE_JOB_LAUNCHER } from "./factorize-job-launcher"; +import { FACTORIZE_FLY_TUNNEL_HELPER } from "./fly-job-tunnel-helper"; + +const SUPPORTED_CODEX_CLI_VERSION = "codex-cli 0.167.1"; + +export type FlyJobMachines = { + findMachineByOperationId?(app: string, operationId: string): Promise; + createMachine(app: string, config: FlyMachineConfig, operationId: string): Promise; + inspectMachine(app: string, machine: string): Promise; + startMachine(app: string, machine: string): Promise; + stopMachine(app: string, machine: string): Promise; + destroyMachine(app: string, machine: string): Promise; +}; + +export type FlyJobGuest = { + run(command: readonly string[], input?: string | Uint8Array): Promise<{ ok: boolean; exitCode: number; stdout: string; stderr: string }>; + upload?(path: string, bytes: Uint8Array): Promise; + download?(path: string): Promise; +}; + +export type FlyProxyGuest = Pick & { + issueVmCertificate?(input: { bootEpoch: string; vmIdentity: string; csr: string }): Promise<{ certificate: string; trustBundle: string }>; +}; + +const capabilities = ["stop", "recovery"] as const; + +/** Fly Machine lifecycle adapter. Machine creation is idempotent by run ID; + * job payload delivery is a separate guest operation and never enters machine config. */ +export class FlyJobExecutionBackend implements ExecutionBackend, JobResultReader, HarnessLogReader, JobFileCollector, ArtifactCollector, ExecutionTerminator { + readonly kind = "fly"; + readonly capabilities = capabilities; + + constructor( + private readonly options: { + app: string; + image: string; + region: string; + size: string; + machines: FlyJobMachines; + guest: (machine: FlyMachine) => FlyJobGuest; + proxyIdentity?: string; + proxyBootEpoch?: string; + proxyGuest?: FlyProxyGuest; + destinationUrl?: (machine: FlyMachine) => string; + }, + ) {} + + reserve(runId: string): RunHandle { + return { backendKind: this.kind, id: `${this.options.app}/${runId}`, operationId: runId }; + } + + private handleParts(handle: RunHandle): { id: string; operationId?: string } { + if (handle.backendKind !== this.kind || !handle.id.startsWith(`${this.options.app}/`)) throw new TypeError("Invalid Fly execution handle"); + const id = handle.id.slice(this.options.app.length + 1); + if (!id || id.includes("/")) throw new TypeError("Invalid Fly execution handle"); + return { id, ...(handle.operationId ? { operationId: handle.operationId } : {}) }; + } + + private async machineId(handle: RunHandle): Promise { + const parts = this.handleParts(handle); + if (parts.operationId && parts.id === parts.operationId) { + const machine = await this.options.machines.findMachineByOperationId?.(this.options.app, parts.operationId); + return machine?.id ?? null; + } + return parts.id; + } + + async launch(request: LaunchRequest): Promise { + const operationId = request.runId; + const handle = request.reservation ?? this.reserve(request.runId); + this.handleParts(handle); + const config = flyJobMachineConfig(this.options.app, { + runId: request.runId, + operationId, + image: this.options.image, + region: this.options.region, + size: this.options.size, + }); + const machine = await this.options.machines.createMachine(this.options.app, config, operationId); + if (!machine.id) throw new Error("Fly returned a Machine without an id"); + const actualHandle = { backendKind: this.kind, id: `${this.options.app}/${machine.id}`, operationId }; + try { + if (request.runProxy) { + if (!this.options.proxyIdentity || !this.options.proxyBootEpoch || !this.options.proxyGuest?.issueVmCertificate) throw new Error("Fly credential proxy delivery is unavailable"); + const vmIdentity = `fly:${this.options.app}:${machine.id}`; + const delivered = await this.options.proxyGuest.run(["/usr/local/bin/factorize-proxy-authorize-run"], `${JSON.stringify({ + runId: request.runId, vmIdentity, credential: request.runProxy.credential, + })}\n`); + if (!delivered.ok || delivered.stdout.trim() !== "Run credential delivered to proxy memory") throw new Error("Fly credential proxy did not confirm run credential delivery"); + } + await this.options.machines.startMachine(this.options.app, machine.id); + const guest = this.options.guest(machine); + const tunnelPrerequisites = request.runProxy ? " && command -v node >/dev/null && command -v openssl >/dev/null" : ""; + const installed = await guest.run(["/bin/sh", "-c", `command -v python3 >/dev/null && command -v runuser >/dev/null${tunnelPrerequisites} && id exedev >/dev/null && mkdir -p /usr/local/bin && install -d -o exedev -g exedev -m 0700 /home/exedev/.factorize-job && cat > /usr/local/bin/factorize-job && chmod 0755 /usr/local/bin/factorize-job`], FACTORIZE_JOB_LAUNCHER); + if (!installed.ok) throw new Error(request.runProxy + ? "Fly guest is missing python3, runuser, Node.js, OpenSSL, or the exedev user, or could not install the shared Factorize job launcher" + : "Fly guest is missing python3, runuser, or the exedev user, or could not install the shared Factorize job launcher"); + if (request.harness?.executable === "codex") { + const version = await guest.run(["/usr/sbin/runuser", "-u", "exedev", "--", "codex", "--version"]); + if (!version.ok || version.stdout.trim() !== SUPPORTED_CODEX_CLI_VERSION) { + throw new Error(`Fly guest requires ${SUPPORTED_CODEX_CLI_VERSION} for Codex jobs`); + } + } + if (request.runProxy) { + if (!guest.upload) throw new Error("Fly guest transport cannot install the credential proxy tunnel helper"); + const vmIdentity = `fly:${this.options.app}:${machine.id}`; + const proxyHost = `${this.options.app}.internal`; + const proxyPort = 9090; + const localPort = 18180; + const directory = "/home/exedev/.factorize-job/proxy"; + const directoryReady = await guest.run(["/usr/sbin/install", "-d", "-o", "exedev", "-g", "exedev", "-m", "0700", directory]); + if (!directoryReady.ok) throw new Error("Fly guest could not prepare its proxy tunnel directory"); + const prepared = await guest.run(["/usr/bin/openssl", "req", "-new", "-newkey", "ec", "-pkeyopt", "ec_paramgen_curve:P-256", "-nodes", "-keyout", `${directory}/client.key`, "-out", `${directory}/client.csr`, "-subj", `/CN=${vmIdentity}`]); + if (!prepared.ok) throw new Error("Fly guest could not create its proxy tunnel identity"); + const csr = await guest.run(["/bin/cat", `${directory}/client.csr`]); + if (!csr.ok) throw new Error("Fly guest did not return its proxy tunnel certificate request"); + const issued = await this.options.proxyGuest!.issueVmCertificate!({ bootEpoch: this.options.proxyBootEpoch!, vmIdentity, csr: csr.stdout.trim() }); + const configured = await guest.run(["/bin/sh", "-c", `install -d -o exedev -g exedev -m 0700 ${directory} && chown exedev:exedev ${directory}/client.key && chmod 0600 ${directory}/client.key && cat > ${directory}/certificates.json`], JSON.stringify({ certificate: issued.certificate, trustBundle: issued.trustBundle })); + if (!configured.ok) throw new Error("Fly guest could not install its proxy tunnel certificate"); + const extracted = await guest.run(["node", "-e", `const fs=require('node:fs');const p=${JSON.stringify(directory)};const v=JSON.parse(fs.readFileSync(p+'/certificates.json','utf8'));if(typeof v.certificate!=='string'||typeof v.trustBundle!=='string')process.exit(2);fs.writeFileSync(p+'/client.crt',v.certificate,{mode:0o600});fs.writeFileSync(p+'/ca.crt',v.trustBundle,{mode:0o600});fs.unlinkSync(p+'/certificates.json');`]); + if (!extracted.ok || !guest.upload) throw new Error("Fly guest could not prepare its proxy tunnel certificate files"); + const config = { runId: request.runId, proxyHost, proxyPort, localPort }; + const tunnelConfig = await guest.run(["/bin/sh", "-c", `chown exedev:exedev ${directory}/client.crt ${directory}/ca.crt && cat > ${directory}/config.json`], JSON.stringify(config)); + if (!tunnelConfig.ok) throw new Error("Fly guest could not configure its proxy tunnel"); + await guest.upload(`${directory}/tunnel.mjs`, new TextEncoder().encode(FACTORIZE_FLY_TUNNEL_HELPER)); + const startTunnel = await guest.run(["/bin/sh", "-c", `chown exedev:exedev ${directory}/config.json ${directory}/tunnel.mjs && chmod 0600 ${directory}/config.json ${directory}/tunnel.mjs && (nohup /usr/sbin/runuser -u exedev -- /usr/bin/env PATH=/usr/local/bin:/usr/bin:/bin node ${directory}/tunnel.mjs >/tmp/factorize-proxy-tunnel.log 2>&1 /tmp/factorize-proxy-tunnel.pid) && sleep 1 && kill -0 "$(cat /tmp/factorize-proxy-tunnel.pid)"`]); + if (!startTunnel.ok) throw new Error("Fly guest could not start its credential proxy tunnel"); + } + const payload = JSON.stringify({ version: 1, runId: request.runId, ...(request.code ? { code: request.code } : { prompt: request.prompt, harness: request.harness }), ...(request.traceSources ? { traceSources: request.traceSources } : {}) }); + const launchCommand = request.runProxy + ? ["/usr/sbin/runuser", "-u", "exedev", "--", "/usr/bin/env", "FACTORIZE_PROXY_URL=http://127.0.0.1:18180", "/usr/local/bin/factorize-job", "launch"] + : ["/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "launch"]; + const started = await guest.run(launchCommand, `${payload}\n`); + if (!started.ok || started.stdout.trim() !== "started") throw new Error("Fly guest did not confirm job launch"); + return { + handle: actualHandle, + destinationUrl: this.options.destinationUrl?.(machine) ?? `fly://${this.options.app}/${machine.id}`, + capabilities, + observation: { state: "running" }, + }; + } catch (error) { + // Keep the persisted machine handle available to scheduler cleanup when + // guest startup fails after provider creation. + return { + handle: actualHandle, + destinationUrl: this.options.destinationUrl?.(machine) ?? `fly://${this.options.app}/${machine.id}`, + capabilities, + observation: { state: "failed", detail: error instanceof Error ? error.message : "Fly guest launch failed", failure: { source: "system", code: "launch_failed", reason: "Fly guest launch failed" } }, + }; + } + } + + async inspect(handle: RunHandle): Promise { + const id = await this.machineId(handle); + if (!id) return { state: "stopped" }; + const machine = await this.options.machines.inspectMachine(this.options.app, id); + switch (String(machine.state ?? "").toLowerCase()) { + case "started": { + try { + const result = await this.options.guest(machine).run(["/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "status"]); + if (result.ok) { + const process = JSON.parse(result.stdout) as { state?: string; exitCode?: number }; + if (process.state === "exited") return process.exitCode === 0 ? { state: "succeeded" } : { state: "failed", detail: "The guest job process exited unsuccessfully", failure: { source: "run", code: "process_failed", reason: "The guest job process exited unsuccessfully" } }; + } + } catch { /* Guest status may be temporarily unavailable while booting. */ } + return { state: "running" }; + } + case "starting": return { state: "running" }; + case "stopped": case "destroyed": return { state: "succeeded" }; + case "created": case "scheduling": return { state: "starting" }; + default: return { state: "failed", detail: "Fly Machine entered an unexpected state", failure: { source: "system", code: "provider_state_invalid", reason: "Fly Machine entered an unexpected state" } }; + } + } + + private async guestFor(handle: RunHandle): Promise { + const machine = await this.machineId(handle); + if (!machine) throw new Error("Fly Machine for the persisted run operation was not found"); + return this.options.guest({ id: machine }); + } + + private async download(handle: RunHandle, path: string): Promise { + const guest = await this.guestFor(handle); + if (!guest.download) throw new Error("Fly guest transport cannot collect files"); + return guest.download(path); + } + + async readJobResult(handle: RunHandle): Promise { + const guest = await this.guestFor(handle); + const exists = await guest.run(["/bin/sh", "-c", "test -f /home/exedev/.factorize-job/results.json && printf present || printf missing"]); + if (!exists.ok) throw new Error("Fly structured result availability check failed"); + if (exists.stdout.trim() === "missing") return null; + const bytes = await this.download(handle, "/home/exedev/.factorize-job/results.json"); + if (bytes.byteLength > 65_536) return "OVERSIZE"; + try { return new TextDecoder("utf-8", { fatal: true }).decode(bytes); } + catch { throw new Error("Fly structured result is not valid UTF-8"); } + } + + async readHarnessLog(handle: RunHandle): Promise { + const guest = await this.guestFor(handle); + const result = await guest.run(["/bin/sh", "-c", "if [ -f /tmp/factorize.stderr ]; then printf /tmp/factorize.stderr; elif [ -f /tmp/factorize.log ]; then printf /tmp/factorize.log; else exit 2; fi"]); + if (!result.ok) throw new Error("Fly harness log is unavailable"); + return safeDiagnosticText(new TextDecoder().decode(await this.download(handle, result.stdout.trim())), 256 * 1024); + } + + async listJobFiles(handle: RunHandle): Promise { + const result = await (await this.guestFor(handle)).run(["/bin/sh", "-c", "find /home/exedev/.factorize-job -mindepth 1 -type f ! -name results.json -printf '%P\\t%s\\n'"]); + if (!result.ok) throw new Error("Fly job output directory collection failed"); + const paths: string[] = []; let total = 0; + for (const line of result.stdout.split("\n").filter(Boolean)) { + const match = /^([^\t\r\0]+)\t(\d+)$/.exec(line); + if (!match || match[1]!.startsWith("/") || match[1]!.split("/").some(part => !part || part === "." || part === "..") || /[\\\r\n]/.test(match[1]!)) throw new Error("Fly job output directory contains an unsafe path"); + const size = Number(match[2]); + if (!Number.isSafeInteger(size) || size > 10 * 1024 * 1024) throw new Error("A Fly job output file exceeds the 10 MiB per-file limit"); + total += size; if (total > 50 * 1024 * 1024) throw new Error("Fly job output directory exceeds the 50 MiB total limit"); + paths.push(match[1]!); + } + if (paths.length > 100) throw new Error("Fly job output directory exceeds the 100 file limit"); + return paths; + } + + async collectJobFile(handle: RunHandle, request: JobFileCollectionRequest) { + if (request.path.split("/").some(part => !part || part === "." || part === "..") || /[\\\r\n\0]/.test(request.path)) throw new TypeError("Unsafe Fly job output path"); + const bytes = await this.download(handle, `/home/exedev/.factorize-job/${request.path}`); + if (bytes.byteLength > 10 * 1024 * 1024) throw new Error("Fly job output file exceeds the 10 MiB limit"); + return this.upload(request.uploadUrl, bytes, "application/octet-stream"); + } + + async collectArtifact(handle: RunHandle, request: ArtifactCollectionRequest) { + const guest = await this.guestFor(handle); + const discover = request.source ? sourceCommand(request.source) : request.discoverCommand; + if (!discover) throw new Error("Fly artifact source is unavailable"); + const found = await guest.run(["/bin/sh", "-c", discover]); + if (!found.ok || !found.stdout.trim().startsWith("/")) throw new Error("Fly artifact source could not be resolved"); + return this.upload(request.uploadUrl, await this.download(handle, found.stdout.trim()), request.contentType); + } + + private async upload(uploadUrl: string, bytes: Uint8Array, contentType: string) { + const sha256 = createHash("sha256").update(bytes).digest("hex"); + const response = await fetch(uploadUrl, { method: "PUT", headers: { "Content-Type": contentType, "Content-Length": String(bytes.byteLength), "X-Artifact-SHA256": sha256 }, body: Buffer.from(bytes) }); + return { ok: response.ok, detail: response.ok ? undefined : `Fly artifact upload failed (${response.status})` }; + } + + async stop(handle: RunHandle): Promise { + const id = await this.machineId(handle); + if (!id) return { state: "stopped" }; + const machine = await this.options.machines.inspectMachine(this.options.app, id); + if (["destroyed", "destroying"].includes(String(machine.state ?? "").toLowerCase())) return { state: "stopped" }; + if (String(machine.state ?? "").toLowerCase() !== "stopped") { + await this.options.machines.stopMachine(this.options.app, id); + } + const after = await this.options.machines.inspectMachine(this.options.app, id); + if (!["stopped", "destroyed"].includes(String(after.state ?? "").toLowerCase())) return { state: "stopping" }; + await this.options.machines.destroyMachine(this.options.app, id); + return { state: "stopped" }; + } + + async terminate(handle: RunHandle): Promise { + const id = await this.machineId(handle); + if (!id) return { state: "stopped" }; + const machine = await this.options.machines.inspectMachine(this.options.app, id); + if (["destroyed", "destroying", "stopped"].includes(String(machine.state ?? "").toLowerCase())) return { state: "stopped" }; + await this.options.machines.stopMachine(this.options.app, id); + const after = await this.options.machines.inspectMachine(this.options.app, id); + return ["stopped", "destroyed"].includes(String(after.state ?? "").toLowerCase()) ? { state: "stopped" } : { state: "stopping" }; + } +} + +/** Production guest adapter; CLI credentials stay in the scheduler process. */ +export function flyJobGuest(options: { app: string; machine: string; token: string }) { + return createFlyGuestTransport({ ...options, timeoutMs: 30_000, outputLimitBytes: 64 * 1024 }); +} diff --git a/packages/api/src/fly-job-machine.ts b/packages/api/src/fly-job-machine.ts new file mode 100644 index 00000000..98ccf2b7 --- /dev/null +++ b/packages/api/src/fly-job-machine.ts @@ -0,0 +1,42 @@ +import type { FlyMachineConfig } from "./fly-machines-client"; + +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; +const APP = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/; +const IMAGE = /^[A-Za-z0-9][A-Za-z0-9._/:@-]{0,511}$/; + +export type FlyJobMachineInput = { + runId: string; + operationId: string; + image: string; + region: string; + size: string; +}; + +/** + * Build the public, non-secret part of a run Machine. Work is started only + * after creation so the caller can deliver its payload through guest exec; + * provider restarts are disabled to avoid replaying a job after a crash. + */ +export function flyJobMachineConfig(app: string, input: FlyJobMachineInput): FlyMachineConfig { + if (!APP.test(app)) throw new TypeError("Invalid Fly app name"); + if (!UUID.test(input.runId) || !input.operationId || input.operationId.length > 200) throw new TypeError("Invalid Fly run identity"); + if (!IMAGE.test(input.image)) throw new TypeError("Invalid Fly execution image"); + if (!/^[a-z0-9]{3}$/.test(input.region)) throw new TypeError("Invalid Fly execution region"); + const size = input.size.match(/^(shared-cpu|performance)-(1|2|4)x$/); + if (!size) throw new TypeError("Invalid Fly execution size"); + const cpuCount = Number(size[2]); + + return { + image: input.image, + region: input.region, + guest: { cpu_kind: size[1] === "shared-cpu" ? "shared" : "performance", cpus: cpuCount, memory_mb: cpuCount * 1024 }, + metadata: { + factorize_role: "job", + factorize_run_id: input.runId.toLowerCase(), + factorize_operation_id: input.operationId, + }, + auto_start: false, + auto_stop: false, + restart: { policy: "no" }, + }; +} diff --git a/packages/api/src/fly-job-tunnel-helper.ts b/packages/api/src/fly-job-tunnel-helper.ts new file mode 100644 index 00000000..16c2f1ef --- /dev/null +++ b/packages/api/src/fly-job-tunnel-helper.ts @@ -0,0 +1,58 @@ +/** Standalone guest helper for the authenticated Fly-to-proxy TCP tunnel. */ +export const FACTORIZE_FLY_TUNNEL_HELPER = String.raw`import { readFile } from "node:fs/promises"; +import { connect as connectTls } from "node:tls"; +import { createServer } from "node:net"; + +const root = "/home/exedev/.factorize-job/proxy"; +const config = JSON.parse(await readFile(root + "/config.json", "utf8")); +const [key, cert, ca] = await Promise.all(["client.key", "client.crt", "ca.crt"].map(name => readFile(root + "/" + name))); +const tls = { key, cert, ca, rejectUnauthorized: true, servername: config.proxyHost }; +const server = createServer(local => { + local.pause(); + let closed = false; + local.once("close", () => { closed = true; }); + void (async () => { + let attempt = 0; + while (!closed && !local.destroyed) { + let remote; + try { + remote = connectTls({ ...tls, host: config.proxyHost, port: config.proxyPort }); + await new Promise((resolve, reject) => { + const timeout = setTimeout(() => remote.destroy(new Error("connect timeout")), 10000); + remote.once("secureConnect", () => { clearTimeout(timeout); resolve(); }); + remote.once("error", error => { clearTimeout(timeout); reject(error); }); + }); + remote.write("FACTORIZE-TUNNEL/3 " + config.runId + "\n"); + await new Promise((resolve, reject) => { + let response = ""; + const timeout = setTimeout(() => reject(new Error("tunnel timeout")), 10000); + const onData = chunk => { + response += chunk.toString("utf8"); + if (response.length > 128) { cleanup(); reject(new Error("invalid tunnel response")); } + else if (response.includes("\n")) { + cleanup(); + response === "FACTORIZE-TUNNEL/1 ACCEPTED\n" ? resolve() : reject(new Error("tunnel rejected")); + } + }; + const cleanup = () => { clearTimeout(timeout); remote.removeListener("data", onData); remote.removeListener("error", onError); }; + const onError = error => { cleanup(); reject(error); }; + remote.on("data", onData); + remote.once("error", onError); + }); + local.pipe(remote); + remote.pipe(local); + local.resume(); + remote.once("close", () => local.destroy()); + local.once("close", () => remote.destroy()); + return; + } catch { + remote?.destroy(); + if (closed || local.destroyed) return; + const delay = Math.min(30000, 1000 * (2 ** Math.min(attempt++, 5))); + await new Promise(resolve => setTimeout(resolve, delay)); + } + } + })(); +}); +server.listen(config.localPort, "127.0.0.1"); +`; diff --git a/packages/api/src/fly-machines-client.ts b/packages/api/src/fly-machines-client.ts index a7139e37..58fe5687 100644 --- a/packages/api/src/fly-machines-client.ts +++ b/packages/api/src/fly-machines-client.ts @@ -129,6 +129,11 @@ export function createFlyMachinesClient(options: FlyMachinesClientOptions) { } return { + /** Resolve a previously persisted provisional run handle after a scheduler restart. */ + findMachineByOperationId(app: string, operationId: string): Promise { + if (!operationId || operationId.length > 200) throw new TypeError("operationId must contain 1 to 200 characters"); + return findByOperationId(app, operationId); + }, async createMachine(app: string, config: FlyMachineConfig, operationId: string): Promise { if (!operationId || operationId.length > 200) throw new TypeError("operationId must contain 1 to 200 characters"); const existing = await findByOperationId(app, operationId); @@ -136,6 +141,10 @@ export function createFlyMachinesClient(options: FlyMachinesClientOptions) { const body = { config: { ...config, + // Job Machines are supervised by Factorize. A provider restart can + // replay a script after an ambiguous guest exit, duplicating side + // effects, so execution guests must stop with their process. + restart: { policy: "no" }, metadata: { ...config.metadata, [OPERATION_METADATA_KEY]: operationId }, }, }; diff --git a/packages/api/src/postgres/encrypted-integration-repository.ts b/packages/api/src/postgres/encrypted-integration-repository.ts index 9e02d3fa..a83dbdcb 100644 --- a/packages/api/src/postgres/encrypted-integration-repository.ts +++ b/packages/api/src/postgres/encrypted-integration-repository.ts @@ -77,6 +77,33 @@ export class EncryptedIntegrationRepository { return rows.length ? safeRevision(rows[0]!.revision) : 0; } + /** Recognize an exact retry before collection-level create checks run. */ + async replay(input: IntegrationWrite): Promise<{ revision: number; duplicate: true } | null> { + validate(input); + const { rows } = await this.database.pool.query("SELECT revision,operation,record_id,kind,name,origin,description,encrypted_payload FROM app.integration_changes WHERE tenant_id=$1 AND idempotency_key=$2", [this.tenantId, input.idempotencyKey]); + const row = rows[0]; + if (!row) return null; + const sameCiphertext = row.encrypted_payload === null + ? input.operation === "delete" && input.encryptedPayload === undefined + : input.operation === "upsert" && typeof input.encryptedPayload === "string" && await decrypt(row.encrypted_payload, this.serverEncryptionKey) === input.encryptedPayload; + if (row.record_id !== input.id || row.kind !== input.kind || row.name !== input.name || row.origin !== input.origin + || row.description !== input.description || row.operation !== input.operation || !sameCiphertext) { + throw new IntegrationIdempotencyConflictError(); + } + return { revision: safeRevision(row.revision), duplicate: true }; + } + + /** Recognize a delete retry after its active record has already become a tombstone. */ + async replayDelete(id: string, kind: EncryptedIntegrationKind, idempotencyKey: string): Promise<{ revision: number; duplicate: true } | null> { + const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + if (!uuid.test(id) || !uuid.test(idempotencyKey)) throw new IntegrationValidationError("id and idempotencyKey must be UUIDs"); + const { rows } = await this.database.pool.query("SELECT revision,operation,record_id,kind,name,origin,description,encrypted_payload FROM app.integration_changes WHERE tenant_id=$1 AND idempotency_key=$2", [this.tenantId, idempotencyKey]); + const row = rows[0]; + if (!row) return null; + if (row.record_id !== id || row.kind !== kind || row.operation !== "delete" || row.encrypted_payload !== null) throw new IntegrationIdempotencyConflictError(); + return { revision: safeRevision(row.revision), duplicate: true }; + } + async write(input: IntegrationWrite): Promise<{ revision: number; duplicate: boolean }> { validate(input); const encryptedPayload = input.operation === "upsert" ? await encrypt(input.encryptedPayload!, this.serverEncryptionKey) : null; @@ -126,6 +153,74 @@ export class EncryptedIntegrationRepository { return Promise.all(rows.map(async row => ({ id: row.id, kind: row.kind, name: row.name, origin: row.origin, description: row.description, revision: safeRevision(row.revision), encryptedPayload: await decrypt(row.encrypted_payload, this.serverEncryptionKey) }))); } + /** Return selected records including tombstones so an authenticated proxy can + * remove a deleted credential from already-established grants. Tombstones + * never include ciphertext. */ + async recordsIncludingDeleted(kind: EncryptedIntegrationKind): Promise> { + const { rows } = await this.database.pool.query<{ id: string; kind: EncryptedIntegrationKind; name: string; origin: string | null; description: string | null; revision: string | number; encrypted_payload: string | null; deleted_at: Date | null }>( + "SELECT id,kind,name,origin,description,revision,encrypted_payload,deleted_at FROM app.integration_records WHERE tenant_id=$1 AND kind=$2 ORDER BY revision,id", [this.tenantId, kind]); + return Promise.all(rows.map(async row => ({ id: row.id, kind: row.kind, name: row.name, origin: row.origin, description: row.description, + revision: safeRevision(row.revision), encryptedPayload: row.deleted_at || row.encrypted_payload === null ? null : await decrypt(row.encrypted_payload, this.serverEncryptionKey), deletedAt: row.deleted_at }))); + } + + /** Owner-only rekey input. Includes tombstones and the retained change feed so + * a browser rotation can preserve pending edits without exposing plaintext to + * the API process. */ + async rotationSnapshot(): Promise<{ revision: number; records: GenerationRecord[]; changes: GenerationChange[] }> { + const revision = await this.revision(); + const records = await this.database.pool.query<{ id: string; kind: EncryptedIntegrationKind; name: string; origin: string | null; description: string | null; revision: string | number; encrypted_payload: string | null; deleted_at: Date | null }>( + "SELECT id,kind,name,origin,description,revision,encrypted_payload,deleted_at FROM app.integration_records WHERE tenant_id=$1 ORDER BY id", [this.tenantId]); + const changes = await this.database.pool.query( + "SELECT revision,idempotency_key,operation,record_id,kind,name,origin,description,encrypted_payload FROM app.integration_changes WHERE tenant_id=$1 ORDER BY revision", [this.tenantId]); + return { + revision, + records: await Promise.all(records.rows.map(async row => ({ id: row.id, kind: row.kind, name: row.name, origin: row.origin, description: row.description, + revision: safeRevision(row.revision), encryptedPayload: row.deleted_at || row.encrypted_payload === null ? null : await decrypt(row.encrypted_payload, this.serverEncryptionKey), deletedAt: row.deleted_at }))), + changes: await Promise.all(changes.rows.map(async row => ({ id: row.record_id, kind: row.kind, name: row.name, origin: row.origin, description: row.description, + revision: safeRevision(row.revision), idempotencyKey: row.idempotency_key, operation: row.operation, + encryptedPayload: row.encrypted_payload === null ? null : await decrypt(row.encrypted_payload, this.serverEncryptionKey) }))), + }; + } + + /** + * Replace only a model account's user-encrypted Codex cache. The account + * revision is independent of the collection revision, so unrelated bearer + * edits do not invalidate the cache's authenticated-encryption binding. + */ + async saveModelCache(input: { id: string; expectedRevision: number; keyGeneration: string; encryptedPayload: string; idempotencyKey: string }): Promise<{ revision: number; collectionRevision: number; duplicate: boolean }> { + const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + if (!uuid.test(input.id) || !uuid.test(input.idempotencyKey) || !uuid.test(input.keyGeneration) || !Number.isSafeInteger(input.expectedRevision) || input.expectedRevision < 0 + || !isUserCiphertext(input.encryptedPayload)) throw new IntegrationValidationError("model cache binding is invalid"); + const encryptedPayload = await encrypt(input.encryptedPayload, this.serverEncryptionKey); + return this.database.transaction(async client => { + await client.query("INSERT INTO app.integration_collections(tenant_id) VALUES ($1) ON CONFLICT DO NOTHING", [this.tenantId]); + const collection = (await client.query<{ revision: string | number }>("SELECT revision FROM app.integration_collections WHERE tenant_id=$1 FOR UPDATE", [this.tenantId])).rows[0]!; + const currentCollectionRevision = safeRevision(collection.revision); + const keyState = (await client.query<{ generation_id: string }>("SELECT generation_id FROM app.unlock_key_verifications WHERE tenant_id=$1 FOR UPDATE", [this.tenantId])).rows[0]; + if (!keyState || keyState.generation_id !== input.keyGeneration) throw new StaleIntegrationRevisionError(currentCollectionRevision); + const current = (await client.query<{ kind: EncryptedIntegrationKind; name: string; origin: string | null; description: string | null; revision: string | number; encrypted_payload: string | null; deleted_at: Date | null }>( + "SELECT kind,name,origin,description,revision,encrypted_payload,deleted_at FROM app.integration_records WHERE tenant_id=$1 AND id=$2 FOR UPDATE", [this.tenantId, input.id])).rows[0]; + if (!current || current.kind !== "model" || current.deleted_at || current.revision === null) throw new IntegrationValidationError("model account is unavailable"); + const currentRevision = safeRevision(current.revision); + const previousChange = (await client.query<{ revision: string | number; record_id: string; operation: string; encrypted_payload: string | null }>( + "SELECT revision,record_id,operation,encrypted_payload FROM app.integration_changes WHERE tenant_id=$1 AND idempotency_key=$2", [this.tenantId,input.idempotencyKey])).rows[0]; + if (previousChange) { + const same = previousChange.record_id === input.id && previousChange.operation === "upsert" && previousChange.encrypted_payload !== null + && await decrypt(previousChange.encrypted_payload, this.serverEncryptionKey) === input.encryptedPayload; + if (!same) throw new IntegrationIdempotencyConflictError(); + return { revision: Math.max(0, input.expectedRevision + 1), collectionRevision: safeRevision(previousChange.revision), duplicate: true }; + } + if (currentRevision !== input.expectedRevision) throw new StaleIntegrationRevisionError(currentRevision); + const collectionRevision = currentCollectionRevision + 1; + const revision = currentRevision + 1; + await client.query("UPDATE app.integration_collections SET revision=$2,updated_at=now() WHERE tenant_id=$1", [this.tenantId, collectionRevision]); + await client.query("UPDATE app.integration_records SET revision=$3,encrypted_payload=$4,updated_at=now() WHERE tenant_id=$1 AND id=$2", [this.tenantId,input.id,revision,encryptedPayload]); + await client.query(`INSERT INTO app.integration_changes(tenant_id,revision,id,idempotency_key,operation,record_id,kind,name,origin,description,encrypted_payload) + VALUES ($1,$2,$3,$4,'upsert',$5,'model',$6,NULL,$7,$8)`, [this.tenantId,collectionRevision,crypto.randomUUID(),input.idempotencyKey,input.id,current.name,current.description,encryptedPayload]); + return { revision, collectionRevision, duplicate: false }; + }); + } + async changesSince(revision: number, limit = 100): Promise { if (!Number.isSafeInteger(revision) || revision < 0) throw new IntegrationValidationError("revision must be a non-negative safe integer"); if (!Number.isSafeInteger(limit) || limit < 1 || limit > 500) throw new IntegrationValidationError("limit must be between 1 and 500"); @@ -168,7 +263,7 @@ export class EncryptedIntegrationRepository { } /** Atomically archive and replace records, tombstones, changes and the collection revision. */ - async commitGeneration(generationId: string, expectedRevision: number): Promise<{ revision: number; duplicate: boolean }> { + async commitGeneration(generationId: string, expectedRevision: number, keyReplacement?: { expectedVerificationGenerationId: string; verificationGenerationId: string; encryptedVerification: string }): Promise<{ revision: number; duplicate: boolean }> { return this.database.transaction(async client => { await client.query("INSERT INTO app.integration_collections(tenant_id) VALUES($1) ON CONFLICT DO NOTHING", [this.tenantId]); const collection = (await client.query<{ revision: string | number }>("SELECT revision FROM app.integration_collections WHERE tenant_id=$1 FOR UPDATE", [this.tenantId])).rows[0]!; @@ -178,6 +273,19 @@ export class EncryptedIntegrationRepository { if (journal.state === "committed") return { revision: safeRevision(journal.committed_revision!), duplicate: true }; if (journal.state !== "staged") throw new IntegrationValidationError("staged generation is no longer available"); if (current !== expectedRevision) throw new StaleIntegrationGenerationError(current); + if (keyReplacement) { + const currentKey = (await client.query<{ generation_id: string }>("SELECT generation_id FROM app.unlock_key_verifications WHERE tenant_id=$1 FOR UPDATE", [this.tenantId])).rows[0]; + if (!currentKey || currentKey.generation_id !== keyReplacement.expectedVerificationGenerationId) throw new StaleIntegrationGenerationError(current); + if (!/^[0-9a-f-]{36}$/i.test(keyReplacement.verificationGenerationId) || keyReplacement.verificationGenerationId === currentKey.generation_id + || keyReplacement.encryptedVerification.length > 4096) throw new IntegrationValidationError("replacement key verification is invalid"); + let envelope: any; + try { envelope = JSON.parse(keyReplacement.encryptedVerification); } catch { throw new IntegrationValidationError("replacement key verification is malformed"); } + if (!envelope || typeof envelope !== "object" || Array.isArray(envelope) || envelope.version !== 1 || envelope.algorithm !== "A256GCM" + || Object.keys(envelope).sort().join(",") !== "algorithm,ciphertext,nonce,version" || typeof envelope.nonce !== "string" + || !/^[A-Za-z0-9_-]{16}$/u.test(envelope.nonce) || typeof envelope.ciphertext !== "string" || !/^[A-Za-z0-9_-]{22,}$/u.test(envelope.ciphertext)) { + throw new IntegrationValidationError("replacement key verification is malformed"); + } + } await client.query("DELETE FROM app.integration_generation_previous_records WHERE tenant_id=$1", [this.tenantId]); await client.query("DELETE FROM app.integration_generation_previous_changes WHERE tenant_id=$1", [this.tenantId]); await client.query(`INSERT INTO app.integration_generation_previous_records(tenant_id,id,kind,name,origin,description,revision,encrypted_payload,deleted_at,created_at,updated_at,generation_id) SELECT tenant_id,id,kind,name,origin,description,revision,encrypted_payload,deleted_at,created_at,updated_at,$2 FROM app.integration_records WHERE tenant_id=$1`, [this.tenantId,generationId]); @@ -191,6 +299,12 @@ export class EncryptedIntegrationRepository { const revision = current + 1; await client.query("UPDATE app.integration_collections SET revision=$2,updated_at=now() WHERE tenant_id=$1", [this.tenantId,revision]); await client.query("UPDATE app.integration_generation_journal SET state='committed',committed_revision=$3,committed_at=now() WHERE tenant_id=$1 AND generation_id=$2", [this.tenantId,generationId,revision]); + if (keyReplacement) { + await client.query("UPDATE app.unlock_key_verifications SET generation_id=$2,encrypted_verification=$3,created_at=now() WHERE tenant_id=$1", [this.tenantId,keyReplacement.verificationGenerationId,keyReplacement.encryptedVerification]); + await client.query("UPDATE app.credential_run_grants SET revoked_at=now() WHERE tenant_id=$1 AND revoked_at IS NULL", [this.tenantId]); + await client.query("UPDATE app.credential_run_enrollments SET consumed_at=now() WHERE tenant_id=$1 AND consumed_at IS NULL", [this.tenantId]); + await client.query("UPDATE app.credential_proxy_states SET unlocked=false,updated_at=now() WHERE tenant_id=$1", [this.tenantId]); + } return { revision, duplicate: false }; }); } diff --git a/packages/api/src/postgres/proxy-control-repository.ts b/packages/api/src/postgres/proxy-control-repository.ts new file mode 100644 index 00000000..84a7be38 --- /dev/null +++ b/packages/api/src/postgres/proxy-control-repository.ts @@ -0,0 +1,93 @@ +import type { Database } from "./database"; + +export type RegisteredProxy = { tenantId: string; proxyIdentity: string; bootEpoch: string; controlCredential: string }; +const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const digest = async (token: string) => new Uint8Array(await crypto.subtle.digest("SHA-256", new TextEncoder().encode(token))); + +/** Credentials for the Factorize-to-proxy control plane. These are never job run credentials. */ +export class ProxyControlRepository { + constructor(private database: Database) {} + + /** Called only from the authenticated infrastructure provisioning path; the token is returned once. */ + async provisionCredential(tenantId: string, proxyIdentity: string, bootEpoch: string): Promise { + if (!uuid.test(tenantId) || !proxyIdentity.trim() || proxyIdentity.length > 255 || !uuid.test(bootEpoch)) throw new TypeError("Invalid proxy identity or boot epoch"); + const token = `fzc_${Buffer.from(crypto.getRandomValues(new Uint8Array(32))).toString("base64url")}`; + await this.database.pool.query(`INSERT INTO app.credential_proxy_control_credentials(tenant_id,proxy_identity,boot_epoch,token_digest,revoked_at,consumed_at) + VALUES ($1,$2,$3,$4,NULL,NULL) ON CONFLICT (tenant_id,proxy_identity) DO UPDATE SET boot_epoch=excluded.boot_epoch,token_digest=excluded.token_digest,proxy_token_digest=NULL,revoked_at=NULL,consumed_at=NULL,created_at=now()`, + [tenantId, proxyIdentity, bootEpoch, Buffer.from(await digest(token))]); + return token; + } + + async authenticate(token: string): Promise | null> { + if (typeof token !== "string" || !/^fzc_[A-Za-z0-9_-]{43}$/.test(token)) return null; + const result = await this.database.pool.query<{ tenant_id: string; proxy_identity: string }>(`SELECT tenant_id,proxy_identity + FROM app.credential_proxy_control_credentials WHERE token_digest=$1 AND revoked_at IS NULL`, [Buffer.from(await digest(token))]); + const row = result.rows[0]; + return row ? { tenantId: row.tenant_id, proxyIdentity: row.proxy_identity } : null; + } + + async registerBoot(token: string, bootEpoch: string, browserOrigin: string): Promise { + if (!uuid.test(bootEpoch)) throw new TypeError("bootEpoch must be a UUID"); + const parsedOrigin = new URL(browserOrigin); + if (parsedOrigin.protocol !== "https:" || parsedOrigin.origin !== browserOrigin || parsedOrigin.username || parsedOrigin.password) throw new TypeError("browser origin must be canonical HTTPS"); + const tokenDigest = Buffer.from(await digest(token)); + return this.database.transaction(async client => { + const credential = await client.query<{ tenant_id: string; proxy_identity: string }>(`SELECT tenant_id,proxy_identity + FROM app.credential_proxy_control_credentials WHERE token_digest=$1 AND boot_epoch=$2 AND revoked_at IS NULL AND consumed_at IS NULL FOR UPDATE`, [tokenDigest, bootEpoch]); + const identity = credential.rows[0]; + if (!identity) return null; + const saved = await client.query(`INSERT INTO app.credential_proxy_states(tenant_id,proxy_identity,boot_epoch) + VALUES ($1,$2,$3) ON CONFLICT (tenant_id) DO UPDATE SET proxy_identity=excluded.proxy_identity,boot_epoch=excluded.boot_epoch,unlocked=false,updated_at=now() + WHERE app.credential_proxy_states.proxy_identity=excluded.proxy_identity + AND app.credential_proxy_states.disabled_at IS NULL AND app.credential_proxy_states.revoked_at IS NULL RETURNING tenant_id`, + [identity.tenant_id, identity.proxy_identity, bootEpoch]); + if (!saved.rowCount) return null; + const proxyToken = `fzc_${Buffer.from(crypto.getRandomValues(new Uint8Array(32))).toString("base64url")}`; + await client.query(`UPDATE app.credential_proxy_control_credentials SET consumed_at=now(),proxy_token_digest=$3 + WHERE token_digest=$1 AND boot_epoch=$2 AND consumed_at IS NULL`, [tokenDigest, bootEpoch, Buffer.from(await digest(proxyToken))]); + await client.query(`UPDATE app.credential_run_grants SET revoked_at=now() + WHERE tenant_id=$1 AND proxy_identity=$2 AND boot_epoch<>$3 AND revoked_at IS NULL`, [identity.tenant_id, identity.proxy_identity, bootEpoch]); + await client.query(`INSERT INTO app.credential_proxy_browser_pairings(tenant_id,proxy_identity,boot_epoch,browser_origin) + VALUES ($1,$2,$3,$4) ON CONFLICT (tenant_id) DO UPDATE SET proxy_identity=excluded.proxy_identity, + boot_epoch=excluded.boot_epoch,browser_origin=excluded.browser_origin,updated_at=now()`, + [identity.tenant_id, identity.proxy_identity, bootEpoch, browserOrigin]); + return { tenantId: identity.tenant_id, proxyIdentity: identity.proxy_identity, bootEpoch, controlCredential: proxyToken }; + }); + } + + async browserPairing(tenantId: string): Promise<{ proxyIdentity: string; bootEpoch: string; browserOrigin: string } | null> { + const result = await this.database.pool.query<{ proxy_identity: string; boot_epoch: string; browser_origin: string }>( + `SELECT b.proxy_identity,b.boot_epoch,b.browser_origin FROM app.credential_proxy_browser_pairings b + JOIN app.credential_proxy_states p ON p.tenant_id=b.tenant_id AND p.proxy_identity=b.proxy_identity AND p.boot_epoch=b.boot_epoch + WHERE b.tenant_id=$1 AND p.disabled_at IS NULL AND p.revoked_at IS NULL`, [tenantId]); + const row = result.rows[0]; + return row ? { proxyIdentity: row.proxy_identity, bootEpoch: row.boot_epoch, browserOrigin: row.browser_origin } : null; + } + + /** Authenticates the proxy's memory-held control credential against its current registered boot. */ + async authenticateProxy(token: string): Promise | null> { + if (typeof token !== "string" || !/^fzc_[A-Za-z0-9_-]{43}$/.test(token)) return null; + const result = await this.database.pool.query<{ tenant_id: string; proxy_identity: string; boot_epoch: string }>(`SELECT c.tenant_id,c.proxy_identity,c.boot_epoch + FROM app.credential_proxy_control_credentials c JOIN app.credential_proxy_states p + ON p.tenant_id=c.tenant_id AND p.proxy_identity=c.proxy_identity AND p.boot_epoch=c.boot_epoch + WHERE c.proxy_token_digest=$1 AND c.revoked_at IS NULL AND p.disabled_at IS NULL AND p.revoked_at IS NULL`, [Buffer.from(await digest(token))]); + const row = result.rows[0]; + return row ? { tenantId: row.tenant_id, proxyIdentity: row.proxy_identity, bootEpoch: row.boot_epoch } : null; + } + + async reportRuntimeStatus(token: string, bootEpoch: string, unlocked: boolean): Promise { + if (!uuid.test(bootEpoch) || typeof unlocked !== "boolean") throw new TypeError("Invalid proxy runtime status"); + const identity = await this.authenticateProxy(token); + if (!identity || identity.bootEpoch !== bootEpoch) return false; + const result = await this.database.pool.query(`UPDATE app.credential_proxy_states SET unlocked=$3,updated_at=now() + WHERE tenant_id=$1 AND proxy_identity=$2 AND boot_epoch=$4 AND disabled_at IS NULL AND revoked_at IS NULL`, + [identity.tenantId, identity.proxyIdentity, unlocked, bootEpoch]); + return Boolean(result.rowCount); + } + + async revokeCredential(tenantId: string, proxyIdentity: string): Promise { + const result = await this.database.pool.query(`UPDATE app.credential_proxy_control_credentials SET revoked_at=now() + WHERE tenant_id=$1 AND proxy_identity=$2 AND revoked_at IS NULL`, [tenantId, proxyIdentity]); + return Boolean(result.rowCount); + } +} diff --git a/packages/api/src/postgres/run-enrollment-repository.ts b/packages/api/src/postgres/run-enrollment-repository.ts index 328ad692..3bc00bd0 100644 --- a/packages/api/src/postgres/run-enrollment-repository.ts +++ b/packages/api/src/postgres/run-enrollment-repository.ts @@ -30,26 +30,38 @@ export class RunEnrollmentRepository { if (!input.proxyIdentity.trim() || input.proxyIdentity.length > 255) throw new TypeError("proxyIdentity must contain 1 to 255 characters"); await this.database.pool.query(`INSERT INTO app.credential_proxy_states(tenant_id,proxy_identity,boot_epoch,disabled_at,revoked_at) VALUES ($1,$2,$3,CASE WHEN $4 THEN now() END,CASE WHEN $5 THEN now() END) - ON CONFLICT (tenant_id) DO UPDATE SET proxy_identity=excluded.proxy_identity,boot_epoch=excluded.boot_epoch,disabled_at=excluded.disabled_at,revoked_at=excluded.revoked_at,updated_at=now()`, + ON CONFLICT (tenant_id) DO UPDATE SET proxy_identity=excluded.proxy_identity,boot_epoch=excluded.boot_epoch,disabled_at=excluded.disabled_at,revoked_at=excluded.revoked_at,unlocked=false,updated_at=now()`, [input.tenantId, input.proxyIdentity, input.bootEpoch, input.disabled ?? false, input.revoked ?? false]); } - async authorizeRun(binding: RunEnrollmentBinding, grantRevision: number): Promise { + /** + * Authorize an active run for the current provider binding. The revision is + * stable for repeated reconnects on the same VM/proxy boot, and advances + * when the run is rebound to a new VM or proxy boot. A revoked grant cannot + * be restored on the same binding. Locking the run and proxy state + * serializes first enrollment and reconnect races. + */ + async authorizeRun(binding: RunEnrollmentBinding): Promise { validateBinding(binding); - if (!Number.isSafeInteger(grantRevision) || grantRevision < 1) throw new TypeError("grantRevision must be a positive safe integer"); return this.database.transaction(async client => { const valid = await client.query(`SELECT 1 FROM app.job_runs r JOIN app.credential_proxy_states p ON p.tenant_id=r.tenant_id - WHERE r.tenant_id=$1 AND r.id=$2 AND r.state IN ${activeStates} AND p.proxy_identity=$3 AND p.boot_epoch=$4 AND p.disabled_at IS NULL AND p.revoked_at IS NULL`, + WHERE r.tenant_id=$1 AND r.id=$2 AND r.state IN ${activeStates} AND p.proxy_identity=$3 AND p.boot_epoch=$4 AND p.disabled_at IS NULL AND p.revoked_at IS NULL + FOR UPDATE OF r,p`, [binding.tenantId, binding.runId, binding.proxyIdentity, binding.bootEpoch]); - if (!valid.rowCount) return false; - const saved = await client.query(`INSERT INTO app.credential_run_grants(tenant_id,run_id,vm_identity,proxy_identity,boot_epoch,grant_revision,revoked_at) - VALUES ($1,$2,$3,$4,$5,$6,NULL) ON CONFLICT (tenant_id,run_id) DO UPDATE SET vm_identity=excluded.vm_identity,proxy_identity=excluded.proxy_identity,boot_epoch=excluded.boot_epoch,grant_revision=excluded.grant_revision,revoked_at=NULL - WHERE app.credential_run_grants.grant_revision < excluded.grant_revision RETURNING grant_revision`, + if (!valid.rowCount) return null; + const current = await client.query<{ vm_identity: string; proxy_identity: string; boot_epoch: string; grant_revision: string | number; revoked_at: Date | null }>( + `SELECT vm_identity,proxy_identity,boot_epoch,grant_revision,revoked_at FROM app.credential_run_grants + WHERE tenant_id=$1 AND run_id=$2 FOR UPDATE`, [binding.tenantId, binding.runId]); + const row = current.rows[0]; + if (row && row.vm_identity === binding.vmIdentity && row.proxy_identity === binding.proxyIdentity + && row.boot_epoch === binding.bootEpoch) return row.revoked_at === null ? Number(row.grant_revision) : null; + const grantRevision = row ? Number(row.grant_revision) + 1 : 1; + const saved = await client.query<{ grant_revision: string | number }>(`INSERT INTO app.credential_run_grants(tenant_id,run_id,vm_identity,proxy_identity,boot_epoch,grant_revision,revoked_at) + VALUES ($1,$2,$3,$4,$5,$6,NULL) ON CONFLICT (tenant_id,run_id) DO UPDATE + SET vm_identity=excluded.vm_identity,proxy_identity=excluded.proxy_identity,boot_epoch=excluded.boot_epoch,grant_revision=excluded.grant_revision,revoked_at=NULL + RETURNING grant_revision`, [binding.tenantId, binding.runId, binding.vmIdentity, binding.proxyIdentity, binding.bootEpoch, grantRevision]); - if (saved.rowCount) return true; - const current = await client.query(`SELECT 1 FROM app.credential_run_grants WHERE tenant_id=$1 AND run_id=$2 AND vm_identity=$3 AND proxy_identity=$4 AND boot_epoch=$5 AND grant_revision=$6 AND revoked_at IS NULL`, - [binding.tenantId, binding.runId, binding.vmIdentity, binding.proxyIdentity, binding.bootEpoch, grantRevision]); - return Boolean(current.rowCount); + return saved.rows[0] ? Number(saved.rows[0].grant_revision) : null; }); } diff --git a/packages/api/src/postgres/run-repository.ts b/packages/api/src/postgres/run-repository.ts index 3e23cd77..a1c28cda 100644 --- a/packages/api/src/postgres/run-repository.ts +++ b/packages/api/src/postgres/run-repository.ts @@ -51,10 +51,37 @@ export class RunRepository { async dueForPoll(limit = 40): Promise { const result = await this.database.pool.query(`SELECT r.tenant_id,r.id,r.job_id,r.invocation_id,r.state,r.encrypted_prompt,x.execution_config,x.trace_sources,x.execution_handle,x.execution_backend_kind,x.execution_capabilities,x.execution_diagnostics,x.failure_code,x.failure_reason,x.vm_cleanup_attempt,x.artifact_state,x.result_protocol_version,r.created_at,r.updated_at FROM app.job_runs r JOIN app.jobs j ON j.tenant_id=r.tenant_id AND j.id=r.job_id JOIN app.runs x ON x.tenant_id=r.tenant_id AND x.id=r.id - WHERE r.state IN ('running','blocked','stopping') AND (r.next_poll_at IS NULL OR r.next_poll_at<=now()) ORDER BY r.next_poll_at NULLS FIRST,r.updated_at LIMIT $1`, [limit]); + WHERE r.state IN ('running','blocked','stopping') AND x.execution_handle IS NOT NULL AND (r.next_poll_at IS NULL OR r.next_poll_at<=now()) ORDER BY r.next_poll_at NULLS FIRST,r.updated_at LIMIT $1`, [limit]); return result.rows.map(mapped); } + /** Persist a pre-provisioning gate without creating a VM or releasing the run's queue slot. */ + async blockLaunch(run: PersistedRun, reason: string): Promise { + const safeReason = reason.slice(0, 500); + await this.database.transaction(async client => { + const changed = await client.query(`UPDATE app.job_runs SET state='blocked',launch_lease_expires_at=NULL,next_poll_at=NULL,updated_at=now() + WHERE tenant_id=$1 AND id=$2 AND state='starting' RETURNING id`, [run.tenantId, run.id]); + if (!changed.rowCount) return; + await client.query(`UPDATE app.runs SET state='blocked',failure_code='proxy_launch_blocked',failure_reason=$3,updated_at=now() + WHERE tenant_id=$1 AND id=$2 AND execution_handle IS NULL`, [run.tenantId, run.id, safeReason]); + await client.query("INSERT INTO app.run_activity(tenant_id,id,run_id,action,detail) VALUES ($1,$2,$3,'launch_blocked',$4)", [run.tenantId, crypto.randomUUID(), run.id, safeReason]); + }); + } + + /** A blocked pre-launch run resumes only after an explicit owner retry. */ + async retryBlockedLaunch(tenantId: string, runId: string): Promise { + return this.database.transaction(async client => { + const changed = await client.query(`UPDATE app.job_runs r SET state='reserved',launch_lease_expires_at=NULL,next_poll_at=NULL,updated_at=now() + FROM app.runs x WHERE r.tenant_id=$1 AND r.id=$2 AND r.tenant_id=x.tenant_id AND r.id=x.id + AND r.state='blocked' AND x.execution_handle IS NULL AND x.failure_code='proxy_launch_blocked' RETURNING r.id`, [tenantId, runId]); + if (!changed.rowCount) return false; + await client.query("UPDATE app.runs SET state='reserved',failure_source=NULL,failure_code=NULL,failure_reason=NULL,updated_at=now() WHERE tenant_id=$1 AND id=$2", [tenantId, runId]); + await client.query("INSERT INTO app.run_activity(tenant_id,id,run_id,action,detail) VALUES ($1,$2,$3,'launch_retry_requested','Owner requested a manual retry')", [tenantId, crypto.randomUUID(), runId]); + await client.query("INSERT INTO app.wake_hints(key,not_before) VALUES ('global',now()) ON CONFLICT (key) DO UPDATE SET not_before=least(app.wake_hints.not_before,excluded.not_before),updated_at=now()"); + return true; + }); + } + /** Claim a bounded batch of terminal records whose provider cleanup is incomplete. */ async claimProviderCleanup(limit = 20): Promise; executionHandle: RunHandle | null; backendKind: string; startedAt: Date | null; attempt: number }>> { return this.database.transaction(async client => { diff --git a/packages/api/src/postgres/run-scheduler.ts b/packages/api/src/postgres/run-scheduler.ts index b2df5c47..23cadfe9 100644 --- a/packages/api/src/postgres/run-scheduler.ts +++ b/packages/api/src/postgres/run-scheduler.ts @@ -22,6 +22,12 @@ import { TraceProjectionRepository, TRACE_PARSER_VERSION } from "./trace-project import { LiveTraceRepository } from "./live-trace-repository"; import { JOB_RESULT_INSTRUCTIONS, parseJobResult } from "../job-result"; import { issueRunCredential, runProxyInstructions } from "../run-credentials"; +import { ProxyLaunchBlockedError, proxyLaunchDecision, requireProxyLaunch } from "../proxy-launch-policy"; +import { FlyConnectionRepository } from "./fly-connection-repository"; +import { FlyExecutionConfigurationRepository } from "./fly-execution-configuration-repository"; +import { createFlyMachinesClient } from "../fly-machines-client"; +import { FlyJobExecutionBackend, flyJobGuest } from "../fly-job-execution-backend"; +import { createFlyCredentialProxyGuest } from "../fly-credential-proxy-provisioner"; export class RunScheduler { private runs: RunRepository; @@ -31,7 +37,12 @@ export class RunScheduler { await this.runs.consumeWakeHint(); await new AutomationScheduler(this.database, this.env).process(); for (const run of await this.runs.dueForPoll()) await this.poll(run).catch(error => this.fail(run, error, "poll")); - for (let count = 0; count < 20; count++) { const run = await this.runs.claimNext(); if (!run) break; await this.launch(run).catch(error => this.fail(run, error, "launch")); } + for (let count = 0; count < 20; count++) { + const run = await this.runs.claimNext(); if (!run) break; + await this.launch(run).catch(error => error instanceof ProxyLaunchBlockedError + ? this.runs.blockLaunch(run, `Credential proxy ${error.reason}; resolve the prerequisite and manually retry this run.`) + : this.fail(run, error, "launch")); + } const retention = new RunRetentionRepository(this.database); await this.reconcileProviderCleanup(); await this.runRetentionMaintenance(retention); @@ -113,6 +124,29 @@ export class RunScheduler { } private async backend(run: PersistedRun): Promise { const id = String(run.executionConfig.connectionId ?? ""); + if (run.backendKind === "fly" || run.executionConfig.executionProvider === "fly") { + const [connection, configuration] = await Promise.all([ + new FlyConnectionRepository(this.database, run.tenantId, this.env.CREDENTIAL_ENCRYPTION_KEY).token(), + new FlyExecutionConfigurationRepository(this.database, run.tenantId).get(id), + ]); + if (!connection) throw new Error("Fly connection is unavailable"); + if (!configuration) throw new Error("Fly execution configuration is unavailable"); + const { rows } = await this.database.pool.query<{ proxy_identity: string; boot_epoch: string }>( + "SELECT proxy_identity, boot_epoch FROM app.credential_proxy_states WHERE tenant_id=$1 AND disabled_at IS NULL AND revoked_at IS NULL", + [run.tenantId], + ); + const proxy = /^fly:([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])):([^:]+)$/.exec(rows[0]?.proxy_identity ?? ""); + if (!proxy) throw new Error("A registered Fly credential proxy is unavailable for this tenant"); + const app = proxy[1]!; + const machines = createFlyMachinesClient({ token: connection }); + return new FlyJobExecutionBackend({ + app, image: configuration.image, region: configuration.region, size: configuration.size, + machines, guest: machine => flyJobGuest({ app, machine: machine.id, token: connection }), + proxyIdentity: rows[0]!.proxy_identity, + proxyBootEpoch: rows[0]!.boot_epoch, + proxyGuest: createFlyCredentialProxyGuest({ app, machine: proxy[2]!, token: connection }), + }); + } const kind = run.executionConfig.backendKind ?? (run.executionConfig.agentKind === "Amp" ? "amp" : "exe-vm"); if (kind === "amp") { const connection = await this.connections(run).get(`amp:${id}`); if (!connection) throw new Error("Amp connection is unavailable"); return new AmpBackend(connection); } if (kind !== "exe-vm") throw new Error(`Unsupported execution backend: ${String(kind)}`); @@ -121,6 +155,15 @@ export class RunScheduler { } private async launch(run: PersistedRun) { + const config = run.executionConfig; + const decision = proxyLaunchDecision({ + executionProvider: config.executionProvider === "fly" ? "fly" : "exe", + executionType: config.executionType === "code" ? "code" : "agent", + portableIntegrationIds: config.portableIntegrationIds, + modelAccess: config.modelAccess, + modelAccountId: config.modelAccountId, + }, await this.proxyLaunchState(run.tenantId)); + requireProxyLaunch(decision); if (run.executionConfig.executionType === "code") { if (typeof run.executionConfig.script !== "string" || !run.executionConfig.script.trim()) throw new Error("The code job has no saved script"); const backend = await this.backend(run), context = run.executionConfig.inputContext; @@ -146,6 +189,14 @@ export class RunScheduler { if (launched.observation.state === "failed") await this.runs.recordObservation(run, launched.observation); } + private async proxyLaunchState(tenantId: string) { + const result = await this.database.pool.query<{ enabled: boolean; unlocked: boolean }>(`SELECT disabled_at IS NULL AND revoked_at IS NULL AS enabled, + unlocked AND updated_at > now() - interval '30 seconds' AS unlocked + FROM app.credential_proxy_states WHERE tenant_id=$1`, [tenantId]); + const state = result.rows[0]; + return { enabled: Boolean(state?.enabled), running: Boolean(state), unlocked: Boolean(state?.unlocked), pendingChanges: false }; + } + private async poll(run: PersistedRun) { if (!run.executionHandle && run.state === "stopping") return this.runs.schedulePoll(run, "stopping"); const backend = await this.backend(run); if (!run.executionHandle) throw new Error("Execution handle is missing"); diff --git a/packages/api/src/postgres/schema.ts b/packages/api/src/postgres/schema.ts index 554df98f..880433c4 100644 --- a/packages/api/src/postgres/schema.ts +++ b/packages/api/src/postgres/schema.ts @@ -51,8 +51,14 @@ export const integrationChanges = app.table("integration_changes", { tenantId: uuid("tenant_id").notNull().references(() => integrationCollections.tenantId, { onDelete: "cascade" }), revision: bigint("revision", { mode: "number" }).notNull(), id: uuid("id").notNull(), idempotencyKey: uuid("idempotency_key").notNull(), operation: text("operation").$type<"upsert" | "delete">().notNull(), recordId: uuid("record_id").notNull(), kind: text("kind").$type<"bearer" | "model">().notNull(), name: text("name").notNull(), origin: text("origin"), description: text("description"), encryptedPayload: text("encrypted_payload"), createdAt: createdAt(), }, t => [primaryKey({ columns: [t.tenantId, t.revision] }), unique().on(t.tenantId, t.idempotencyKey), index("integration_changes_page").on(t.tenantId, t.revision), check("integration_changes_operation_check", sql`${t.operation} IN ('upsert','delete')`), check("integration_changes_kind_check", sql`${t.kind} IN ('bearer','model')`), check("integration_changes_origin_check", sql`(${t.kind} = 'bearer' AND ${t.origin} IS NOT NULL) OR (${t.kind} = 'model' AND ${t.origin} IS NULL)`), check("integration_changes_payload_check", sql`(${t.operation} = 'upsert' AND ${t.encryptedPayload} IS NOT NULL) OR (${t.operation} = 'delete' AND ${t.encryptedPayload} IS NULL)`)]); export const credentialProxyStates = app.table("credential_proxy_states", { - tenantId: uuid("tenant_id").primaryKey().references(() => tenants.id, { onDelete: "cascade" }), proxyIdentity: text("proxy_identity").notNull(), bootEpoch: uuid("boot_epoch").notNull(), disabledAt: timestamp("disabled_at", { withTimezone: true }), revokedAt: timestamp("revoked_at", { withTimezone: true }), updatedAt: updatedAt(), + tenantId: uuid("tenant_id").primaryKey().references(() => tenants.id, { onDelete: "cascade" }), proxyIdentity: text("proxy_identity").notNull(), bootEpoch: uuid("boot_epoch").notNull(), disabledAt: timestamp("disabled_at", { withTimezone: true }), revokedAt: timestamp("revoked_at", { withTimezone: true }), unlocked: boolean("unlocked").notNull().default(false), updatedAt: updatedAt(), }); +export const credentialProxyBrowserPairings = app.table("credential_proxy_browser_pairings", { + tenantId: uuid("tenant_id").primaryKey().references(() => tenants.id, { onDelete: "cascade" }), + proxyIdentity: text("proxy_identity").notNull(), bootEpoch: uuid("boot_epoch").notNull(), + browserOrigin: text("browser_origin").notNull(), updatedAt: updatedAt(), +}); +export const credentialProxyControlCredentials = app.table("credential_proxy_control_credentials", { tenantId: uuid("tenant_id").notNull().references(() => tenants.id, { onDelete: "cascade" }), proxyIdentity: text("proxy_identity").notNull(), bootEpoch: uuid("boot_epoch").notNull(), tokenDigest: bytea("token_digest").notNull().unique(), proxyTokenDigest: bytea("proxy_token_digest").unique(), revokedAt: timestamp("revoked_at", { withTimezone: true }), consumedAt: timestamp("consumed_at", { withTimezone: true }), createdAt: createdAt() }, t => [primaryKey({ columns: [t.tenantId, t.proxyIdentity] })]); export const credentialRunGrants = app.table("credential_run_grants", { tenantId: uuid("tenant_id").notNull(), runId: uuid("run_id").notNull(), vmIdentity: text("vm_identity").notNull(), proxyIdentity: text("proxy_identity").notNull(), bootEpoch: uuid("boot_epoch").notNull(), grantRevision: bigint("grant_revision", { mode: "number" }).notNull(), revokedAt: timestamp("revoked_at", { withTimezone: true }), createdAt: createdAt(), }, t => [primaryKey({ columns: [t.tenantId, t.runId] }), foreignKey({ columns: [t.tenantId, t.runId], foreignColumns: [jobRuns.tenantId, jobRuns.id] }).onDelete("cascade"), check("credential_run_grants_revision_check", sql`${t.grantRevision} > 0`)]); @@ -162,7 +168,7 @@ export const runRelations = relations(runs, ({ one, many }) => ({ export const databaseSchema = { schemaMigrations, tenants, oauthConnections, oauthConsentPreviews, authUsers, authAccounts, authResetTokens, authSessions, oauthDeviceAuthorizations, members, accessTokens, - connections, integrationCollections, integrationRecords, integrationChanges, credentialProxyStates, credentialRunGrants, credentialRunEnrollments, linearWorkspaces, githubInstallations, githubSetupStates, projects, jobs, vmImageVerifications, skills, jobSkills, triggers, webhookConditionsCutover, invocations, manualInvocationPermissions, jobRuns, runCredentials, runs, runArtifacts, runRetentionCleanup, + connections, integrationCollections, integrationRecords, integrationChanges, credentialProxyStates, credentialProxyControlCredentials, credentialRunGrants, credentialRunEnrollments, linearWorkspaces, githubInstallations, githubSetupStates, projects, jobs, vmImageVerifications, skills, jobSkills, triggers, webhookConditionsCutover, invocations, manualInvocationPermissions, jobRuns, runCredentials, runs, runArtifacts, runRetentionCleanup, runTraceEvents, runActivity, activeClaims, scheduleState, automaticWakes, lifecycleDeliveries, lifecycleEvents, jobEditDeliveries, jobEvents, webhookDeliveries, webhookDeliveryEvents, pendingVerifications, tailFingerprints, wakeHints, runTraceCursors, runTraceChunks, runTraceProjections, traceReplayLimits, traceReplayOperations, diff --git a/packages/api/src/proxy-browser-capability.ts b/packages/api/src/proxy-browser-capability.ts new file mode 100644 index 00000000..fcb3ed9e --- /dev/null +++ b/packages/api/src/proxy-browser-capability.ts @@ -0,0 +1,38 @@ +const encoder = new TextEncoder(); +const base64url = (bytes: Uint8Array) => btoa(String.fromCharCode(...bytes)).replace(/=/g, "").replace(/\+/g, "-").replace(/\//g, "_"); + +function decodePem(pem: string): Uint8Array { + const body = pem.replace(/-----BEGIN PRIVATE KEY-----|-----END PRIVATE KEY-----|\s/g, ""); + if (!/^[A-Za-z0-9+/]+={0,2}$/.test(body)) throw new TypeError("invalid browser capability signing key"); + const binary = atob(body); + return Uint8Array.from(binary, character => character.charCodeAt(0)); +} + +/** Issue a one-use capability accepted only by the paired proxy listener. */ +export async function issueModelAccountCapability(input: { + privateKeyPem: string; + proxyIdentity: string; + ownerId: string; + accountId: string; + action: "authorize" | "status" | "cancel"; + appOrigin: string; + generation: number | string; + now?: number; + jti?: string; +}): Promise { + const now = input.now ?? Math.floor(Date.now() / 1000); + const header = base64url(encoder.encode(JSON.stringify({ alg: "EdDSA", typ: "JWT" }))); + const claims = base64url(encoder.encode(JSON.stringify({ + iss: "factorize", aud: input.proxyIdentity, sub: input.ownerId, + action: `model-account:${input.action}`, accountId: input.accountId, + origin: input.appOrigin, generation: input.generation, exp: now + 60, + jti: input.jti ?? base64url(crypto.getRandomValues(new Uint8Array(24))), + }))); + const payload = `${header}.${claims}`; + const rawKey = decodePem(input.privateKeyPem); + const keyData = new ArrayBuffer(rawKey.byteLength); + new Uint8Array(keyData).set(rawKey); + const key = await crypto.subtle.importKey("pkcs8", keyData, { name: "Ed25519" }, false, ["sign"]); + const signature = await crypto.subtle.sign("Ed25519", key, encoder.encode(payload)); + return `${payload}.${base64url(new Uint8Array(signature))}`; +} diff --git a/packages/api/src/proxy-control.ts b/packages/api/src/proxy-control.ts new file mode 100644 index 00000000..2b382b11 --- /dev/null +++ b/packages/api/src/proxy-control.ts @@ -0,0 +1,168 @@ +import { databaseFor } from "./postgres/database"; +import { ProxyControlRepository } from "./postgres/proxy-control-repository"; +import { RunEnrollmentRepository } from "./postgres/run-enrollment-repository"; +import { EncryptedIntegrationRepository } from "./postgres/encrypted-integration-repository"; +import { authenticateRunCredential } from "./run-credentials"; + +const json = (body: unknown, status = 200) => Response.json(body, { status, headers: { "Cache-Control": "no-store" } }); + +/** Dedicated service-to-service endpoint. The run API credential is intentionally not accepted here. */ +export async function proxyControlFetch(request: Request, env: Parameters[0]): Promise { + const url = new URL(request.url); + if (!["/internal/credential-proxy/register-boot", "/internal/credential-proxy/status", "/internal/credential-proxy/enroll", "/internal/credential-proxy/consume-enrollment", "/internal/credential-proxy/integration-snapshot", "/internal/credential-proxy/model-account-cache", "/internal/credential-proxy/revocations"].includes(url.pathname)) return json({ error: "not_found" }, 404); + if (request.method !== "POST") return json({ error: "method_not_allowed" }, 405); + if (request.headers.get("content-type")?.split(";", 1)[0]?.trim().toLowerCase() !== "application/json") return json({ error: "unsupported_media_type" }, 415); + const authorization = request.headers.get("authorization") ?? ""; + const token = authorization.startsWith("Bearer ") ? authorization.slice(7) : ""; + if (!/^fzc_[A-Za-z0-9_-]{43}$/.test(token)) return json({ error: "unauthorized" }, 401); + const declaredLength = Number(request.headers.get("content-length") ?? "0"); + const maxBodyBytes = url.pathname === "/internal/credential-proxy/model-account-cache" ? 70_000 : 1024; + if (declaredLength > maxBodyBytes) return json({ error: "invalid_request" }, 413); + let body: unknown; + try { + const reader = request.body?.getReader(); + const chunks: Uint8Array[] = []; let length = 0; + if (reader) for (;;) { + const { done, value } = await reader.read(); if (done) break; + length += value.byteLength; + if (length > maxBodyBytes) { await reader.cancel(); return json({ error: "invalid_request" }, 413); } + chunks.push(value); + } + const bytes = new Uint8Array(length); let offset = 0; + for (const chunk of chunks) { bytes.set(chunk, offset); offset += chunk.byteLength; } + body = JSON.parse(new TextDecoder().decode(bytes)); + } catch { return json({ error: "invalid_request" }, 400); } + const repository = new ProxyControlRepository(databaseFor(env)); + if (url.pathname === "/internal/credential-proxy/register-boot") { + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).sort().join(",") !== "bootEpoch,browserOrigin" || typeof (body as any).bootEpoch !== "string" || typeof (body as any).browserOrigin !== "string") return json({ error: "invalid_request" }, 400); + const registration = await repository.registerBoot(token, (body as any).bootEpoch, (body as any).browserOrigin); + if (!registration) return json({ error: "proxy_registration_unavailable" }, 403); + return json({ registered: true, tenantId: registration.tenantId, proxyIdentity: registration.proxyIdentity, bootEpoch: registration.bootEpoch, controlCredential: registration.controlCredential }); + } + + if (url.pathname === "/internal/credential-proxy/status") { + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).sort().join(",") !== "bootEpoch,unlocked" + || typeof (body as any).bootEpoch !== "string" || typeof (body as any).unlocked !== "boolean") return json({ error: "invalid_request" }, 400); + const updated = await repository.reportRuntimeStatus(token, (body as any).bootEpoch, (body as any).unlocked); + return updated ? json({ updated: true }) : json({ error: "proxy_status_unavailable" }, 403); + } + + const proxy = await repository.authenticateProxy(token); + if (!proxy) return json({ error: "unauthorized" }, 401); + if (url.pathname === "/internal/credential-proxy/revocations") { + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).some(key => key !== "bootEpoch") + || typeof (body as any).bootEpoch !== "string" || (body as any).bootEpoch !== proxy.bootEpoch) return json({ error: "invalid_request" }, 400); + const revoked = await databaseFor(env).pool.query<{ run_id: string }>(`SELECT run_id FROM app.credential_run_grants + WHERE tenant_id=$1 AND proxy_identity=$2 AND boot_epoch=$3 AND revoked_at IS NOT NULL + UNION + SELECT c.run_id FROM app.run_credentials c JOIN app.job_runs r ON r.tenant_id=c.tenant_id AND r.id=c.run_id + WHERE c.tenant_id=$1 AND r.state IN ('succeeded','failed','stopped') ORDER BY run_id`, + [proxy.tenantId, proxy.proxyIdentity, proxy.bootEpoch]); + const state = await databaseFor(env).pool.query<{ unlocked: boolean }>("SELECT unlocked FROM app.credential_proxy_states WHERE tenant_id=$1 AND proxy_identity=$2 AND boot_epoch=$3", [proxy.tenantId, proxy.proxyIdentity, proxy.bootEpoch]); + return json({ revokedRunIds: revoked.rows.map(row => row.run_id), lockProxy: !state.rows[0]?.unlocked }); + } + if (url.pathname === "/internal/credential-proxy/model-account-cache") { + if (!body || typeof body !== "object" || Array.isArray(body)) return json({ error: "invalid_request" }, 400); + const input = body as Record; + const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + if (input.operation === "list" && Object.keys(input).length === 1) { + const accounts = await new EncryptedIntegrationRepository(databaseFor(env), proxy.tenantId, env.CREDENTIAL_ENCRYPTION_KEY).records("model"); + const keyState = await databaseFor(env).pool.query<{ generation_id: string }>("SELECT generation_id FROM app.unlock_key_verifications WHERE tenant_id=$1", [proxy.tenantId]); + if (!keyState.rows[0]) return json({ error: "model_account_unavailable" }, 404); + return json({ ownerId: proxy.tenantId, keyGeneration: keyState.rows[0].generation_id, + accounts: accounts.map(({ id, revision }) => ({ id, revision, kind: "model", origin: null })) }); + } + if (input.operation === "read" && Object.keys(input).sort().join(",") === "accountId,operation" + && typeof input.accountId === "string" && uuid.test(input.accountId)) { + const account = (await new EncryptedIntegrationRepository(databaseFor(env), proxy.tenantId, env.CREDENTIAL_ENCRYPTION_KEY).records("model")) + .find(record => record.id === input.accountId); + if (!account) return json({ error: "model_account_unavailable" }, 404); + const keyState = await databaseFor(env).pool.query<{ generation_id: string }>("SELECT generation_id FROM app.unlock_key_verifications WHERE tenant_id=$1", [proxy.tenantId]); + if (!keyState.rows[0]) return json({ error: "model_account_unavailable" }, 404); + return json({ ownerId: proxy.tenantId, account: { ...account, keyGeneration: keyState.rows[0].generation_id } }); + } + if (input.operation !== "save" || Object.keys(input).sort().join(",") !== "accountId,encryptedPayload,expectedRevision,idempotencyKey,keyGeneration,operation" + || typeof input.accountId !== "string" || !uuid.test(input.accountId) + || typeof input.keyGeneration !== "string" || !uuid.test(input.keyGeneration) + || typeof input.idempotencyKey !== "string" || !uuid.test(input.idempotencyKey) + || !Number.isSafeInteger(input.expectedRevision) || (input.expectedRevision as number) < 0 + || typeof input.encryptedPayload !== "string") return json({ error: "invalid_request" }, 400); + try { + const saved = await new EncryptedIntegrationRepository(databaseFor(env), proxy.tenantId, env.CREDENTIAL_ENCRYPTION_KEY).saveModelCache({ + id: input.accountId, expectedRevision: input.expectedRevision as number, keyGeneration: input.keyGeneration, + idempotencyKey: input.idempotencyKey, encryptedPayload: input.encryptedPayload, + }); + return json({ ...saved, accountId: input.accountId }); + } catch (error) { + if (error instanceof Error && error.name === "StaleIntegrationRevisionError") return json({ error: "stale_revision" }, 409); + if (error instanceof Error && error.name === "IntegrationIdempotencyConflictError") return json({ error: "idempotency_conflict" }, 409); + if (error instanceof Error && error.name === "IntegrationValidationError") return json({ error: "model_account_unavailable" }, 404); + throw error; + } + } + if (url.pathname === "/internal/credential-proxy/integration-snapshot") { + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).some(key => !["runId", "grantRevision"].includes(key)) + || typeof (body as any).runId !== "string" || !Number.isSafeInteger((body as any).grantRevision) || (body as any).grantRevision < 0) return json({ error: "invalid_request" }, 400); + const database = databaseFor(env); + const active = await database.pool.query<{ execution_config: Record; grant_revision: string | number }>(`SELECT r.execution_config,g.grant_revision + FROM app.runs r JOIN app.credential_run_grants g ON g.tenant_id=r.tenant_id AND g.run_id=r.id + JOIN app.credential_proxy_states p ON p.tenant_id=g.tenant_id AND p.proxy_identity=g.proxy_identity AND p.boot_epoch=g.boot_epoch + WHERE r.tenant_id=$1 AND r.id=$2 AND r.state IN ('starting','running','blocked') + AND g.proxy_identity=$3 AND g.boot_epoch=$4 AND g.grant_revision=$5 AND g.revoked_at IS NULL + AND p.disabled_at IS NULL AND p.revoked_at IS NULL`, [proxy.tenantId, (body as any).runId, proxy.proxyIdentity, proxy.bootEpoch, (body as any).grantRevision]); + const run = active.rows[0]; + if (!run) return json({ error: "integration_snapshot_unavailable" }, 403); + const selected = run.execution_config?.portableIntegrationIds; + if (!Array.isArray(selected) || selected.some(id => typeof id !== "string") || new Set(selected).size !== selected.length) return json({ error: "integration_snapshot_unavailable" }, 403); + const currentRecords = selected.length + ? await new EncryptedIntegrationRepository(database, proxy.tenantId, env.CREDENTIAL_ENCRYPTION_KEY).recordsIncludingDeleted("bearer") + : []; + const byId = new Map(currentRecords.map(record => [record.id, record])); + // A generation reset may remove a row rather than leave its CRUD tombstone. + // Represent a missing selected record as a ciphertext-free tombstone so + // established proxy connections can remove that grant on refresh. + const records = selected.map(id => byId.get(id) ?? { id, kind: "bearer" as const, name: "", origin: null, + description: null, revision: 0, encryptedPayload: null, deletedAt: new Date(0) }); + if (records.length !== selected.length) return json({ error: "integration_snapshot_unavailable" }, 403); + const modelAccess = run.execution_config?.modelAccess ?? "native"; + const modelAccountId = run.execution_config?.modelAccountId; + let modelAccount = null; + if (modelAccess === "portable") { + if (typeof modelAccountId !== "string") return json({ error: "integration_snapshot_unavailable" }, 403); + modelAccount = (await new EncryptedIntegrationRepository(database, proxy.tenantId, env.CREDENTIAL_ENCRYPTION_KEY).records("model")) + .find(record => record.id === modelAccountId) ?? null; + if (!modelAccount) return json({ error: "integration_snapshot_unavailable" }, 403); + } else if (modelAccess !== "native" || modelAccountId !== undefined) { + return json({ error: "integration_snapshot_unavailable" }, 403); + } + // Only the selected model account's user-encrypted payload crosses the + // control plane. It remains encrypted until the proxy is unlocked. + return json({ ownerId: proxy.tenantId, records, modelAccount }); + } + const enrollments = new RunEnrollmentRepository(databaseFor(env)); + if (url.pathname === "/internal/credential-proxy/enroll") { + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).some(key => !["runId", "vmIdentity", "runCredential"].includes(key)) + || typeof (body as any).runId !== "string" || typeof (body as any).vmIdentity !== "string" || typeof (body as any).runCredential !== "string" + || (body as any).vmIdentity.length === 0 || (body as any).vmIdentity.length > 255) return json({ error: "invalid_request" }, 400); + // A proxy-authenticated request alone must not let a peer that can reach + // the proxy claim another active run. The VM's distinct run credential is + // validated against the current run and tenant before issuing enrollment. + const runAuth = await authenticateRunCredential(new Request(request.url, { + headers: { Authorization: `Bearer ${(body as any).runCredential}` }, + }), env); + if (!runAuth || !runAuth.runId || runAuth.runId !== (body as any).runId || runAuth.tenantId !== proxy.tenantId + || !runAuth.scopes.includes("integrations:read")) return json({ error: "enrollment_unavailable" }, 403); + const binding = { tenantId: proxy.tenantId, runId: runAuth.runId, vmIdentity: (body as any).vmIdentity, proxyIdentity: proxy.proxyIdentity, bootEpoch: proxy.bootEpoch }; + // This is the production grant-creation path. Authenticate the active run + // before binding it to its provider VM, and make reconnect authorization + // idempotent for the same VM and proxy boot while advancing stale bindings. + if (await enrollments.authorizeRun(binding) === null) return json({ error: "enrollment_unavailable" }, 403); + const enrollment = await enrollments.issue(binding, new Date(Date.now() + 30_000)); + return enrollment ? json(enrollment) : json({ error: "enrollment_unavailable" }, 403); + } + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).some(key => !["runId", "vmIdentity", "enrollmentToken"].includes(key)) + || typeof (body as any).runId !== "string" || typeof (body as any).vmIdentity !== "string" || typeof (body as any).enrollmentToken !== "string") return json({ error: "invalid_request" }, 400); + const binding = { tenantId: proxy.tenantId, runId: (body as any).runId, vmIdentity: (body as any).vmIdentity, proxyIdentity: proxy.proxyIdentity, bootEpoch: proxy.bootEpoch }; + const consumed = await enrollments.consume((body as any).enrollmentToken, binding); + return consumed ? json({ consumed: true, binding: consumed }) : json({ error: "enrollment_unavailable" }, 403); +} diff --git a/packages/api/src/proxy-launch-policy.ts b/packages/api/src/proxy-launch-policy.ts new file mode 100644 index 00000000..da8cb5ab --- /dev/null +++ b/packages/api/src/proxy-launch-policy.ts @@ -0,0 +1,61 @@ +/** Inputs required to decide whether a queued execution may create a guest VM. */ +export type ProxyLaunchState = { + enabled: boolean; + running: boolean; + unlocked: boolean; + pendingChanges: boolean; +}; + +export type ProxyLaunchJob = { + executionProvider: "exe" | "fly"; + executionType: "agent" | "code"; + portableIntegrationIds?: readonly string[]; + modelAccess?: "native" | "portable"; + modelAccountId?: string; +}; + +export type ProxyLaunchDecision = + | { kind: "allow" } + | { kind: "start_locked" } + | { kind: "blocked"; reason: "proxy_disabled" | "proxy_unavailable" | "proxy_locked" | "pending_changes"; retry: "manual" }; + +export class ProxyLaunchBlockedError extends Error { + constructor(readonly reason: Extract["reason"]) { + super(`Job launch is blocked until proxy prerequisite is resolved: ${reason}`); + this.name = "ProxyLaunchBlockedError"; + } +} + +export function requireProxyLaunch(decision: ProxyLaunchDecision): void { + if (decision.kind === "blocked") throw new ProxyLaunchBlockedError(decision.reason); +} + +/** + * Encodes the GEN-2824 launch gate. This function is deliberately side-effect + * free: the scheduler must persist a blocked decision and wait for an explicit + * retry instead of treating a later unlock as a queued launch signal. + */ +export function proxyLaunchDecision(job: ProxyLaunchJob, proxy: ProxyLaunchState): ProxyLaunchDecision { + const needsUserCredentials = Boolean(job.portableIntegrationIds?.length) + || job.modelAccess === "portable" + || Boolean(job.modelAccountId); + + // exe.dev-native integrations keep their existing launch behavior. + if (job.executionProvider === "exe" && !needsUserCredentials) return { kind: "allow" }; + + if (!proxy.enabled) return { kind: "blocked", reason: "proxy_disabled", retry: "manual" }; + + // Only code jobs with Factorize access may automatically boot a stopped + // proxy, and that boot is always locked. + if (!needsUserCredentials && job.executionType === "code") { + if (!proxy.running) return { kind: "start_locked" }; + return { kind: "allow" }; + } + + // User credential grants require an already-running proxy with edits + // applied and a user unlock. Unlocking alone does not resume this run. + if (!proxy.running) return { kind: "blocked", reason: "proxy_unavailable", retry: "manual" }; + if (proxy.pendingChanges) return { kind: "blocked", reason: "pending_changes", retry: "manual" }; + if (!proxy.unlocked) return { kind: "blocked", reason: "proxy_locked", retry: "manual" }; + return { kind: "allow" }; +} diff --git a/packages/api/src/proxy-provisioner.ts b/packages/api/src/proxy-provisioner.ts new file mode 100644 index 00000000..c32e7e5d --- /dev/null +++ b/packages/api/src/proxy-provisioner.ts @@ -0,0 +1,40 @@ +import type { ProxyControlRepository } from "./postgres/proxy-control-repository"; + +export type ProxyBootstrapProvisioning = { + tenantId: string; + proxyIdentity: string; + /** A new UUID for every process boot. */ + bootEpoch: string; + controlOrigin: string; + browserOrigin: string; +}; + +/** + * Connects one-use bootstrap credential issuance to the infrastructure + * provisioner's private control channel. Implementations must deliver the + * credential directly to proxy stdin (or an equally short-lived memory-only + * channel), never through Machine config, environment, metadata, disk, or logs. + */ +export async function provisionCredentialProxy( + input: ProxyBootstrapProvisioning, + dependencies: { + control: Pick; + startProxy: (request: ProxyBootstrapProvisioning & { credential: string }) => Promise; + }, +): Promise<{ proxyIdentity: string; bootEpoch: string }> { + const origin = new URL(input.controlOrigin); + if (origin.protocol !== "https:" || origin.username || origin.password || origin.pathname !== "/" || origin.search || origin.hash) { + throw new TypeError("controlOrigin must be an HTTPS origin"); + } + const browserOrigin = new URL(input.browserOrigin); + if (browserOrigin.protocol !== "https:" || browserOrigin.origin !== input.browserOrigin || browserOrigin.username || browserOrigin.password) throw new TypeError("browserOrigin must be an HTTPS origin"); + const credential = await dependencies.control.provisionCredential(input.tenantId, input.proxyIdentity, input.bootEpoch); + try { + await dependencies.startProxy({ ...input, controlOrigin: origin.origin, browserOrigin: browserOrigin.origin, credential }); + } catch (error) { + // A failed/ambiguous start must not leave a valid bootstrap credential. + await dependencies.control.revokeCredential(input.tenantId, input.proxyIdentity).catch(() => false); + throw error; + } + return { proxyIdentity: input.proxyIdentity, bootEpoch: input.bootEpoch }; +} diff --git a/packages/api/src/types.ts b/packages/api/src/types.ts index 1fc1e396..f70bbbfa 100644 --- a/packages/api/src/types.ts +++ b/packages/api/src/types.ts @@ -15,6 +15,9 @@ export interface Env { CLICKUP_CLIENT_SECRET?: string; CREDENTIAL_ENCRYPTION_KEY: string; SESSION_SIGNING_SECRET: string; + /** PKCS#8 Ed25519 key used only for short-lived proxy browser capabilities. */ + MODEL_ACCOUNT_CAPABILITY_PRIVATE_KEY?: string; + MODEL_ACCOUNT_CAPABILITY_PUBLIC_KEY?: string; GITHUB_APP_ID?: string; GITHUB_APP_SLUG?: string; GITHUB_APP_PRIVATE_KEY?: string; diff --git a/packages/api/src/worker.ts b/packages/api/src/worker.ts index d7868347..e32c339b 100644 --- a/packages/api/src/worker.ts +++ b/packages/api/src/worker.ts @@ -14,6 +14,7 @@ import { artifactUpload } from "./artifact-upload"; import { traceChunkUpload } from "./trace-chunk-upload"; import { databaseFor } from "./postgres/database"; import { IdentityRepository } from "./postgres/identity-repository"; +import { proxyControlFetch } from "./proxy-control"; const scopes = [...ACCESS_SCOPES]; // Keep accepting grants issued by the pre-resource-scope protocol. New @@ -85,6 +86,7 @@ function provider(env: Env) { return new OAuthProvider(providerOptions(env) export { AlarmCoordinator } from "./alarm-coordinator"; const backend = { async fetch(request: Request, env: Env, ctx: ExecutionContext) { + if (new URL(request.url).pathname.startsWith("/internal/credential-proxy/")) return proxyControlFetch(request, env); const staticResponse = !["/authorize", "/device"].includes(new URL(request.url).pathname) ? await staticApp(request, env) : null; if (staticResponse) return staticResponse; const oauth = provider(env); diff --git a/packages/api/test/api-contract.test.ts b/packages/api/test/api-contract.test.ts index 688d7528..abfbcb01 100644 --- a/packages/api/test/api-contract.test.ts +++ b/packages/api/test/api-contract.test.ts @@ -64,6 +64,31 @@ describe("versioned API contract", () => { expect(post?.body?.safeParse({ generationId: "00000000-0000-4000-8000-000000000001", encryptedVerification: "ciphertext", key: "plain" }).success).toBe(false); expect(openapi.paths["/api/v1/unlock-key/verification"].post.requestBody).toBeDefined(); }); + it("publishes model-account proxy capabilities as owner-only API operations", () => { + const route = API_ROUTES.find(item => item.method === "POST" && item.path === "/api/v1/model-accounts/{accountId}/browser-capability"); + expect(route?.scope).toBe("integrations:write"); + expect(route?.ownerSession).toBe(true); + expect(route?.parameters.shape.accountId).toBeDefined(); + expect(route?.body?.safeParse({ action: "authorize" }).success).toBe(true); + expect(route?.body?.safeParse({ action: "authorize", accountId: "attacker" }).success).toBe(false); + expect(openapi.paths["/api/v1/model-accounts/{accountId}/browser-capability"].post.requestBody).toBeDefined(); + }); + it("lists only safe model account metadata for the owner browser", () => { + const route = API_ROUTES.find(item => item.method === "GET" && item.path === "/api/v1/model-accounts"); + const create = API_ROUTES.find(item => item.method === "POST" && item.path === "/api/v1/model-accounts"); + const remove = API_ROUTES.find(item => item.method === "DELETE" && item.path === "/api/v1/model-accounts/{accountId}"); + expect(route?.scope).toBe("integrations:read"); + expect(route?.ownerSession).toBe(true); + expect(route?.documentation.responses["200"].content["application/json"].schema.properties.accounts.items.properties.encryptedPayload).toBeUndefined(); + expect(create?.scope).toBe("integrations:write"); expect(create?.ownerSession).toBe(true); + expect(create?.body?.safeParse({ id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", expectedRevision: 0, idempotencyKey: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb", name: "Account", description: null, encryptedPayload: "cipher" }).success).toBe(true); + expect(create?.body?.safeParse({ id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", expectedRevision: 0, idempotencyKey: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb", name: "Account", description: null, encryptedPayload: "cipher", unlockKey: "must-not-be-accepted" }).success).toBe(false); + expect(remove?.scope).toBe("integrations:write"); expect(remove?.ownerSession).toBe(true); + expect(remove?.parameters.shape.accountId).toBeDefined(); + expect(openapi.paths["/api/v1/model-accounts"].post).toBeDefined(); + expect(openapi.paths["/api/v1/model-accounts/{accountId}"].delete).toBeDefined(); + expect(openapi.paths["/api/v1/model-accounts"].get).toBeDefined(); + }); it("accepts code jobs without agent prompts or models and rejects missing scripts", () => { const code = { name: "Export", slug: "export", executionType: "code", script: "#!/bin/sh\nprintf '{\"status\":\"succeeded\"}' > \"$FACTORIZE_RESULT_PATH\"", executionConfigId: "exe", triggers: [] }; expect(jobInputSchema.safeParse(code).success).toBe(true); diff --git a/packages/api/test/encrypted-integration-api.postgres.test.ts b/packages/api/test/encrypted-integration-api.postgres.test.ts index 7aa42ad5..81c1143a 100644 --- a/packages/api/test/encrypted-integration-api.postgres.test.ts +++ b/packages/api/test/encrypted-integration-api.postgres.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it, vi } from "vitest"; import worker from "../src/worker"; import { protectedApiFetch } from "../src/protected-api"; import { IdentityRepository } from "../src/postgres/identity-repository"; +import { RunEnrollmentRepository } from "../src/postgres/run-enrollment-repository"; import { signSession } from "../src/session"; import { loadingFixture } from "./helpers/loading-fixture"; @@ -37,6 +38,68 @@ describe.skipIf(!databaseUrl)("browser-only encrypted bearer integration API (Po expect((await request("POST", { ...body, unlockKey: "fzk1.must-not-be-accepted" })).status).toBe(400); } finally { await f.cleanup(); } }); + it("owner-only key replacement atomically resets encrypted records, changes verification, and locks the proxy", async () => { + const f = await loadingFixture(databaseUrl!); + try { + const request = (path: string, method = "GET", body?: unknown, cookie = f.cookie) => worker.fetch(new Request(f.env.APP_ORIGIN + "/api/v1" + path, { + method, headers: { Cookie: cookie, Origin: f.env.APP_ORIGIN, "Content-Type": "application/json" }, ...(body === undefined ? {} : { body: JSON.stringify(body) }), + }), f.env, { waitUntil() {} } as any); + const oldGeneration = crypto.randomUUID(), newGeneration = crypto.randomUUID(); + const verification = (generationId: string) => JSON.stringify({ version: 1, algorithm: "A256GCM", nonce: "AAAAAAAAAAAAAAAA", ciphertext: "AAAAAAAAAAAAAAAAAAAAAA" }); + expect((await request("/unlock-key/verification", "POST", { generationId: oldGeneration, encryptedVerification: verification(oldGeneration) })).status).toBe(200); + const encrypted = await userEncrypt("synthetic bearer payload"); + const integrationId = crypto.randomUUID(); + expect((await request("/integrations/bearer", "POST", { id: integrationId, expectedRevision: 0, idempotencyKey: crypto.randomUUID(), name: "Synthetic", origin: "https://api.example.test", description: null, encryptedPayload: encrypted })).status).toBe(201); + await f.db.pool.query("INSERT INTO app.connections(tenant_id,kind,encrypted_value) VALUES($1,'fly:connection-fixture','encrypted-fly-fixture')", [f.tenantId]); + await f.db.pool.query("UPDATE app.jobs SET execution_config=$3 WHERE tenant_id=$1 AND id=$2", [f.tenantId, f.jobIds[0], { provider: "fly", connectionId: "fly:connection-fixture", region: "ord", vmSize: "shared-cpu-1x" }]); + const flyConnectionBefore = (await f.db.pool.query("SELECT kind,encrypted_value FROM app.connections WHERE tenant_id=$1 ORDER BY kind", [f.tenantId])).rows; + const executionConfigBefore = (await f.db.pool.query("SELECT id,execution_config FROM app.jobs WHERE tenant_id=$1 ORDER BY id", [f.tenantId])).rows; + await f.db.pool.query("INSERT INTO app.credential_proxy_states(tenant_id,proxy_identity,boot_epoch,unlocked) VALUES($1,'proxy-test',$2,true)", [f.tenantId, crypto.randomUUID()]); + const runId = crypto.randomUUID(), jobId = f.jobIds[0], invocationId = crypto.randomUUID(); + const triggerId = (await f.db.pool.query("SELECT id FROM app.triggers WHERE tenant_id=$1 AND job_id=$2 AND kind='manual' LIMIT 1", [f.tenantId, jobId])).rows[0].id; + await f.db.pool.query("INSERT INTO app.invocations(tenant_id,id,job_id,source,claim_key,trigger_id,context) VALUES ($1,$2,$3,'manual','rotation-test',$4,'{}')", [f.tenantId, invocationId, jobId, triggerId]); + await f.db.pool.query("INSERT INTO app.job_runs(tenant_id,id,job_id,invocation_id,state,encrypted_prompt) VALUES ($1,$2,$3,$4,'running','test')", [f.tenantId, runId, jobId, invocationId]); + const enrollmentRepository = new RunEnrollmentRepository(f.db); + const binding = { tenantId: f.tenantId, runId, vmIdentity: "vm-rotation", proxyIdentity: "proxy-test", bootEpoch: (await f.db.pool.query("SELECT boot_epoch FROM app.credential_proxy_states WHERE tenant_id=$1", [f.tenantId])).rows[0].boot_epoch }; + expect(await enrollmentRepository.authorizeRun(binding)).toBe(1); + const issued = await enrollmentRepository.issue(binding, new Date(Date.now() + 60_000)); + expect(issued).not.toBeNull(); + + const snapshot = await request("/unlock-key/rotation-snapshot"); + expect(snapshot.status).toBe(200); + expect(await snapshot.json()).toMatchObject({ ownerId: f.userId, generationId: oldGeneration, revision: 1, records: [{ id: integrationId, kind: "bearer", encryptedPayload: encrypted }] }); + const replacement = await request("/unlock-key/rotation", "POST", { + expectedGenerationId: oldGeneration, generationId: newGeneration, expectedRevision: 1, + encryptedVerification: verification(newGeneration), records: [], changes: [], + }); + expect(replacement.status).toBe(200); + expect(await replacement.json()).toEqual({ generationId: newGeneration, revision: 2, duplicate: false }); + expect((await f.db.pool.query("SELECT generation_id FROM app.unlock_key_verifications WHERE tenant_id=$1", [f.tenantId])).rows[0].generation_id).toBe(newGeneration); + expect((await f.db.pool.query("SELECT id FROM app.integration_records WHERE tenant_id=$1", [f.tenantId])).rows).toEqual([]); + expect((await f.db.pool.query("SELECT unlocked FROM app.credential_proxy_states WHERE tenant_id=$1", [f.tenantId])).rows[0].unlocked).toBe(false); + expect((await f.db.pool.query("SELECT kind,encrypted_value FROM app.connections WHERE tenant_id=$1 ORDER BY kind", [f.tenantId])).rows).toEqual(flyConnectionBefore); + expect((await f.db.pool.query("SELECT id,execution_config FROM app.jobs WHERE tenant_id=$1 ORDER BY id", [f.tenantId])).rows).toEqual(executionConfigBefore); + expect((await f.db.pool.query("SELECT revoked_at FROM app.credential_run_grants WHERE tenant_id=$1 AND run_id=$2", [f.tenantId, runId])).rows[0].revoked_at).toBeInstanceOf(Date); + expect((await f.db.pool.query("SELECT consumed_at FROM app.credential_run_enrollments WHERE tenant_id=$1 AND run_id=$2", [f.tenantId, runId])).rows[0].consumed_at).toBeInstanceOf(Date); + expect(await enrollmentRepository.consume(issued!.token, binding)).toBeNull(); + const rotatedCipher = await userEncrypt("rotated synthetic bearer payload"), rotatedId = crypto.randomUUID(); + expect((await request("/integrations/bearer", "POST", { id: rotatedId, expectedRevision: 2, idempotencyKey: crypto.randomUUID(), name: "Rotated", origin: "https://api.example.test", description: null, encryptedPayload: encrypted })).status).toBe(201); + const rotationSnapshot = await (await request("/unlock-key/rotation-snapshot")).json() as any; + const finalGeneration = crypto.randomUUID(); + const rotated = await request("/unlock-key/rotation", "POST", { + expectedGenerationId: newGeneration, generationId: finalGeneration, expectedRevision: 3, + encryptedVerification: verification(finalGeneration), + records: rotationSnapshot.records.map((record: any) => ({ ...record, encryptedPayload: record.id === rotatedId ? rotatedCipher : record.encryptedPayload })), + changes: rotationSnapshot.changes.map((change: any) => ({ ...change, encryptedPayload: change.id === rotatedId && change.operation === "upsert" ? rotatedCipher : change.encryptedPayload })), + }); + expect(rotated.status).toBe(200); + expect(await rotated.json()).toEqual({ generationId: finalGeneration, revision: 4, duplicate: false }); + expect((await request("/integrations/bearer")).status).toBe(200); + expect((await (await request("/integrations/bearer")).json() as any).integrations[0].encryptedPayload).toBe(rotatedCipher); + const agent = await request("/unlock-key/rotation-snapshot", "GET", undefined, "untrusted-cookie"); + expect(agent.status).toBe(401); + } finally { await f.cleanup(); } + }); it("supports safe create, list, update, delete and rejects malformed and stale writes", async () => { const f = await loadingFixture(databaseUrl!); try { @@ -76,6 +139,43 @@ describe.skipIf(!databaseUrl)("browser-only encrypted bearer integration API (Po } finally { await f.cleanup(); } }); + it("creates and deletes model accounts through owner-only revision-checked writes", async () => { + const f = await loadingFixture(databaseUrl!); + try { + const path = f.env.APP_ORIGIN + "/api/v1/model-accounts", id = crypto.randomUUID(); + const request = (url: string, method = "GET", body?: unknown, cookie = f.cookie) => worker.fetch(new Request(url, { + method, headers: { Cookie: cookie, Origin: f.env.APP_ORIGIN, "Content-Type": "application/json" }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + }), f.env, { waitUntil() {} } as any); + const ciphertext = await userEncrypt("{}"), createBody = { + id, expectedRevision: 0, idempotencyKey: crypto.randomUUID(), name: "Work account", + description: "Personal subscription", encryptedPayload: ciphertext, + }; + const created = await request(path, "POST", createBody); + expect(created.status).toBe(201); + expect(await created.json()).toEqual({ revision: 1, duplicate: false }); + expect(await (await request(path, "POST", createBody)).json()).toEqual({ revision: 1, duplicate: true }); + + const listed = await request(path); + expect(listed.status).toBe(200); + expect(await listed.json()).toEqual({ revision: 1, accounts: [{ id, name: "Work account", description: "Personal subscription", revision: 1 }] }); + + const stale = await request(path + `/${id}`, "DELETE", { expectedRevision: 0, idempotencyKey: crypto.randomUUID() }); + expect(stale.status).toBe(409); + expect((await stale.json() as any).error.code).toBe("stale_revision"); + + const deleteBody = { expectedRevision: 1, idempotencyKey: crypto.randomUUID() }; + const deleted = await request(path + `/${id}`, "DELETE", deleteBody); + expect(deleted.status).toBe(200); + expect(await deleted.json()).toEqual({ id, deleted: true, revision: 2, duplicate: false }); + expect(await (await request(path + `/${id}`, "DELETE", deleteBody)).json()).toEqual({ id, deleted: true, revision: 2, duplicate: true }); + expect(await (await request(path)).json()).toEqual({ revision: 2, accounts: [] }); + + const agent = await protectedApiFetch(new Request(path, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ ...createBody, id: crypto.randomUUID() }) }), f.env, { tenantId: f.tenantId, userId: f.userId, sessionVersion: 0, scopes: ["integrations:write"], authMethod: "access_token" } as any, {} as any); + expect(agent.status).toBe(403); + } finally { await f.cleanup(); } + }); + it("requires a browser session and isolates records across tenants", async () => { const f = await loadingFixture(databaseUrl!); try { diff --git a/packages/api/test/encrypted-integration-repository.postgres.test.ts b/packages/api/test/encrypted-integration-repository.postgres.test.ts index d4911b84..c2a5e7ce 100644 --- a/packages/api/test/encrypted-integration-repository.postgres.test.ts +++ b/packages/api/test/encrypted-integration-repository.postgres.test.ts @@ -85,6 +85,28 @@ describe.skipIf(!url)("encrypted integration persistence (PostgreSQL)", () => { expect(results.filter(result => result.status === "rejected" && result.reason instanceof StaleIntegrationRevisionError)).toHaveLength(1); }); + it("rejects a model cache save encrypted under a replaced key generation even when its account revision is unchanged", async () => { + const { tenantId, repository } = await freshRepository(); + const generation = crypto.randomUUID(); + const replacementGeneration = crypto.randomUUID(); + await db.pool.query("INSERT INTO app.unlock_key_verifications(tenant_id,generation_id,encrypted_verification) VALUES ($1,$2,'verification')", [tenantId, generation]); + const id = crypto.randomUUID(); + const created = await repository.write({ id, idempotencyKey: crypto.randomUUID(), expectedRevision: 0, operation: "upsert", kind: "model", name: "Codex", origin: null, description: null, encryptedPayload: await userEncrypt("initial cache") }); + const oldKeyCache = await userEncrypt("old generation cache"); + const saved = await repository.saveModelCache({ id, expectedRevision: created.revision, keyGeneration: generation, idempotencyKey: crypto.randomUUID(), encryptedPayload: oldKeyCache }); + expect(saved.revision).toBe(created.revision + 1); + + await db.pool.query("UPDATE app.unlock_key_verifications SET generation_id=$2 WHERE tenant_id=$1", [tenantId, replacementGeneration]); + const staleKeyCache = await userEncrypt("late old generation refresh"); + await expect(repository.saveModelCache({ id, expectedRevision: saved.revision, keyGeneration: generation, idempotencyKey: crypto.randomUUID(), encryptedPayload: staleKeyCache })) + .rejects.toMatchObject({ name: "StaleIntegrationRevisionError" }); + expect((await repository.records("model"))[0]?.encryptedPayload).toBe(oldKeyCache); + + const currentKeyCache = await userEncrypt("replacement generation cache"); + await expect(repository.saveModelCache({ id, expectedRevision: saved.revision, keyGeneration: replacementGeneration, idempotencyKey: crypto.randomUUID(), encryptedPayload: currentKeyCache })) + .resolves.toMatchObject({ revision: saved.revision + 1, duplicate: false }); + }); + it("returns edits and deletion tombstones in revision order and deduplicates retries", async () => { const { repository, tenantId } = await freshRepository(); const id = crypto.randomUUID(), idempotencyKey = crypto.randomUUID(); diff --git a/packages/api/test/execution-backend.test.ts b/packages/api/test/execution-backend.test.ts index c1c3da7c..511288a4 100644 --- a/packages/api/test/execution-backend.test.ts +++ b/packages/api/test/execution-backend.test.ts @@ -1,5 +1,4 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { execFileSync } from "node:child_process"; import { ExeImageVerificationError, ExeVmBackend, isOwnedVmName, tagsFromInventory, vmNameFor } from "../src/exe-vm-backend"; import type { ExecutionBackend, ExecutionObservation, LaunchRequest, RunHandle } from "../src/execution"; import { effortOptions, selectableExeTags, type ExeRunConnection } from "../src/exe"; @@ -9,6 +8,15 @@ import { runProxyName } from "../src/run-credentials"; afterEach(() => vi.unstubAllGlobals()); const connection = { apiToken: "secret", agentKind: "codex", tags: ["github", "llm"], model: "gpt-5.5", effort: "high" } satisfies ExeRunConnection; +function launchPayload(command: string): any { + for (const match of command.matchAll(/'([A-Za-z0-9+/=]{20,})'/g)) { + try { + const value = JSON.parse(Buffer.from(match[1]!, "base64").toString("utf8")); + if (value?.version === 1 && (value.code || value.harness)) return value; + } catch { /* Other encoded files include the launcher and code context. */ } + } + throw new Error("Missing encoded Factorize launch payload"); +} describe("ExeVmBackend", () => { it("reserves its legacy VM and proxy handle for scheduler recovery", () => { @@ -66,7 +74,7 @@ describe("ExeVmBackend", () => { expect(result.missingPermissions).toEqual([]); }); - it("creates a tagged VM and launches Codex directly in the workspace", async () => { + it("creates a tagged VM and launches Codex through the shared guest launcher", async () => { const requests: string[] = []; vi.stubGlobal("fetch", vi.fn(async (_url: string, init: RequestInit) => { requests.push(String(init.body)); @@ -81,21 +89,17 @@ describe("ExeVmBackend", () => { expect(requests[0]).toContain("--tag='llm'"); expect(requests[1]).toContain("cd /home/exedev/workspace"); expect(requests[1]).not.toContain("git clone"); - expect(requests[1]).toContain("codex"); - expect(requests[1]).toContain("--color"); - expect(requests[1]).toContain("never"); - expect(requests[1]).toContain("model_provider=factorize"); - expect(requests[1]).toContain("model_providers.factorize.base_url=\"https://models.example/v1\""); - expect(requests[1]).toContain("model_providers.factorize.requires_openai_auth=false"); + expect(requests[1]).toContain("factorize-job launch --foreground"); + expect(requests[1]).toContain("base64 -d > /usr/local/bin/factorize-job"); + expect(launchPayload(requests[1]).harness).toMatchObject({ executable: "codex", args: expect.arrayContaining(["--color", "never", "-c", "model_provider=factorize", "-c", "model_providers.factorize.base_url=\"https://models.example/v1\""]) }); expect(requests[1]).toContain("sudo systemd-run"); - expect(requests[1]).toContain("'--setenv=FACTORIZE_RESULT_PATH=/home/exedev/.factorize-job/results.json'"); expect(requests[1]).toContain("--uid=exedev --gid=exedev"); expect(requests[1]).toContain("--property=Type=exec"); expect(requests[1]).toContain("--property=RemainAfterExit=yes"); - expect(requests[1]).toContain("--property=StandardInput=file:/tmp/factorize-prompt.md"); + expect(requests[1]).toContain("--property=StandardInput=/dev/null"); expect(requests[1]).toContain("--property=StandardOutput=append:/tmp/factorize.log"); expect(requests[1]).toContain("--property=StandardError=append:/tmp/factorize.stderr"); - expect(requests[1]).toContain("/tmp/trace/events.jsonl"); + expect(launchPayload(requests[1]).traceSources.primary.path).toBe("/tmp/trace/events.jsonl"); expect(requests[1]).not.toContain("StandardOutput=append:/tmp/trace/events.jsonl"); expect(requests[1]).not.toContain("nohup"); expect(requests[1]).toContain("; echo started"); @@ -114,14 +118,12 @@ describe("ExeVmBackend", () => { vi.stubGlobal("fetch", vi.fn(async (_url: string, init: RequestInit) => { requests.push(String(init.body)); return new Response(requests.length === 1 ? '{"name":"factorize-code"}' : "started", { headers: { "X-Exe-Exit": "0" } }); })); const launched = await new ExeVmBackend({ apiToken: "secret", tags: ["jobs"], image: "example/base:latest" }).launch({ runId: "code", prompt: "", code: { script: "echo done", context: { source: { outputs: { count: 3 } } } } }); expect(launched.handle.id).toBe("factorize-code"); - expect(requests[1]).toContain("FACTORIZE_RESULT_PATH=/home/exedev/.factorize-job/results.json"); - expect(requests[1]).toContain("FACTORIZE_INPUT_PATH=/home/exedev/.factorize-job/input.json"); - expect(requests[1]).toContain("bash /tmp/factorize-code-job.sh"); + expect(launchPayload(requests[1]).code).toEqual({ script: "echo done", context: { source: { outputs: { count: 3 } } } }); expect(requests[1]).toContain("--uid=exedev --gid=exedev"); expect(requests[1]).not.toContain("codex"); }); - it.each(["codex", "claude", "pi"] as const)("passes %s harness environment as systemd options instead of an executable", async kind => { + it.each(["codex", "claude", "pi"] as const)("passes %s harness environment through the shared launcher payload", async kind => { const requests: string[] = []; vi.stubGlobal("fetch", vi.fn(async (_url: string, init: RequestInit) => { requests.push(String(init.body)); @@ -132,16 +134,8 @@ describe("ExeVmBackend", () => { harness.env.FACTORIZE_TEST_LITERAL = literal; await new ExeVmBackend({ ...connection, agentKind: kind }).launch({ runId: "run-env", prompt: "work", harness }); - // Decode the actual SSH and remote shell quoting, without executing provisioning. - const split = (command: string) => execFileSync("/bin/sh", ["-c", `set -- ${command}; printf '%s\\0' "$@"`], { encoding: "utf8" }).split("\0").slice(0, -1); - const remote = split(requests.find(command => command.includes("sudo systemd-run"))!)[2]; - const command = remote.slice(remote.indexOf("sudo systemd-run"), remote.lastIndexOf("; fi")); - const argv = split(command).slice(2); - expect(argv).toContain("--setenv=FACTORIZE_RESULT_PATH=/home/exedev/.factorize-job/results.json"); - expect(argv).toContain(`--setenv=FACTORIZE_TEST_LITERAL=${literal}`); - // The first non-option argument must remain the harness executable for all agents. - const executable = argv.findIndex(value => !value.startsWith("--")); - expect(argv.slice(executable)).toEqual([harness.executable, ...harness.args]); + const payload = launchPayload(requests.find(command => command.includes("sudo systemd-run"))!); + expect(payload.harness).toMatchObject({ executable: harness.executable, args: harness.args, env: { FACTORIZE_TEST_LITERAL: literal } }); }); it("verifies an image over SSH and deletes only its marked temporary VM", async () => { @@ -217,7 +211,7 @@ describe("ExeVmBackend", () => { expect(supervisor).toContain("append:/tmp/factorize-artifacts/run-1/codex-exec.jsonl"); expect(supervisor).not.toContain("append:/tmp/factorize.log"); expect(supervisor).toContain("StandardError=append:/tmp/factorize.stderr"); - expect(supervisor).toContain("--json"); + expect(launchPayload(supervisor).harness.args).toContain("--json"); }); it("uses the systemd unit as the authoritative run state", async () => { @@ -229,7 +223,7 @@ describe("ExeVmBackend", () => { ]; const requests: string[] = []; vi.stubGlobal("fetch", vi.fn(async (_url: string, init: RequestInit) => { - requests.push(String(init.body)); + const command = String(init.body); requests.push(command); return new Response(states.shift(), { headers: { "X-Exe-Exit": "0" } }); })); const backend = new ExeVmBackend(connection), handle = { backendKind: "exe-vm", id: "factorize-run-1" }; @@ -237,8 +231,8 @@ describe("ExeVmBackend", () => { await expect(backend.inspect(handle)).resolves.toMatchObject({ state: "succeeded" }); await expect(backend.inspect(handle)).resolves.toMatchObject({ state: "failed", systemd: { result: "exit-code", execMainCode: 1, execMainStatus: 42, activeState: "failed", subState: "failed" }, detail: expect.stringContaining("unsuccessfully") }); await expect(backend.inspect(handle)).resolves.toMatchObject({ state: "failed", detail: expect.stringContaining("supervisor disappeared") }); + expect(requests).toHaveLength(4); expect(requests.every(request => request.includes("systemctl show"))).toBe(true); - expect(requests.every(request => !request.includes("factorize.status"))).toBe(true); }); it("uploads an immutable snapshot of a session that may still be growing", async () => { diff --git a/packages/api/test/factorize-job-launcher.test.ts b/packages/api/test/factorize-job-launcher.test.ts new file mode 100644 index 00000000..e46079a1 --- /dev/null +++ b/packages/api/test/factorize-job-launcher.test.ts @@ -0,0 +1,129 @@ +import { execFileSync } from "node:child_process"; +import { mkdir, mkdtemp, readFile, rm, watch, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { FACTORIZE_JOB_LAUNCHER } from "../src/factorize-job-launcher"; + +describe("shared factorize-job launcher", () => { + it("runs code jobs asynchronously and records process completion and output", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-job-launcher-")); + const root = join(directory, "job"), log = join(directory, "factorize.log"), helper = join(directory, "factorize-job"); + const source = FACTORIZE_JOB_LAUNCHER.replace("/home/exedev/.factorize-job", root).replace("/tmp/factorize.log", log).replace("/tmp/factorize.stderr", join(directory, "factorize.stderr")); + try { + await writeFile(helper, source, { mode: 0o700 }); + await mkdir(root, { mode: 0o700 }); + const events = watch(root, { persistent: false }); + const completion = (async () => { + const deadline = Date.now() + 5_000; + for await (const _event of events) { + try { + const state = JSON.parse(await readFile(join(root, "process.json"), "utf8")); + if (state.state === "exited") return state; + } catch { /* The launcher writes process state by atomic rename. */ } + if (Date.now() > deadline) throw new Error("factorize-job did not record completion before the test deadline"); + } + throw new Error("factorize-job state watcher closed before completion"); + })(); + const payload = JSON.stringify({ version: 1, runId: "test-run", code: { script: "printf '%s' '{\"status\":\"succeeded\"}' > \"$FACTORIZE_RESULT_PATH\"; printf 'launcher-ok\\n'", context: { fixture: true } } }); + expect(execFileSync("python3", [helper, "launch"], { input: payload, encoding: "utf8" }).trim()).toBe("started"); + await expect(completion).resolves.toEqual({ state: "exited", exitCode: 0 }); + expect(await readFile(log, "utf8")).toContain("launcher-ok"); + expect(JSON.parse(await readFile(join(root, "results.json"), "utf8"))).toEqual({ status: "succeeded" }); + expect(JSON.parse(execFileSync("python3", [helper, "status"], { encoding: "utf8" }))).toEqual({ state: "exited", exitCode: 0 }); + } finally { await rm(directory, { recursive: true, force: true }); } + }); + + it("preserves a failed code process exit state, structured result, and output", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-job-launcher-failure-")); + const root = join(directory, "job"), log = join(directory, "factorize.log"), helper = join(directory, "factorize-job"); + const source = FACTORIZE_JOB_LAUNCHER.replace("/home/exedev/.factorize-job", root).replace("/tmp/factorize.log", log).replace("/tmp/factorize.stderr", join(directory, "factorize.stderr")); + try { + await writeFile(helper, source, { mode: 0o700 }); + await mkdir(root, { mode: 0o700 }); + const events = watch(root, { persistent: false }); + const completion = (async () => { + const deadline = Date.now() + 5_000; + for await (const _event of events) { + try { + const state = JSON.parse(await readFile(join(root, "process.json"), "utf8")); + if (state.state === "exited") return state; + } catch { /* The launcher writes process state by atomic rename. */ } + if (Date.now() > deadline) throw new Error("factorize-job did not record failure before the test deadline"); + } + throw new Error("factorize-job state watcher closed before failure was recorded"); + })(); + const payload = JSON.stringify({ + version: 1, + runId: "failed-code-run", + code: { script: "printf '%s' '{\"status\":\"failed\",\"reason\":\"fixture failure\"}' > \"$FACTORIZE_RESULT_PATH\"; printf 'failure-output\\n'; exit 7", context: {} }, + }); + + expect(execFileSync("python3", [helper, "launch"], { input: payload, encoding: "utf8" }).trim()).toBe("started"); + await expect(completion).resolves.toEqual({ state: "exited", exitCode: 7 }); + expect(await readFile(log, "utf8")).toContain("failure-output"); + expect(JSON.parse(await readFile(join(root, "results.json"), "utf8"))).toEqual({ status: "failed", reason: "fixture failure" }); + expect(JSON.parse(execFileSync("python3", [helper, "status"], { encoding: "utf8" }))).toEqual({ state: "exited", exitCode: 7 }); + } finally { await rm(directory, { recursive: true, force: true }); } + }); + + it("runs an agent harness with its prompt and environment and records its result", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-agent-launcher-")); + const root = join(directory, "job"), log = join(directory, "factorize.log"), helper = join(directory, "factorize-job"); + const agent = join(directory, "fake-agent.py"); + const source = FACTORIZE_JOB_LAUNCHER.replace("/home/exedev/.factorize-job", root).replace("/tmp/factorize.log", log).replace("/tmp/factorize.stderr", join(directory, "factorize.stderr")); + try { + await writeFile(helper, source, { mode: 0o700 }); + await writeFile(agent, "#!/usr/bin/env python3\nimport json, os, pathlib, sys\nprompt = sys.stdin.read()\nresult = {\"status\": \"succeeded\", \"prompt\": prompt, \"account\": os.environ.get(\"FACTORIZE_FIXTURE_ACCOUNT\")}\npathlib.Path(os.environ[\"FACTORIZE_RESULT_PATH\"]).write_text(json.dumps(result))\nprint(\"agent-harness-ok\")\n", { mode: 0o700 }); + await mkdir(root, { mode: 0o700 }); + const events = watch(root, { persistent: false }); + const completion = (async () => { + const deadline = Date.now() + 5_000; + for await (const _event of events) { + try { + const state = JSON.parse(await readFile(join(root, "process.json"), "utf8")); + if (state.state === "exited") return state; + } catch { /* The launcher writes process state by atomic rename. */ } + if (Date.now() > deadline) throw new Error("factorize-job did not record agent completion before the test deadline"); + } + throw new Error("factorize-job state watcher closed before agent completion"); + })(); + const payload = JSON.stringify({ + version: 1, + runId: "agent-fixture-run", + prompt: "summarize the fixture", + harness: { executable: agent, args: ["--fixture"], env: { FACTORIZE_FIXTURE_ACCOUNT: "account-a" } }, + }); + + expect(execFileSync("python3", [helper, "launch"], { input: payload, encoding: "utf8" }).trim()).toBe("started"); + await expect(completion).resolves.toEqual({ state: "exited", exitCode: 0 }); + expect(await readFile(log, "utf8")).toContain("agent-harness-ok"); + expect(JSON.parse(await readFile(join(root, "results.json"), "utf8"))).toEqual({ + status: "succeeded", prompt: "summarize the fixture", account: "account-a", + }); + expect(JSON.parse(execFileSync("python3", [helper, "status"], { encoding: "utf8" }))).toEqual({ state: "exited", exitCode: 0 }); + } finally { await rm(directory, { recursive: true, force: true }); } + }); + + it("allows a custom image entrypoint to finish setup before execing the shared launcher", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-custom-image-launcher-")); + const root = join(directory, "job"), log = join(directory, "factorize.log"), helper = join(directory, "factorize-job"); + const setup = join(directory, "image-setup.complete"); + const source = FACTORIZE_JOB_LAUNCHER.replace("/home/exedev/.factorize-job", root).replace("/tmp/factorize.log", log).replace("/tmp/factorize.stderr", join(directory, "factorize.stderr")); + try { + await writeFile(helper, source, { mode: 0o700 }); + await mkdir(root, { mode: 0o700 }); + const entrypoint = `printf '%s' ready > '${setup}'; exec python3 '${helper}' launch --foreground`; + const payload = JSON.stringify({ + version: 1, + runId: "custom-image-run", + code: { script: "printf '%s' '{\"status\":\"succeeded\"}' > \"$FACTORIZE_RESULT_PATH\"; printf 'custom-image-ok\\n'", context: {} }, + }); + + expect(execFileSync("sh", ["-c", entrypoint], { input: payload, encoding: "utf8" }).trim()).toBe("custom-image-ok"); + expect(await readFile(setup, "utf8")).toBe("ready"); + expect(JSON.parse(await readFile(join(root, "results.json"), "utf8"))).toEqual({ status: "succeeded" }); + expect(JSON.parse(await readFile(join(root, "process.json"), "utf8"))).toEqual({ state: "exited", exitCode: 0 }); + } finally { await rm(directory, { recursive: true, force: true }); } + }); +}); diff --git a/packages/api/test/fly-credential-proxy-provisioner.test.ts b/packages/api/test/fly-credential-proxy-provisioner.test.ts new file mode 100644 index 00000000..5d5e51a5 --- /dev/null +++ b/packages/api/test/fly-credential-proxy-provisioner.test.ts @@ -0,0 +1,255 @@ +import { describe, expect, it, vi } from "vitest"; +import { createFlyCredentialProxyProvisioner, createFlyCredentialProxyGuest } from "../src/fly-credential-proxy-provisioner"; + +const epoch = "11111111-1111-4111-8111-111111111111"; +const machine = { + id: "machine-1", + state: "created", + config: { metadata: { + factorize_role: "credential-proxy", + factorize_tenant_id: "11111111-1111-4111-8111-111111111112", + factorize_operation_id: "tenant-proxy-start-1", + factorize_boot_epoch: "11111111-1111-4111-8111-111111111111", + } }, +}; +const input = { + tenantId: "11111111-1111-4111-8111-111111111112", + app: "factorize-proxies", + image: `ghcr.io/generationsoftware/credential-proxy@sha256:${"a".repeat(64)}`, + region: "iad", + size: "shared-cpu-1x", + browserCapabilityPublicKey: "-----BEGIN PUBLIC KEY-----\nfixture\n-----END PUBLIC KEY-----", + controlOrigin: "https://factorize.example", + operationId: "tenant-proxy-start-1", +}; + +function fixture() { + const control = { + provisionCredential: vi.fn(async () => `fzc_${"b".repeat(43)}`), + revokeCredential: vi.fn(async () => true), + }; + const machines = { + createMachine: vi.fn(async (_app: string, _config: any, _operationId: string) => machine), + startMachine: vi.fn(async () => undefined), + destroyMachine: vi.fn(async () => undefined), + }; + const guest = vi.fn(() => ({ run: vi.fn(async () => ({ ok: true, exitCode: 0, stdout: "", stderr: "" })), startProxy: vi.fn(async () => undefined), issueVmCertificate: vi.fn(async () => ({ certificate: "", trustBundle: "" })) })); + const provisioner = createFlyCredentialProxyProvisioner({ control, machines, guest, reviewedImage: input.image, newBootEpoch: () => epoch }); + return { control, machines, guest, provisioner }; +} + +describe("Fly credential proxy startup", () => { + it("creates a digest-pinned dedicated Machine and delivers bootstrap only through startup", async () => { + const { control, machines, guest, provisioner } = fixture(); + const started = await provisioner.start(input); + expect(started).toMatchObject({ proxyIdentity: "fly:factorize-proxies:machine-1", bootEpoch: epoch, machine }); + expect(machines.createMachine).toHaveBeenCalledWith(input.app, expect.objectContaining({ + image: input.image, + region: input.region, + cmd: ["/usr/local/bin/factorize-credential-proxy", "--boot-epoch", epoch], + guest: { cpu_kind: "shared", cpus: 1, memory_mb: 512 }, + env: { FACTORIZE_PROXY_TUNNEL_SERVER_NAME: "factorize-proxies.internal", FACTORIZE_PROXY_TUNNEL_PORT: "9090", FACTORIZE_PROXY_BROWSER_CAPABILITY_PUBLIC_KEY: input.browserCapabilityPublicKey }, + services: [ + { protocol: "tcp", internal_port: 8080, ports: [{ port: 443, handlers: ["tls", "http"] }], force_instance_key: true }, + { protocol: "tcp", internal_port: 9090, ports: [] }, + ], + restart: { policy: "no" }, + auto_start: false, + auto_stop: false, + metadata: expect.objectContaining({ factorize_role: "credential-proxy", factorize_tenant_id: input.tenantId }), + }), input.operationId); + expect(JSON.stringify(machines.createMachine.mock.calls[0])).not.toContain("fzc_"); + expect(machines.startMachine).toHaveBeenCalledWith(input.app, machine.id); + expect(control.provisionCredential).toHaveBeenCalledWith(input.tenantId, "fly:factorize-proxies:machine-1", epoch); + expect(guest).toHaveBeenCalledWith(machine); + const guestAdapter = guest.mock.results[0]!.value; + expect(guestAdapter.startProxy).toHaveBeenCalledWith(expect.objectContaining({ + tenantId: input.tenantId, + proxyIdentity: "fly:factorize-proxies:machine-1", + bootEpoch: epoch, + controlOrigin: input.controlOrigin, + browserOrigin: `https://${input.app}.fly.dev`, + credential: `fzc_${"b".repeat(43)}`, + })); + }); + + it.each([ + ["shared-cpu-1x", "shared", 1], + ["shared-cpu-2x", "shared", 2], + ["shared-cpu-4x", "shared", 4], + ["performance-1x", "performance", 1], + ["performance-2x", "performance", 2], + ["performance-4x", "performance", 4], + ] as const)("uses the requested region, %s size and memory without enabling suspension", async (size, cpuKind, cpus) => { + const { machines, provisioner } = fixture(); + await provisioner.start({ ...input, region: "ord", size, memoryMb: 2048 }); + + const [, config, operationId] = machines.createMachine.mock.calls[0]!; + expect(config).toMatchObject({ + image: input.image, + region: "ord", + guest: { cpu_kind: cpuKind, cpus, memory_mb: 2048 }, + auto_start: false, + auto_stop: false, + restart: { policy: "no" }, + }); + expect(operationId).toBe(input.operationId); + expect(config).not.toHaveProperty("suspend"); + expect(config).not.toHaveProperty("auto_suspend"); + }); + + it("revokes bootstrap authority and removes a Machine when startup fails", async () => { + const { control, machines, guest } = fixture(); + guest.mockReturnValue({ run: vi.fn(), startProxy: vi.fn().mockRejectedValue(new Error("private startup channel failed")), issueVmCertificate: vi.fn() }); + const provisioner = createFlyCredentialProxyProvisioner({ control, machines, guest, reviewedImage: input.image, newBootEpoch: () => epoch }); + await expect(provisioner.start(input)).rejects.toThrow("private startup channel failed"); + expect(control.revokeCredential).toHaveBeenCalledWith(input.tenantId, "fly:factorize-proxies:machine-1"); + expect(machines.destroyMachine).toHaveBeenCalledWith(input.app, machine.id); + }); + + it("removes a created Machine when Fly returns an invalid boot epoch", async () => { + const f = fixture(); + f.machines.createMachine.mockResolvedValue({ + ...machine, + config: { metadata: { ...machine.config.metadata, factorize_boot_epoch: "not-a-uuid" } }, + } as any); + + await expect(f.provisioner.start(input)).rejects.toThrow("mismatched provisioning identity"); + + expect(f.machines.destroyMachine).toHaveBeenCalledWith(input.app, machine.id); + expect(f.control.provisionCredential).not.toHaveBeenCalled(); + expect(f.guest).not.toHaveBeenCalled(); + }); + + it("rejects a valid but different returned boot epoch before granting bootstrap authority", async () => { + const f = fixture(); + f.machines.createMachine.mockResolvedValue({ + ...machine, + config: { metadata: { ...machine.config.metadata, factorize_boot_epoch: "22222222-2222-4222-8222-222222222222" } }, + } as any); + + await expect(f.provisioner.start(input)).rejects.toThrow("mismatched provisioning identity"); + + expect(f.control.provisionCredential).not.toHaveBeenCalled(); + expect(f.guest).not.toHaveBeenCalled(); + expect(f.machines.destroyMachine).toHaveBeenCalledWith(input.app, machine.id); + }); + + it("rejects returned tenant or operation metadata that belongs to another provisioning request", async () => { + for (const metadata of [ + { ...machine.config.metadata, factorize_tenant_id: "another-tenant" }, + { ...machine.config.metadata, factorize_operation_id: "another-operation" }, + { ...machine.config.metadata, factorize_role: "job" }, + ]) { + const f = fixture(); + f.machines.createMachine.mockResolvedValue({ ...machine, config: { metadata } } as any); + + await expect(f.provisioner.start(input)).rejects.toThrow("mismatched provisioning identity"); + + expect(f.control.provisionCredential).not.toHaveBeenCalled(); + expect(f.guest).not.toHaveBeenCalled(); + expect(f.machines.destroyMachine).toHaveBeenCalledWith(input.app, machine.id); + } + }); + + it("fails closed when Fly omits any returned provisioning identity metadata", async () => { + for (const key of ["factorize_boot_epoch", "factorize_role", "factorize_tenant_id", "factorize_operation_id"]) { + const f = fixture(); + const metadata = { ...machine.config.metadata }; + delete (metadata as Record)[key]; + f.machines.createMachine.mockResolvedValue({ ...machine, config: { metadata } } as any); + + await expect(f.provisioner.start(input)).rejects.toThrow("mismatched provisioning identity"); + + expect(f.control.provisionCredential).not.toHaveBeenCalled(); + expect(f.guest).not.toHaveBeenCalled(); + expect(f.machines.destroyMachine).toHaveBeenCalledWith(input.app, machine.id); + } + }); + + it("revokes bootstrap authority and attempts Machine cleanup when explicit start fails", async () => { + const f = fixture(); + const startupFailure = new Error("local Fly fixture start failed"); + f.machines.startMachine.mockRejectedValue(startupFailure); + // Cleanup failure must not replace the original provisioning diagnosis. + f.machines.destroyMachine.mockRejectedValue(new Error("local Fly fixture cleanup failed")); + + await expect(f.provisioner.start(input)).rejects.toBe(startupFailure); + + expect(f.control.provisionCredential).toHaveBeenCalledOnce(); + expect(f.control.revokeCredential).toHaveBeenCalledWith(input.tenantId, "fly:factorize-proxies:machine-1"); + expect(f.control.revokeCredential.mock.invocationCallOrder[0]).toBeLessThan(f.machines.destroyMachine.mock.invocationCallOrder[0]!); + expect(f.machines.destroyMachine).toHaveBeenCalledWith(input.app, machine.id); + expect(f.guest).not.toHaveBeenCalled(); + }); + + it("rejects mutable images and unsafe control origins before provisioning", async () => { + const { machines, provisioner } = fixture(); + await expect(provisioner.start({ ...input, image: "ghcr.io/example/proxy:latest" })).rejects.toThrow("pinned by sha256 digest"); + await expect(provisioner.start({ ...input, controlOrigin: "http://factorize.example" })).rejects.toThrow("HTTPS origin"); + await expect(provisioner.start({ ...input, app: "invalid/app" })).rejects.toThrow("Fly app name"); + expect(machines.createMachine).not.toHaveBeenCalled(); + }); + + it("rejects unsupported Fly sizes before provisioning", async () => { + const { machines, provisioner } = fixture(); + await expect(provisioner.start({ ...input, size: "custom-3x" })).rejects.toThrow("Invalid credential proxy size"); + expect(machines.createMachine).not.toHaveBeenCalled(); + }); + + it.each([255, 8193, 512.5])("rejects unsupported proxy memory %s MiB before provisioning", async memoryMb => { + const { machines, provisioner } = fixture(); + await expect(provisioner.start({ ...input, memoryMb })).rejects.toThrow("between 256 and 8192 MiB"); + expect(machines.createMachine).not.toHaveBeenCalled(); + }); + + it("rejects a different digest-pinned image instead of treating any immutable image as reviewed", async () => { + const { machines, provisioner } = fixture(); + await expect(provisioner.start({ ...input, image: `ghcr.io/generationsoftware/credential-proxy@sha256:${"c".repeat(64)}` })) + .rejects.toThrow("configured reviewed image"); + expect(machines.createMachine).not.toHaveBeenCalled(); + }); +}); + +describe("Fly credential proxy guest adapter", () => { + it("sends the bootstrap credential as stdin to the reviewed startup command", async () => { + const calls: unknown[][] = []; + const spawnProcess = ((command: string, args: string[], options: { stdio: string[] }) => { + calls.push([command, args, options]); + const stdoutHandlers: Array<(chunk: Buffer) => void> = []; + const child: any = { pid: undefined, stdin: { end: (value: string) => { calls.push(["stdin", value]); } }, + stdout: { on: (_event: string, callback: (chunk: Buffer) => void) => stdoutHandlers.push(callback) }, + stderr: { on: () => undefined }, on: (event: string, callback: (code: number) => void) => { + if (event === "close") queueMicrotask(() => { stdoutHandlers.forEach(handler => handler(Buffer.from("Credential proxy started\n"))); callback(0); }); + }, kill: () => undefined }; + return child; + }) as any; + const guest = createFlyCredentialProxyGuest({ app: input.app, machine: machine.id, token: "server-token-fixture", spawnProcess }); + await guest.startProxy({ tenantId: input.tenantId, proxyIdentity: "fly:factorize-proxies:machine-1", bootEpoch: epoch, controlOrigin: input.controlOrigin, browserOrigin: `https://${input.app}.fly.dev`, credential: `fzc_${"c".repeat(43)}` }); + expect(calls[0]?.[1]).toContain("/usr/local/bin/factorize-proxy-startup"); + expect(String(calls[1]?.[1])).toContain(`"type":"register-boot"`); + expect(String(calls[1]?.[1])).toContain(`"credential":"fzc_${"c".repeat(43)}"`); + }); + + it("requests a VM CSR signature through the private guest channel and verifies both bindings", async () => { + const calls: unknown[][] = []; + const spawnProcess = ((command: string, args: string[], options: { stdio: string[] }) => { + calls.push([command, args, options]); + const stdoutHandlers: Array<(chunk: Buffer) => void> = []; + const child: any = { pid: undefined, stdin: { end: (value: string) => { calls.push(["stdin", value]); } }, + stdout: { on: (_event: string, callback: (chunk: Buffer) => void) => stdoutHandlers.push(callback) }, + stderr: { on: () => undefined }, on: (event: string, callback: (code: number) => void) => { + if (event === "close") queueMicrotask(() => { stdoutHandlers.forEach(handler => handler(Buffer.from(JSON.stringify({ + bootEpoch: epoch, vmIdentity: "fly:job-1", certificate: "-----BEGIN CERTIFICATE-----cert", trustBundle: "-----BEGIN CERTIFICATE-----ca", + }) + "\n"))); callback(0); }); + }, kill: () => undefined }; + return child; + }) as any; + const guest = createFlyCredentialProxyGuest({ app: input.app, machine: machine.id, token: "server-token-fixture", spawnProcess }); + await expect(guest.issueVmCertificate({ bootEpoch: epoch, vmIdentity: "fly:job-1", csr: `-----BEGIN CERTIFICATE REQUEST-----\n${"A".repeat(120)}\n-----END CERTIFICATE REQUEST-----` })) + .resolves.toEqual({ certificate: "-----BEGIN CERTIFICATE-----cert", trustBundle: "-----BEGIN CERTIFICATE-----ca" }); + expect(calls[0]?.[1]).toContain("/usr/local/bin/factorize-proxy-certify-vm"); + expect(String(calls[1]?.[1])).toContain("\"vmIdentity\":\"fly:job-1\""); + await expect(guest.issueVmCertificate({ bootEpoch: epoch, vmIdentity: "fly:job-1", csr: "bad" })).rejects.toThrow("Invalid VM certificate provisioning request"); + }); +}); diff --git a/packages/api/test/fly-job-execution-backend.test.ts b/packages/api/test/fly-job-execution-backend.test.ts new file mode 100644 index 00000000..5305f682 --- /dev/null +++ b/packages/api/test/fly-job-execution-backend.test.ts @@ -0,0 +1,287 @@ +import { describe, expect, it, vi } from "vitest"; +import { FlyJobExecutionBackend } from "../src/fly-job-execution-backend"; +import { agentDriver } from "../src/agent-driver"; + +const machine = { id: "machine-1", state: "created" }; +function fixture() { + const machines = { + findMachineByOperationId: vi.fn(async () => machine), + createMachine: vi.fn(async (_app: string, _config: any, _operationId: string) => machine), + inspectMachine: vi.fn(async () => machine), + startMachine: vi.fn(async () => undefined), + stopMachine: vi.fn(async () => undefined), + destroyMachine: vi.fn(async () => undefined), + }; + const guest = { + run: vi.fn(async (command: readonly string[], _input?: string | Uint8Array) => ({ + ok: true, exitCode: 0, + stdout: command.at(-1) === "--version" ? "codex-cli 0.167.1\n" : "started\n", stderr: "", + })), + upload: vi.fn(async () => undefined), + download: vi.fn(async (path: string) => new TextEncoder().encode(path.endsWith("results.json") ? '{"status":"succeeded"}' : "saved log")), + }; + const proxyGuest = { + run: vi.fn(async () => ({ ok: true, exitCode: 0, stdout: "Run credential delivered to proxy memory\n", stderr: "" })), + issueVmCertificate: vi.fn(async () => ({ certificate: "-----BEGIN CERTIFICATE-----\npublic\n-----END CERTIFICATE-----", trustBundle: "-----BEGIN CERTIFICATE-----\nroot\n-----END CERTIFICATE-----" })), + }; + const backend = new FlyJobExecutionBackend({ app: "factorize-test", image: "ghcr.io/acme/code:v1", region: "iad", size: "shared-cpu-1x", machines, guest: () => guest, + proxyIdentity: "fly:factorize-test:proxy-1", proxyBootEpoch: "22222222-2222-4222-8222-222222222222", proxyGuest }); + return { backend, machines, guest, proxyGuest }; +} + +describe("Fly job execution backend", () => { + it("reserves a stable launch identity and delivers code payload after starting the Machine", async () => { + const f = fixture(); + const runId = "11111111-1111-4111-8111-111111111111"; + const reservation = f.backend.reserve!(runId); + const receipt = await f.backend.launch({ runId, reservation, prompt: "", code: { script: "echo ok", context: { value: 1 } } }); + expect(f.machines.createMachine).toHaveBeenCalledWith("factorize-test", expect.objectContaining({ + image: "ghcr.io/acme/code:v1", + region: "iad", + guest: { cpu_kind: "shared", cpus: 1, memory_mb: 1024 }, + auto_start: false, + restart: { policy: "no" }, + }), runId); + expect(f.machines.startMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + expect(f.guest.run).toHaveBeenNthCalledWith(1, ["/bin/sh", "-c", expect.stringContaining("factorize-job")], expect.stringContaining("#!/usr/bin/env python3")); + expect(f.guest.run).toHaveBeenNthCalledWith(2, ["/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "launch"], expect.stringContaining('"script":"echo ok"')); + expect(receipt).toMatchObject({ handle: { backendKind: "fly", id: "factorize-test/machine-1" }, observation: { state: "running" } }); + expect(JSON.stringify(f.machines.createMachine.mock.calls)).not.toContain("echo ok"); + }); + + it("reports a nonzero Fly guest process exit as a failed run", async () => { + const f = fixture(); + f.machines.inspectMachine.mockResolvedValue({ ...machine, state: "started" }); + f.guest.run.mockResolvedValue({ + ok: true, + exitCode: 0, + stdout: JSON.stringify({ state: "exited", exitCode: 7 }), + stderr: "", + }); + + await expect(f.backend.inspect({ + backendKind: "fly", + id: "factorize-test/machine-1", + })).resolves.toMatchObject({ + state: "failed", + detail: "The guest job process exited unsuccessfully", + failure: { source: "run", code: "process_failed" }, + }); + + expect(f.guest.run).toHaveBeenCalledWith([ + "/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "status", + ]); + }); + + it("uses the selected custom image, region, and size for Codex launches", async () => { + const f = fixture(); + const backend = new FlyJobExecutionBackend({ + app: "factorize-test", image: "registry.example/factorize-codex:reviewed", + region: "ord", size: "performance-2x", machines: f.machines, guest: () => f.guest, + }); + const runId = "11111111-1111-4111-8111-111111111111"; + const harness = agentDriver("codex").launch(runId, { model: "gpt-5.5" }); + + const receipt = await backend.launch({ runId, prompt: "continue", harness }); + + expect(f.guest.run).toHaveBeenCalledWith(["/usr/sbin/runuser", "-u", "exedev", "--", "codex", "--version"]); + expect(f.machines.createMachine).toHaveBeenCalledWith("factorize-test", expect.objectContaining({ + image: "registry.example/factorize-codex:reviewed", + region: "ord", + guest: { cpu_kind: "performance", cpus: 2, memory_mb: 2048 }, + auto_start: false, + restart: { policy: "no" }, + }), runId); + expect(f.guest.run).toHaveBeenLastCalledWith( + ["/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "launch"], + expect.stringContaining('"executable":"codex"'), + ); + expect(receipt.observation).toMatchObject({ state: "running" }); + }); + + it.each([ + { ok: false, exitCode: 127, stdout: "", stderr: "codex: not found" }, + { ok: true, exitCode: 0, stdout: "codex-cli 0.166.0\n", stderr: "" }, + ])("rejects a custom Codex image without the supported CLI version", async version => { + const f = fixture(); + f.guest.run.mockResolvedValueOnce({ ok: true, exitCode: 0, stdout: "", stderr: "" }); + f.guest.run.mockResolvedValueOnce(version); + + const receipt = await f.backend.launch({ + runId: "11111111-1111-4111-8111-111111111111", + prompt: "continue", + harness: agentDriver("codex").launch("11111111-1111-4111-8111-111111111111", {}), + }); + + expect(receipt).toMatchObject({ + handle: { backendKind: "fly", id: "factorize-test/machine-1" }, + observation: { + state: "failed", + detail: "Fly guest requires codex-cli 0.167.1 for Codex jobs", + failure: { code: "launch_failed" }, + }, + }); + expect(f.guest.run).toHaveBeenCalledTimes(2); + expect(f.guest.run.mock.calls[1]?.[0]).toEqual(["/usr/sbin/runuser", "-u", "exedev", "--", "codex", "--version"]); + expect(receipt.observation.detail?.length).toBeLessThanOrEqual(120); + }); + + it("lets custom image setup finish before invoking the shared launcher without replacing its entrypoint", async () => { + const f = fixture(); + const runId = "11111111-1111-4111-8111-111111111111"; + const backend = new FlyJobExecutionBackend({ + app: "factorize-test", image: "registry.example/custom-code:reviewed", + region: "ord", size: "performance-2x", machines: f.machines, guest: () => f.guest, + }); + + const receipt = await backend.launch({ + runId, + prompt: "", + code: { script: "echo ready", context: {} }, + }); + + const machineConfig = f.machines.createMachine.mock.calls[0]![1]; + expect(machineConfig).toMatchObject({ + image: "registry.example/custom-code:reviewed", + region: "ord", + guest: { cpu_kind: "performance", cpus: 2, memory_mb: 2048 }, + auto_start: false, + restart: { policy: "no" }, + }); + // Leave cmd unset so Fly runs the image's own entrypoint during startup. + expect(machineConfig).not.toHaveProperty("cmd"); + expect(f.machines.startMachine.mock.invocationCallOrder[0]).toBeLessThan(f.guest.run.mock.invocationCallOrder[0]!); + expect(f.guest.run).toHaveBeenNthCalledWith( + 2, + ["/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "launch"], + expect.stringContaining('"script":"echo ready"'), + ); + expect(receipt.observation).toMatchObject({ state: "running" }); + }); + + it("retains provider cleanup ownership when guest startup fails", async () => { + const f = fixture(); + f.guest.run.mockResolvedValue({ ok: false, exitCode: 1, stdout: "", stderr: "failure" }); + const receipt = await f.backend.launch({ runId: "11111111-1111-4111-8111-111111111111", prompt: "" }); + expect(receipt.handle.id).toBe("factorize-test/machine-1"); + expect(receipt.observation).toMatchObject({ state: "failed", failure: { code: "launch_failed" } }); + }); + + it("returns a bounded prerequisite diagnosis and retains cleanup ownership for an incompatible custom image", async () => { + const f = fixture(); + f.guest.run.mockResolvedValueOnce({ ok: false, exitCode: 1, stdout: "", stderr: "python3: not found" }); + + const receipt = await f.backend.launch({ + runId: "11111111-1111-4111-8111-111111111111", + prompt: "", + }); + + expect(receipt.handle).toMatchObject({ backendKind: "fly", id: "factorize-test/machine-1" }); + expect(receipt.observation).toMatchObject({ + state: "failed", + detail: "Fly guest is missing python3, runuser, or the exedev user, or could not install the shared Factorize job launcher", + failure: { code: "launch_failed", reason: "Fly guest launch failed" }, + }); + expect(f.guest.run).toHaveBeenCalledOnce(); + expect(receipt.observation.detail).not.toContain("python3: not found"); + }); + + it("reports a bounded launcher error when a custom image entrypoint does not invoke factorize-job", async () => { + const f = fixture(); + f.guest.run.mockResolvedValueOnce({ ok: true, exitCode: 0, stdout: "", stderr: "" }); + f.guest.run.mockResolvedValueOnce({ ok: false, exitCode: 127, stdout: "", stderr: "factorize-job: command not found" }); + + const receipt = await f.backend.launch({ + runId: "11111111-1111-4111-8111-111111111111", + prompt: "", + }); + + expect(f.machines.startMachine).toHaveBeenCalledOnce(); + expect(f.guest.run).toHaveBeenCalledTimes(2); + expect(f.guest.run.mock.calls[1]?.[0]).toEqual([ + "/usr/sbin/runuser", "-u", "exedev", "--", "/usr/local/bin/factorize-job", "launch", + ]); + expect(receipt.observation).toMatchObject({ + state: "failed", + detail: "Fly guest did not confirm job launch", + failure: { code: "launch_failed", reason: "Fly guest launch failed" }, + }); + expect(receipt.observation.detail?.length).toBeLessThanOrEqual(120); + }); + + it("retains cleanup ownership when Fly starts a Machine but loses the start acknowledgement", async () => { + const f = fixture(); + f.machines.startMachine.mockRejectedValueOnce(new Error("Fly start response timed out")); + + const receipt = await f.backend.launch({ runId: "11111111-1111-4111-8111-111111111111", prompt: "" }); + + expect(receipt).toMatchObject({ + handle: { backendKind: "fly", id: "factorize-test/machine-1" }, + observation: { state: "failed", failure: { code: "launch_failed" } }, + }); + expect(f.guest.run).not.toHaveBeenCalled(); + + f.machines.inspectMachine.mockResolvedValueOnce({ ...machine, state: "started" }).mockResolvedValueOnce({ ...machine, state: "stopped" }); + await expect(f.backend.stop(receipt.handle)).resolves.toEqual({ state: "stopped" }); + + expect(f.machines.stopMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + expect(f.machines.destroyMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + }); + + it("delivers the scoped run credential to proxy memory before starting the VM and omits it from guest payload", async () => { + const f = fixture(); + const runId = "11111111-1111-4111-8111-111111111111"; + const credential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + await f.backend.launch({ runId, prompt: "task", runProxy: { targetOrigin: "https://factorize.example", credential } }); + expect(f.proxyGuest.run).toHaveBeenCalledOnce(); + expect(f.proxyGuest.run).toHaveBeenCalledWith(["/usr/local/bin/factorize-proxy-authorize-run"], expect.stringContaining(`"credential":"${credential}"`)); + expect(f.proxyGuest.run.mock.invocationCallOrder[0]).toBeLessThan(f.machines.startMachine.mock.invocationCallOrder[0]); + expect(f.proxyGuest.issueVmCertificate).toHaveBeenCalledWith(expect.objectContaining({ + bootEpoch: "22222222-2222-4222-8222-222222222222", vmIdentity: "fly:factorize-test:machine-1", + })); + expect(f.guest.upload).toHaveBeenCalledWith("/home/exedev/.factorize-job/proxy/tunnel.mjs", expect.any(Uint8Array)); + expect(f.guest.run.mock.calls.some(([command]) => command.includes("FACTORIZE_PROXY_URL=http://127.0.0.1:18180"))).toBe(true); + const payload = String(f.guest.run.mock.calls.at(-1)?.[1] ?? ""); + expect(payload).not.toContain(credential); + expect(payload).toContain(runId); + }); + + it("destroys a stopped Machine after confirming provider state", async () => { + const f = fixture(); + f.machines.inspectMachine.mockResolvedValue({ ...machine, state: "stopped" }); + await expect(f.backend.stop({ backendKind: "fly", id: "factorize-test/machine-1" })).resolves.toEqual({ state: "stopped" }); + expect(f.machines.stopMachine).not.toHaveBeenCalled(); + expect(f.machines.destroyMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + }); + + it("keeps a Machine when the provider has not confirmed it stopped", async () => { + const f = fixture(); + f.machines.inspectMachine.mockResolvedValueOnce({ ...machine, state: "started" }).mockResolvedValueOnce({ ...machine, state: "stopping" }); + await expect(f.backend.stop({ backendKind: "fly", id: "factorize-test/machine-1" })).resolves.toEqual({ state: "stopping" }); + expect(f.machines.stopMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + expect(f.machines.destroyMachine).not.toHaveBeenCalled(); + }); + + it("collects structured results and logs through the guest transport", async () => { + const f = fixture(), handle = { backendKind: "fly", id: "factorize-test/machine-1" }; + await expect(f.backend.readJobResult(handle)).resolves.toBe('{"status":"succeeded"}'); + await expect(f.backend.readHarnessLog(handle)).resolves.toBe("saved log"); + expect(f.guest.download).toHaveBeenCalledWith("/home/exedev/.factorize-job/results.json"); + }); + + it("terminates a run without destroying its Machine before collection", async () => { + const f = fixture(), handle = { backendKind: "fly", id: "factorize-test/machine-1" }; + f.machines.inspectMachine.mockResolvedValueOnce({ ...machine, state: "started" }).mockResolvedValueOnce({ ...machine, state: "stopped" }); + await expect(f.backend.terminate(handle)).resolves.toEqual({ state: "stopped" }); + expect(f.machines.stopMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + expect(f.machines.destroyMachine).not.toHaveBeenCalled(); + }); + + it("resolves a persisted provisional operation handle after scheduler recovery", async () => { + const f = fixture(), runId = "11111111-1111-4111-8111-111111111111"; + const handle = f.backend.reserve!(runId); + await expect(f.backend.inspect(handle)).resolves.toEqual({ state: "starting" }); + expect(f.machines.findMachineByOperationId).toHaveBeenCalledWith("factorize-test", runId); + expect(f.machines.inspectMachine).toHaveBeenCalledWith("factorize-test", "machine-1"); + }); +}); diff --git a/packages/api/test/fly-job-machine.test.ts b/packages/api/test/fly-job-machine.test.ts new file mode 100644 index 00000000..17c5e2ea --- /dev/null +++ b/packages/api/test/fly-job-machine.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import { flyJobMachineConfig } from "../src/fly-job-machine"; + +const input = { + runId: "0904d375-1c40-4274-8a0b-2b5f1d988799", + operationId: "run:0904d375-1c40-4274-8a0b-2b5f1d988799", + image: "registry.example.org/jobs/codex:stable", + region: "iad", + size: "shared-cpu-2x", +}; + +describe("Fly job Machine configuration", () => { + it("records run ownership, uses the selected size, and disables provider restarts", () => { + expect(flyJobMachineConfig("factorize-user", input)).toEqual({ + image: input.image, + region: "iad", + guest: { cpu_kind: "shared", cpus: 2, memory_mb: 2048 }, + metadata: { + factorize_role: "job", + factorize_run_id: input.runId, + factorize_operation_id: input.operationId, + }, + auto_start: false, + auto_stop: false, + restart: { policy: "no" }, + }); + }); + + it("rejects malformed ownership and machine sizing before a provider call", () => { + expect(() => flyJobMachineConfig("invalid/app", input)).toThrow("Invalid Fly app name"); + expect(() => flyJobMachineConfig("factorize-user", { ...input, runId: "../other" })).toThrow("Invalid Fly run identity"); + expect(() => flyJobMachineConfig("factorize-user", { ...input, size: "large" })).toThrow("Invalid Fly execution size"); + }); + + it("never places the Fly API credential or Factorize run credential into config", () => { + const serialized = JSON.stringify(flyJobMachineConfig("factorize-user", input)); + expect(serialized).not.toContain("FACTORIZE_API_TOKEN"); + expect(serialized).not.toContain("fzt_run_"); + }); +}); diff --git a/packages/api/test/fly-machines-client.test.ts b/packages/api/test/fly-machines-client.test.ts index 9b5bf80c..60ab6d26 100644 --- a/packages/api/test/fly-machines-client.test.ts +++ b/packages/api/test/fly-machines-client.test.ts @@ -42,6 +42,7 @@ describe("bounded Fly Machines client", () => { let raw = ""; for await (const chunk of req) raw += chunk; const body = JSON.parse(raw); expect(body.config.metadata.factorize_operation_id).toBe("op-1"); + expect(body.config.restart).toEqual({ policy: "no" }); return json(res, 200, machine); } if (req.method === "GET" && req.url === "/v1/apps/app-a/machines/m-1") return json(res, 200, machine); @@ -53,7 +54,7 @@ describe("bounded Fly Machines client", () => { json(res, 404, { error: "not found" }); }); const client = createFlyMachinesClient({ token, apiOrigin: origin, wait: async () => undefined }); - const created = await client.createMachine("app-a", { image: "example/image", metadata: { keep: "yes" } }, "op-1"); + const created = await client.createMachine("app-a", { image: "example/image", restart: { policy: "always" }, metadata: { keep: "yes" } }, "op-1"); expect(created.id).toBe("m-1"); expect((await client.inspectMachine("app-a", "m-1")).id).toBe("m-1"); await client.startMachine("app-a", "m-1"); diff --git a/packages/api/test/fly-run-scheduler-lifecycle.test.ts b/packages/api/test/fly-run-scheduler-lifecycle.test.ts new file mode 100644 index 00000000..f7de4679 --- /dev/null +++ b/packages/api/test/fly-run-scheduler-lifecycle.test.ts @@ -0,0 +1,220 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { terminalDiagnostics } from "../src/harness-diagnostics"; +import { RunScheduler } from "../src/postgres/run-scheduler"; +import { RunRepository } from "../src/postgres/run-repository"; +import { ArtifactRepository } from "../src/postgres/artifact-repository"; +import { FlyJobExecutionBackend } from "../src/fly-job-execution-backend"; +import { FlyClientError, type FlyMachine } from "../src/fly-machines-client"; + +afterEach(() => { vi.restoreAllMocks(); vi.unstubAllGlobals(); }); + +function fixture() { + const events: string[] = [], stored: any[] = []; + let observation: any; + const machine = { backendKind: "fly", id: "factorize-test/machine-1" }; + const backend = { + kind: "fly", capabilities: ["stop", "recovery"], + inspect: vi.fn(async () => ({ state: "succeeded" })), + terminate: vi.fn(async () => { events.push("terminate"); return { state: "stopped" }; }), + stop: vi.fn(async () => { events.push("destroy"); return { state: "stopped" }; }), + readJobResult: vi.fn(async () => { events.push("result"); return '{"status":"succeeded"}'; }), + readHarnessLog: vi.fn(async () => { events.push("log"); return "Fly guest log\n"; }), + listJobFiles: vi.fn(async () => []), + collectJobFile: vi.fn(), + collectArtifact: vi.fn(), + }; + const run: any = { + tenantId: "tenant", id: "run", jobId: "job", state: "stopping", + executionConfig: { executionProvider: "fly", executionType: "code" }, + executionHandle: machine, resultProtocolVersion: 1, finalizationAttempt: 0, + }; + vi.spyOn(RunRepository.prototype, "recordStopAttempt").mockResolvedValue(); + vi.spyOn(RunRepository.prototype, "recordObservation").mockImplementation(async (_run, value) => { + events.push("observation"); observation ??= terminalDiagnostics(value); return observation; + }); + vi.spyOn(RunRepository.prototype, "recordJobResult").mockImplementation(async () => { events.push("persist-result"); }); + vi.spyOn(RunRepository.prototype, "terminal").mockImplementation(async () => { events.push("terminal"); }); + vi.spyOn(RunRepository.prototype, "retryFinalization").mockImplementation(async () => { run.finalizationAttempt++; events.push("retry"); }); + vi.spyOn(ArtifactRepository.prototype, "list").mockImplementation(async () => stored); + vi.spyOn(ArtifactRepository.prototype, "record").mockImplementation(async item => { stored.push({ ...item, state: "stored" }); }); + const scheduler: any = new RunScheduler({} as any, { + SESSION_SIGNING_SECRET: "test-signing-secret", APP_ORIGIN: "https://app.example.test", + RUN_ARTIFACTS: { put: async () => { events.push("store-log"); } }, + } as any); + vi.spyOn(scheduler, "backend").mockResolvedValue(backend); + return { scheduler, run, backend, events, stored }; +} + +function localFlyFixture(options: { ambiguousCreate?: boolean; failFirstInspect?: boolean; failFirstResult?: boolean; failFirstDestroy?: boolean; jobFiles?: string[] } = {}) { + const events: string[] = []; + const runId = "11111111-1111-4111-8111-111111111111"; + let machine: FlyMachine | undefined = options.ambiguousCreate ? undefined : { + id: "machine-1", state: "started", config: { metadata: { factorize_operation_id: runId } }, + }; + let resultReads = 0; + let inspections = 0; + let destroys = 0; + const machines = { + findMachineByOperationId: vi.fn(async (_app: string, operationId: string) => + machine?.config?.metadata?.factorize_operation_id === operationId ? machine : undefined), + createMachine: vi.fn(async (_app: string, config: any, operationId: string) => { + events.push("create"); + machine = { id: "machine-1", state: "created", config: { ...config, metadata: { ...config.metadata, factorize_operation_id: operationId } } }; + if (options.ambiguousCreate) throw new FlyClientError("Fly Machine creation outcome is ambiguous", "ambiguous_create"); + return machine; + }), + inspectMachine: vi.fn(async () => { + if (options.failFirstInspect && inspections++ === 0) { events.push("inspect-timeout"); throw new FlyClientError("Fly request timed out", "timeout"); } + if (!machine) throw new Error("Machine is missing"); + events.push(`inspect:${machine.state}`); return machine; + }), + startMachine: vi.fn(async () => { if (machine) machine.state = "started"; events.push("start"); }), + stopMachine: vi.fn(async () => { if (machine) machine.state = "stopped"; events.push("stop"); }), + destroyMachine: vi.fn(async () => { + events.push("destroy"); + if (options.failFirstDestroy && destroys++ === 0) throw new Error("local Fly fixture destroy response timed out"); + if (machine) machine.state = "destroyed"; + }), + }; + const guest = { + run: vi.fn(async (command: readonly string[]) => { + if (command.at(-1) === "status") return { ok: true, exitCode: 0, stdout: JSON.stringify({ state: "exited", exitCode: 0 }), stderr: "" }; + if (command.at(-1)?.startsWith("find ")) return { ok: true, exitCode: 0, stdout: (options.jobFiles ?? []).map(path => `${path}\t4`).join("\n"), stderr: "" }; + return { ok: true, exitCode: 0, stdout: "", stderr: "" }; + }), + download: vi.fn(async (path: string) => { + events.push(path.endsWith("results.json") ? "result" : "log"); + if (path.endsWith("results.json") && options.failFirstResult && resultReads++ === 0) throw new Error("local guest result transport timed out"); + if (path.endsWith("report.json")) events.push("download-job-file"); + return new TextEncoder().encode(path.endsWith("results.json") ? '{"status":"succeeded"}' : "local Fly guest log\n"); + }), + }; + const backend = new FlyJobExecutionBackend({ app: "factorize-test", image: "local/image", region: "iad", size: "shared-cpu-1x", machines, guest: () => guest }); + return { backend, machines, guest, events }; +} + +function schedulerForLocalFly(backend: FlyJobExecutionBackend) { + const stored: any[] = [], events: string[] = []; + const run: any = { + tenantId: "tenant", id: "11111111-1111-4111-8111-111111111111", jobId: "job", state: "running", + executionConfig: { executionProvider: "fly", executionType: "code" }, + executionHandle: { backendKind: "fly", id: "factorize-test/machine-1", operationId: "11111111-1111-4111-8111-111111111111" }, + resultProtocolVersion: 1, finalizationAttempt: 0, + }; + vi.spyOn(RunRepository.prototype, "recordObservation").mockImplementation(async (_run, value) => { + events.push("observation"); + return terminalDiagnostics(value); + }); + vi.spyOn(RunRepository.prototype, "recordJobResult").mockImplementation(async () => { events.push("persist-result"); }); + vi.spyOn(RunRepository.prototype, "terminal").mockImplementation(async () => { events.push("terminal"); }); + vi.spyOn(RunRepository.prototype, "retryFinalization").mockImplementation(async () => { run.finalizationAttempt++; events.push("retry"); }); + vi.spyOn(RunRepository.prototype, "markLaunched").mockImplementation(async (_run, handle) => { run.executionHandle = handle; events.push("mark-launched"); }); + vi.spyOn(RunRepository.prototype, "schedulePoll").mockImplementation(async () => { events.push("schedule-poll"); }); + vi.spyOn(ArtifactRepository.prototype, "list").mockImplementation(async () => stored); + vi.spyOn(ArtifactRepository.prototype, "record").mockImplementation(async item => { stored.push({ ...item, state: "stored" }); }); + const scheduler: any = new RunScheduler({} as any, { + SESSION_SIGNING_SECRET: "test-signing-secret", APP_ORIGIN: "https://app.example.test", + RUN_ARTIFACTS: { put: async () => { events.push("store-log"); } }, + } as any); + vi.spyOn(scheduler, "backend").mockResolvedValue(backend); + return { scheduler, run, stored, events }; +} + +describe("Fly scheduler lifecycle", () => { + it("collects a cancelled guest's log before destroying its Machine", async () => { + const f = fixture(); + await f.scheduler.poll(f.run).catch((error: unknown) => f.scheduler.fail(f.run, error, "poll")); + expect(f.events).toEqual(["terminate", "observation", "log", "store-log", "destroy", "terminal"]); + expect(f.backend.stop).toHaveBeenCalledWith(f.run.executionHandle); + expect(f.scheduler.runs.terminal).toHaveBeenCalledWith(f.run, "stopped", "stored", undefined, true, undefined); + }); + + it("keeps a Fly Machine available when result collection needs a retry", async () => { + const f = fixture(); + f.run.state = "running"; + f.backend.readJobResult.mockImplementationOnce(async () => { f.events.push("result"); throw new Error("guest transport unavailable"); }); + await f.scheduler.poll(f.run).catch((error: unknown) => f.scheduler.fail(f.run, error, "poll")); + expect(f.events).toEqual(["observation", "result", "retry"]); + expect(f.backend.stop).not.toHaveBeenCalled(); + expect(f.run.finalizationAttempt).toBe(1); + }); + + it("recovers an ambiguously timed out create through the persisted scheduler operation handle", async () => { + const fly = localFlyFixture({ ambiguousCreate: true }); + const f = schedulerForLocalFly(fly.backend); + f.run.state = "starting"; + const reservation = fly.backend.reserve(f.run.id); + vi.spyOn(RunRepository.prototype, "recordLaunchHandle").mockImplementation(async (_run, handle) => { f.run.executionHandle = handle; }); + + await f.scheduler.runs.recordLaunchHandle(f.run, reservation); + await expect(fly.backend.launch({ runId: f.run.id, reservation, prompt: "" })).rejects.toMatchObject({ kind: "ambiguous_create" }); + await f.scheduler.fail(f.run, new FlyClientError("Fly Machine creation outcome is ambiguous", "ambiguous_create"), "launch"); + await f.scheduler.poll(f.run); + + expect(fly.machines.createMachine).toHaveBeenCalledTimes(1); + expect(fly.machines.findMachineByOperationId).toHaveBeenCalledWith("factorize-test", f.run.id); + expect(f.run.executionHandle).toEqual(reservation); + expect(f.events).toContain("schedule-poll"); + }); + + it("retries a transient Fly inspection timeout without losing or destroying the Machine", async () => { + const fly = localFlyFixture({ failFirstInspect: true }); + const f = schedulerForLocalFly(fly.backend); + await f.scheduler.poll(f.run).catch((error: unknown) => f.scheduler.fail(f.run, error, "poll")); + + expect(f.events).toEqual(["schedule-poll"]); + expect(fly.events).toEqual(["inspect-timeout"]); + expect(fly.machines.destroyMachine).not.toHaveBeenCalled(); + + await f.scheduler.poll(f.run); + expect(fly.events).toContain("result"); + expect(fly.machines.destroyMachine).toHaveBeenCalledTimes(1); + expect(f.events.at(-1)).toBe("terminal"); + }); + + it("retries structured result collection on the same Fly Machine before cleanup", async () => { + const fly = localFlyFixture({ failFirstResult: true }); + const f = schedulerForLocalFly(fly.backend); + await f.scheduler.poll(f.run).catch((error: unknown) => f.scheduler.fail(f.run, error, "poll")); + + expect(fly.events).toEqual(["inspect:started", "result"]); + expect(f.events).toEqual(["observation", "retry"]); + expect(fly.machines.destroyMachine).not.toHaveBeenCalled(); + expect(f.run.finalizationAttempt).toBe(1); + + await f.scheduler.poll(f.run); + expect(fly.events).toContain("destroy"); + expect(fly.events.filter(event => event === "result")).toHaveLength(2); + expect(f.events).toContain("terminal"); + }); + + it("recovers a timed out Fly destroy during finalization without losing collected results", async () => { + const fly = localFlyFixture({ failFirstDestroy: true }); + const f = schedulerForLocalFly(fly.backend); + await f.scheduler.poll(f.run); + expect(fly.events.indexOf("result")).toBeLessThan(fly.events.indexOf("destroy")); + expect(f.events).toEqual(["observation", "persist-result", "store-log", "retry"]); + expect(f.run.finalizationAttempt).toBe(1); + + await f.scheduler.poll(f.run); + expect(fly.machines.destroyMachine).toHaveBeenCalledTimes(2); + expect(f.events.at(-1)).toBe("terminal"); + }); + + it("collects and uploads Fly job files before destroying the Machine", async () => { + const fly = localFlyFixture({ jobFiles: ["report.json"] }); + const f = schedulerForLocalFly(fly.backend); + vi.stubGlobal("fetch", vi.fn(async () => { + fly.events.push("upload-job-file"); + return new Response(null, { status: 200 }); + })); + + await f.scheduler.poll(f.run); + + expect(fly.events.indexOf("download-job-file")).toBeGreaterThanOrEqual(0); + expect(fly.events.indexOf("download-job-file")).toBeLessThan(fly.events.indexOf("destroy")); + expect(fly.events.indexOf("upload-job-file")).toBeLessThan(fly.events.indexOf("destroy")); + expect(fly.guest.download).toHaveBeenCalledWith("/home/exedev/.factorize-job/report.json"); + expect(f.events.at(-1)).toBe("terminal"); + }); +}); diff --git a/packages/api/test/proxy-browser-capability.test.ts b/packages/api/test/proxy-browser-capability.test.ts new file mode 100644 index 00000000..3729c98a --- /dev/null +++ b/packages/api/test/proxy-browser-capability.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it } from "vitest"; +import { generateKeyPairSync, verify } from "node:crypto"; +import { issueModelAccountCapability } from "../src/proxy-browser-capability"; + +describe("proxy browser capabilities", () => { + it("signs an account/action/origin/boot-bound one-minute EdDSA token", async () => { + const { privateKey, publicKey } = generateKeyPairSync("ed25519"); + const token = await issueModelAccountCapability({ + privateKeyPem: privateKey.export({ type: "pkcs8", format: "pem" }).toString(), + proxyIdentity: "fly:proxy:machine-1", ownerId: "11111111-1111-4111-8111-111111111111", + accountId: "22222222-2222-4222-8222-222222222222", action: "authorize", + appOrigin: "https://factorize.example", generation: "33333333-3333-4333-8333-333333333333", + now: 1_900_000_000, jti: "one-use-fixture-nonce-123456", + }); + const [header, claims, signature] = token.split("."); + expect(JSON.parse(Buffer.from(header!, "base64url").toString())).toEqual({ alg: "EdDSA", typ: "JWT" }); + expect(JSON.parse(Buffer.from(claims!, "base64url").toString())).toMatchObject({ + iss: "factorize", aud: "fly:proxy:machine-1", sub: "11111111-1111-4111-8111-111111111111", + action: "model-account:authorize", accountId: "22222222-2222-4222-8222-222222222222", + origin: "https://factorize.example", generation: "33333333-3333-4333-8333-333333333333", + exp: 1_900_000_060, jti: "one-use-fixture-nonce-123456", + }); + expect(verify(null, Buffer.from(`${header}.${claims}`), publicKey, Buffer.from(signature!, "base64url"))).toBe(true); + }); +}); diff --git a/packages/api/test/proxy-control-repository.test.ts b/packages/api/test/proxy-control-repository.test.ts new file mode 100644 index 00000000..b622e0df --- /dev/null +++ b/packages/api/test/proxy-control-repository.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it, vi } from "vitest"; +import { ProxyControlRepository } from "../src/postgres/proxy-control-repository"; + +function database(query: ReturnType, transactionQuery = query) { + return { pool: { query }, transaction: (work: (client: any) => Promise) => work({ query: transactionQuery }) } as any; +} + +describe("dedicated credential proxy control identity", () => { + it("issues a scoped one-time secret and stores only its digest", async () => { + const query = vi.fn().mockResolvedValue({ rows: [], rowCount: 1 }); + const repository = new ProxyControlRepository(database(query)); + const token = await repository.provisionCredential("11111111-1111-4111-8111-111111111111", "proxy-a", "22222222-2222-4222-8222-222222222222"); + expect(token).toMatch(/^fzc_[A-Za-z0-9_-]{43}$/); + const stored = query.mock.calls[0][1][3] as Buffer; + expect(stored).toBeInstanceOf(Buffer); + expect(stored.toString()).not.toContain(token); + expect(stored.byteLength).toBe(32); + }); + + it("authenticates only a live credential and never accepts a guest run token", async () => { + const query = vi.fn().mockResolvedValue({ rows: [{ tenant_id: "tenant-a", proxy_identity: "proxy-a" }], rowCount: 1 }); + const repository = new ProxyControlRepository(database(query)); + expect(await repository.authenticate("fzt_run_" + "a".repeat(43))).toBeNull(); + expect(query).not.toHaveBeenCalled(); + expect(await repository.authenticate("fzc_" + "a".repeat(43))).toEqual({ tenantId: "tenant-a", proxyIdentity: "proxy-a" }); + }); + + it("registers a fresh boot for its credential identity and rejects revoked, disabled, or mismatched identities", async () => { + const identity = { tenant_id: "tenant-a", proxy_identity: "proxy-a" }; + const credentialQuery = vi.fn().mockResolvedValue({ rows: [identity], rowCount: 1 }); + const stateQuery = vi.fn().mockResolvedValue({ rows: [{ tenant_id: identity.tenant_id }], rowCount: 1 }); + const repository = new ProxyControlRepository(database(vi.fn(), vi.fn())); + const tx = { query: vi.fn().mockImplementation((sql: string) => sql.includes("credential_proxy_control_credentials") ? credentialQuery() : sql.includes("INSERT INTO app.credential_proxy_states") ? stateQuery() : Promise.resolve({ rows: [], rowCount: 1 })) }; + (repository as any).database = { transaction: (work: (client: any) => Promise) => work(tx) }; + const epoch = "22222222-2222-4222-8222-222222222222"; + const registration = await repository.registerBoot("fzc_" + "a".repeat(43), epoch, "https://proxy.example"); + expect(registration).toMatchObject({ tenantId: identity.tenant_id, proxyIdentity: identity.proxy_identity, bootEpoch: epoch }); + expect(registration?.controlCredential).toMatch(/^fzc_[A-Za-z0-9_-]{43}$/); + expect(tx.query.mock.calls.map(call => call[0])).toHaveLength(5); + expect(tx.query.mock.calls[0][0]).toContain("boot_epoch=$2 AND revoked_at IS NULL AND consumed_at IS NULL"); + expect(tx.query.mock.calls[2][0]).toContain("consumed_at=now()"); + expect(tx.query.mock.calls[3][0]).toContain("boot_epoch<>$3"); + expect(tx.query.mock.calls[4][0]).toContain("credential_proxy_browser_pairings"); + credentialQuery.mockResolvedValueOnce({ rows: [], rowCount: 0 }); + expect(await repository.registerBoot("fzc_" + "a".repeat(43), epoch, "https://proxy.example")).toBeNull(); + }); + + it("rejects malformed boot epochs before opening a transaction", async () => { + const transaction = vi.fn(); + const repository = new ProxyControlRepository({ transaction, pool: { query: vi.fn() } } as any); + await expect(repository.registerBoot("fzc_" + "a".repeat(43), "old-boot", "https://proxy.example")).rejects.toThrow("bootEpoch must be a UUID"); + expect(transaction).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/api/test/proxy-control.test.ts b/packages/api/test/proxy-control.test.ts new file mode 100644 index 00000000..4663a104 --- /dev/null +++ b/packages/api/test/proxy-control.test.ts @@ -0,0 +1,211 @@ +import { describe, expect, it, vi } from "vitest"; +import { proxyControlFetch } from "../src/proxy-control"; +import { encrypt } from "../src/crypto"; + +const token = `fzc_${"A".repeat(43)}`; +const bootEpoch = "22222222-2222-4222-8222-222222222222"; +const runId = "33333333-3333-4333-8333-333333333333"; +const runCredential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + +function envFor({ credentialFound = true, stateSaved = true } = {}) { + const transactionQuery = vi.fn(async (sql: string) => { + if (sql.includes("credential_proxy_control_credentials")) return { rows: credentialFound ? [{ tenant_id: "11111111-1111-4111-8111-111111111111", proxy_identity: "proxy-fixture" }] : [], rowCount: credentialFound ? 1 : 0 }; + if (sql.includes("INSERT INTO app.credential_proxy_states")) return { rows: stateSaved ? [{ tenant_id: "11111111-1111-4111-8111-111111111111" }] : [], rowCount: stateSaved ? 1 : 0 }; + return { rows: [], rowCount: 1 }; + }); + const query = vi.fn(async (sql: string) => sql.includes("proxy_token_digest") + ? { rows: credentialFound ? [{ tenant_id: "11111111-1111-4111-8111-111111111111", proxy_identity: "proxy-fixture", boot_epoch: bootEpoch }] : [], rowCount: credentialFound ? 1 : 0 } + : sql.includes("SET unlocked=$3") ? { rows: [], rowCount: credentialFound ? 1 : 0 } + : sql.includes("FROM app.run_credentials") + ? { rows: credentialFound ? [{ tenant_id: "11111111-1111-4111-8111-111111111111", user_id: "owner-fixture", session_version: 1, job_id: "job-fixture", scopes: ["integrations:read"] }] : [], rowCount: credentialFound ? 1 : 0 } + : { rows: [], rowCount: 0 }); + return { DATABASE: { pool: { query }, transaction: (work: (client: any) => Promise) => work({ query: transactionQuery }) }, transactionQuery, query } as any; +} + +function request({ authorization, body = { bootEpoch, browserOrigin: "https://proxy.example" }, method = "POST", path = "/internal/credential-proxy/register-boot" }: { authorization?: string; body?: unknown; method?: string; path?: string } = {}) { + return new Request(`https://factorize.example${path}`, { method, headers: { "Content-Type": "application/json", ...(authorization ? { Authorization: authorization } : {}) }, body: method === "POST" ? JSON.stringify(body) : undefined }); +} + +describe("dedicated proxy boot registration endpoint", () => { + it("binds registration to the proxy identity owned by its distinct control credential", async () => { + const env = envFor(); + const response = await proxyControlFetch(request({ authorization: `Bearer ${token}` }), env); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ registered: true, tenantId: "11111111-1111-4111-8111-111111111111", proxyIdentity: "proxy-fixture", bootEpoch, controlCredential: expect.stringMatching(/^fzc_[A-Za-z0-9_-]{43}$/) }); + expect(env.transactionQuery.mock.calls.map((call: any[]) => call[0])).toHaveLength(5); + expect(env.transactionQuery.mock.calls[0][0]).toContain("boot_epoch=$2 AND revoked_at IS NULL AND consumed_at IS NULL"); + expect(env.transactionQuery.mock.calls[2][0]).toContain("consumed_at=now()"); + expect(env.transactionQuery.mock.calls[4][0]).toContain("credential_proxy_browser_pairings"); + }); + + it("rejects guest job tokens before database access", async () => { + const env = envFor(); + const response = await proxyControlFetch(request({ authorization: `Bearer fzt_run_${"B".repeat(32)}` }), env); + expect(response.status).toBe(401); + expect(env.transactionQuery).not.toHaveBeenCalled(); + }); + + it("accepts runtime status only from the current registered proxy credential", async () => { + const env = envFor(); + const response = await proxyControlFetch(request({ authorization: `Bearer ${token}`, path: "/internal/credential-proxy/status", body: { bootEpoch, unlocked: true } }), env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ updated: true }); + expect(env.query.mock.calls.some((call: any[]) => call[0].includes("SET unlocked=$3") && call[0].includes("boot_epoch=$4"))).toBe(true); + expect((await proxyControlFetch(request({ authorization: `Bearer ${token}`, path: "/internal/credential-proxy/status", body: { bootEpoch, unlocked: "yes" } }), env)).status).toBe(400); + }); + + it("revokes delivered proxy-memory credentials for runs that ended before first tunnel enrollment", async () => { + const env = envFor(); + env.query.mockImplementation(async (sql: string) => sql.includes("proxy_token_digest") + ? { rows: [{ tenant_id: "11111111-1111-4111-8111-111111111111", proxy_identity: "proxy-fixture", boot_epoch: bootEpoch }], rowCount: 1 } + : sql.includes("FROM app.credential_run_grants") + ? { rows: [{ run_id: runId }], rowCount: 1 } + : { rows: [], rowCount: 0 }); + const response = await proxyControlFetch(request({ + authorization: `Bearer ${token}`, + path: "/internal/credential-proxy/revocations", + body: { bootEpoch }, + }), env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ revokedRunIds: [runId], lockProxy: true }); + const revocationsQuery = env.query.mock.calls.find((call: any[]) => call[0].includes("FROM app.credential_run_grants"))?.[0] as string; + expect(revocationsQuery).toContain("JOIN app.job_runs"); + expect(revocationsQuery).toContain("r.state IN ('succeeded','failed','stopped')"); + }); + + it("rejects revoked credentials, disabled proxy identities and tenant or proxy substitution", async () => { + for (const env of [envFor({ credentialFound: false }), envFor({ stateSaved: false })]) { + const response = await proxyControlFetch(request({ authorization: `Bearer ${token}` }), env); + expect(response.status).toBe(403); + } + }); + + it("validates the endpoint, method and exact boot payload", async () => { + const env = envFor(); + expect((await proxyControlFetch(new Request("https://factorize.example/internal/credential-proxy/register-boot", { method: "GET" }), env)).status).toBe(405); + expect((await proxyControlFetch(request({ authorization: `Bearer ${token}`, body: { bootEpoch, tenantId: "attacker-tenant" } }), env)).status).toBe(400); + expect(env.transactionQuery).not.toHaveBeenCalled(); + }); + + it("bounds registration payloads before parsing", async () => { + const env = envFor(); + const response = await proxyControlFetch(request({ authorization: `Bearer ${token}`, body: { bootEpoch, padding: "x".repeat(2_000) } }), env); + expect(response.status).toBe(413); + expect(env.transactionQuery).not.toHaveBeenCalled(); + }); + + it("issues a short-lived enrollment only through the registered proxy identity", async () => { + const env = envFor(); + env.transactionQuery.mockImplementation(async (sql: string) => sql.includes("SELECT g.grant_revision") + ? { rows: [{ grant_revision: 3 }], rowCount: 1 } + : sql.includes("INSERT INTO app.credential_run_grants") + ? { rows: [{ grant_revision: 3 }], rowCount: 1 } + : { rows: [], rowCount: 1 }); + const response = await proxyControlFetch(request({ + authorization: `Bearer ${token}`, + path: "/internal/credential-proxy/enroll", + body: { runId, vmIdentity: "fly:vm-123", runCredential }, + }), env); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ grantRevision: 3, enrollmentId: expect.any(String), token: expect.any(String) }); + expect(env.query).toHaveBeenCalled(); + expect(env.query.mock.calls.some((call: any[]) => call[0].includes("FROM app.run_credentials"))).toBe(true); + expect(env.transactionQuery.mock.calls.some((call: any[]) => call[0].includes("INSERT INTO app.credential_run_grants"))).toBe(true); + expect(env.transactionQuery.mock.calls[0][0]).toContain("JOIN app.credential_proxy_states"); + }); + + it("refuses enrollment when the run credential is absent, invalid, cross-tenant, mismatched, or lacks integration scope", async () => { + for (const [body, expectedStatus] of [ + [{ runId, vmIdentity: "fly:vm-123" }, 400], + [{ runId, vmIdentity: "fly:vm-123", runCredential: "fzt_run_invalid" }, 403], + [{ runId: "44444444-4444-4444-8444-444444444444", vmIdentity: "fly:vm-123", runCredential }, 403], + ] as const) { + const env = envFor(); + expect((await proxyControlFetch(request({ authorization: `Bearer ${token}`, path: "/internal/credential-proxy/enroll", body }), env)).status).toBe(expectedStatus); + expect(env.transactionQuery).not.toHaveBeenCalled(); + } + const crossTenant = envFor(); + crossTenant.query.mockImplementation(async (sql: string) => sql.includes("proxy_token_digest") + ? { rows: [{ tenant_id: "other-tenant", proxy_identity: "proxy-fixture", boot_epoch: bootEpoch }], rowCount: 1 } + : { rows: [], rowCount: 0 }); + expect((await proxyControlFetch(request({ authorization: `Bearer ${token}`, path: "/internal/credential-proxy/enroll", body: { runId, vmIdentity: "fly:vm-123", runCredential } }), crossTenant)).status).toBe(403); + const noScope = envFor(); + noScope.query.mockImplementation(async (sql: string) => sql.includes("proxy_token_digest") + ? { rows: [{ tenant_id: "11111111-1111-4111-8111-111111111111", proxy_identity: "proxy-fixture", boot_epoch: bootEpoch }], rowCount: 1 } + : sql.includes("FROM app.run_credentials") + ? { rows: [{ tenant_id: "11111111-1111-4111-8111-111111111111", user_id: "owner-fixture", session_version: 1, job_id: "job-fixture", scopes: ["runs:read"] }], rowCount: 1 } + : { rows: [], rowCount: 0 }); + expect((await proxyControlFetch(request({ authorization: `Bearer ${token}`, path: "/internal/credential-proxy/enroll", body: { runId, vmIdentity: "fly:vm-123", runCredential } }), noScope)).status).toBe(403); + expect(noScope.transactionQuery).not.toHaveBeenCalled(); + }); + + it("consumes enrollment against the authenticated proxy boot and rejects untrusted extra bindings", async () => { + const env = envFor(); + env.transactionQuery.mockImplementation(async (sql: string) => sql.includes("UPDATE app.credential_run_enrollments") + ? { rows: [{ id: "44444444-4444-4444-8444-444444444444", grant_revision: 3 }], rowCount: 1 } + : { rows: [], rowCount: 1 }); + const accepted = await proxyControlFetch(request({ + authorization: `Bearer ${token}`, + path: "/internal/credential-proxy/consume-enrollment", + body: { runId: "33333333-3333-4333-8333-333333333333", vmIdentity: "fly:vm-123", enrollmentToken: "e".repeat(43) }, + }), env); + expect(accepted.status).toBe(200); + expect(await accepted.json()).toMatchObject({ consumed: true, binding: { bootEpoch, proxyIdentity: "proxy-fixture", vmIdentity: "fly:vm-123" } }); + const rejected = await proxyControlFetch(request({ authorization: `Bearer ${token}`, path: "/internal/credential-proxy/enroll", body: { runId: "33333333-3333-4333-8333-333333333333", vmIdentity: "fly:vm-123", tenantId: "other" } }), env); + expect(rejected.status).toBe(400); + }); + + it("returns only the current run's selected user-encrypted bearer and model records to its registered proxy", async () => { + const integrationId = "55555555-5555-4555-8555-555555555555"; + const modelAccountId = "77777777-7777-4777-8777-777777777777"; + const key = Buffer.alloc(32, 7).toString("base64"); + const env = envFor(); + env.CREDENTIAL_ENCRYPTION_KEY = key; + env.query.mockImplementation(async (sql: string, values: unknown[] = []) => { + if (sql.includes("proxy_token_digest")) return { rows: [{ tenant_id: "11111111-1111-4111-8111-111111111111", proxy_identity: "proxy-fixture", boot_epoch: bootEpoch }], rowCount: 1 }; + if (sql.includes("FROM app.runs r")) return { rows: [{ execution_config: { portableIntegrationIds: [integrationId, "99999999-9999-4999-8999-999999999999"], modelAccess: "portable", modelAccountId }, grant_revision: 3 }], rowCount: 1 }; + if (sql.includes("FROM app.integration_records")) return values[1] === "bearer" ? { rows: [ + { id: integrationId, kind: "bearer", name: "selected", origin: "https://api.example", description: null, revision: 4, encrypted_payload: await encrypt("user-ciphertext", key), deleted_at: null }, + { id: "99999999-9999-4999-8999-999999999999", kind: "bearer", name: "deleted", origin: "https://deleted.example", description: null, revision: 8, encrypted_payload: null, deleted_at: new Date("2026-01-01T00:00:00Z") }, + { id: "66666666-6666-4666-8666-666666666666", kind: "bearer", name: "not-selected", origin: "https://other.example", description: null, revision: 5, encrypted_payload: await encrypt("other-ciphertext", key) }, + ], rowCount: 2 } : { rows: [ + { id: modelAccountId, kind: "model", name: "selected-model", origin: null, description: "fixture account", revision: 6, encrypted_payload: await encrypt("model-user-ciphertext", key) }, + { id: "88888888-8888-4888-8888-888888888888", kind: "model", name: "other-model", origin: null, description: null, revision: 7, encrypted_payload: await encrypt("other-model-ciphertext", key) }, + ], rowCount: 2 }; + return { rows: [], rowCount: 0 }; + }); + const response = await proxyControlFetch(request({ + authorization: `Bearer ${token}`, + path: "/internal/credential-proxy/integration-snapshot", + body: { runId, grantRevision: 3 }, + }), env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ ownerId: "11111111-1111-4111-8111-111111111111", records: [ + { id: integrationId, kind: "bearer", name: "selected", origin: "https://api.example", description: null, revision: 4, encryptedPayload: "user-ciphertext", deletedAt: null }, + { id: "99999999-9999-4999-8999-999999999999", kind: "bearer", name: "deleted", origin: "https://deleted.example", description: null, revision: 8, encryptedPayload: null, deletedAt: "2026-01-01T00:00:00.000Z" }, + ], modelAccount: { id: modelAccountId, kind: "model", name: "selected-model", origin: null, description: "fixture account", revision: 6, encryptedPayload: "model-user-ciphertext" } }); + expect(env.query.mock.calls.map((call: any[]) => call[0])).toEqual(expect.arrayContaining([expect.stringContaining("grant_revision=$5"), expect.stringContaining("FROM app.integration_records")])); + }); + + it("returns durable revoked runs only for the authenticated current proxy boot", async () => { + const env = envFor(); + env.query.mockImplementation(async (sql: string) => sql.includes("proxy_token_digest") + ? { rows: [{ tenant_id: "11111111-1111-4111-8111-111111111111", proxy_identity: "proxy-fixture", boot_epoch: bootEpoch }], rowCount: 1 } + : { rows: [{ run_id: runId }], rowCount: 1 }); + const response = await proxyControlFetch(request({ + authorization: `Bearer ${token}`, + path: "/internal/credential-proxy/revocations", + body: { bootEpoch }, + }), env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ revokedRunIds: [runId], lockProxy: true }); + expect(env.query.mock.calls[1][0]).toContain("revoked_at IS NOT NULL"); + expect(env.query.mock.calls[1][1]).toEqual(["11111111-1111-4111-8111-111111111111", "proxy-fixture", bootEpoch]); + const stale = await proxyControlFetch(request({ + authorization: `Bearer ${token}`, + path: "/internal/credential-proxy/revocations", + body: { bootEpoch: "44444444-4444-4444-8444-444444444444" }, + }), env); + expect(stale.status).toBe(400); + }); +}); diff --git a/packages/api/test/proxy-launch-policy.test.ts b/packages/api/test/proxy-launch-policy.test.ts new file mode 100644 index 00000000..48952ca5 --- /dev/null +++ b/packages/api/test/proxy-launch-policy.test.ts @@ -0,0 +1,42 @@ +import { describe, expect, it } from "vitest"; +import { proxyLaunchDecision, ProxyLaunchBlockedError, requireProxyLaunch, type ProxyLaunchJob, type ProxyLaunchState } from "../src/proxy-launch-policy"; + +const stopped: ProxyLaunchState = { enabled: true, running: false, unlocked: false, pendingChanges: false }; +const locked: ProxyLaunchState = { enabled: true, running: true, unlocked: false, pendingChanges: false }; +const unlocked: ProxyLaunchState = { enabled: true, running: true, unlocked: true, pendingChanges: false }; +const flyCode: ProxyLaunchJob = { executionProvider: "fly", executionType: "code" }; + +describe("GEN-2824 proxy launch policy", () => { + it("starts an enabled proxy locked for Fly code jobs with no user credential grants", () => { + expect(proxyLaunchDecision(flyCode, stopped)).toEqual({ kind: "start_locked" }); + }); + + it("allows code-only Factorize access while the proxy is running and locked", () => { + expect(proxyLaunchDecision(flyCode, locked)).toEqual({ kind: "allow" }); + }); + + it.each([ + { label: "portable tools", job: { ...flyCode, portableIntegrationIds: ["integration"] } }, + { label: "portable model account", job: { ...flyCode, executionType: "agent" as const, modelAccess: "portable" as const, modelAccountId: "model" } }, + ])("requires an unlocked proxy with applied updates for $label", ({ job }) => { + expect(proxyLaunchDecision(job, stopped)).toEqual({ kind: "blocked", reason: "proxy_unavailable", retry: "manual" }); + expect(proxyLaunchDecision(job, locked)).toEqual({ kind: "blocked", reason: "proxy_locked", retry: "manual" }); + expect(proxyLaunchDecision(job, { ...unlocked, pendingChanges: true })).toEqual({ kind: "blocked", reason: "pending_changes", retry: "manual" }); + expect(proxyLaunchDecision(job, unlocked)).toEqual({ kind: "allow" }); + }); + + it("never auto-starts a disabled proxy and preserves native exe.dev behavior", () => { + expect(proxyLaunchDecision(flyCode, { ...stopped, enabled: false })).toEqual({ kind: "blocked", reason: "proxy_disabled", retry: "manual" }); + expect(proxyLaunchDecision({ executionProvider: "exe", executionType: "agent" }, { ...stopped, enabled: false })).toEqual({ kind: "allow" }); + }); + + it("does not auto-start a stopped proxy for credential-dependent jobs", () => { + expect(proxyLaunchDecision({ ...flyCode, portableIntegrationIds: ["integration"] }, stopped)).toMatchObject({ kind: "blocked", retry: "manual" }); + }); + + it("turns a blocked decision into a scheduler-recognized launch outcome", () => { + expect(() => requireProxyLaunch(proxyLaunchDecision({ ...flyCode, portableIntegrationIds: ["integration"] }, locked))) + .toThrow(new ProxyLaunchBlockedError("proxy_locked")); + expect(() => requireProxyLaunch(proxyLaunchDecision(flyCode, locked))).not.toThrow(); + }); +}); diff --git a/packages/api/test/proxy-management-authorization.test.ts b/packages/api/test/proxy-management-authorization.test.ts new file mode 100644 index 00000000..1fedfe43 --- /dev/null +++ b/packages/api/test/proxy-management-authorization.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, it, vi } from "vitest"; +import { protectedApiFetch } from "../src/protected-api"; + +vi.mock("cloudflare:workers", () => ({ WorkerEntrypoint: class {} })); +vi.mock("@cloudflare/workers-oauth-provider", () => ({ OAuthProvider: class {}, AuthorizationError: class extends Error {}, getOAuthApi: vi.fn() })); + +const ownerOnlyOperations = [ + ["GET", "/api/v1/integrations/bearer"], + ["POST", "/api/v1/integrations/bearer"], + ["GET", "/api/v1/model-accounts"], + ["POST", "/api/v1/model-accounts"], + ["POST", "/api/v1/model-accounts/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/browser-capability"], + ["GET", "/api/v1/unlock-key/verification"], + ["POST", "/api/v1/unlock-key/verification"], + ["GET", "/api/v1/unlock-key/rotation-snapshot"], + ["POST", "/api/v1/unlock-key/rotation"], +] as const; + +describe("credential and model-account management authorization", () => { + it.each(["access_token", "oauth"] as const)("rejects %s credentials even when granted integration write", async authMethod => { + const auth = { + tenantId: "tenant-fixture", + userId: "owner-fixture", + sessionVersion: 1, + authMethod, + scopes: ["integrations:read", "integrations:write", "jobs:write"], + } as any; + for (const [method, path] of ownerOnlyOperations) { + const response = await protectedApiFetch(new Request(`https://app.example${path}`, { + method, + headers: { + Origin: "https://app.example", + "Content-Type": "application/json", + ...(method === "POST" ? {} : {}), + }, + ...(method === "POST" ? { body: "{}" } : {}), + }), { APP_ORIGIN: "https://app.example" } as any, auth, {} as any); + expect(response.status, `${method} ${path}`).toBe(403); + expect(await response.json(), `${method} ${path}`).toMatchObject({ error: { code: "session_required" } }); + } + }); + + it("rejects a cross-origin interactive mutation before service access", async () => { + const response = await protectedApiFetch(new Request("https://app.example/api/v1/integrations/bearer", { + method: "POST", + headers: { Origin: "https://attacker.example", "Content-Type": "application/json" }, + body: "{}", + }), { APP_ORIGIN: "https://app.example" } as any, { + tenantId: "tenant-fixture", userId: "owner-fixture", sessionVersion: 1, + authMethod: "session", scopes: ["integrations:write"], + } as any, {} as any); + expect(response.status).toBe(403); + expect(await response.json()).toMatchObject({ error: { code: "invalid_origin" } }); + }); +}); diff --git a/packages/api/test/proxy-provisioner.test.ts b/packages/api/test/proxy-provisioner.test.ts new file mode 100644 index 00000000..bc6e9eca --- /dev/null +++ b/packages/api/test/proxy-provisioner.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it, vi } from "vitest"; +import { provisionCredentialProxy } from "../src/proxy-provisioner"; + +const input = { + tenantId: "11111111-1111-4111-8111-111111111111", + proxyIdentity: "fly-app/proxy-machine", + bootEpoch: "22222222-2222-4222-8222-222222222222", + controlOrigin: "https://factorize.example/", + browserOrigin: "https://proxy.example", +}; + +describe("credential proxy bootstrap provisioning", () => { + it("issues an epoch-bound credential only to the provisioner's private start channel", async () => { + const control = { + provisionCredential: vi.fn().mockResolvedValue("fzc_one-use-secret"), + revokeCredential: vi.fn().mockResolvedValue(true), + }; + const startProxy = vi.fn().mockResolvedValue(undefined); + + await expect(provisionCredentialProxy(input, { control, startProxy })).resolves.toEqual({ + proxyIdentity: input.proxyIdentity, + bootEpoch: input.bootEpoch, + }); + + expect(control.provisionCredential).toHaveBeenCalledWith(input.tenantId, input.proxyIdentity, input.bootEpoch); + expect(startProxy).toHaveBeenCalledWith({ ...input, controlOrigin: "https://factorize.example", credential: "fzc_one-use-secret" }); + expect(control.revokeCredential).not.toHaveBeenCalled(); + }); + + it("revokes the one-use credential if proxy startup fails", async () => { + const control = { + provisionCredential: vi.fn().mockResolvedValue("fzc_one-use-secret"), + revokeCredential: vi.fn().mockResolvedValue(true), + }; + const failure = new Error("startup failed"); + await expect(provisionCredentialProxy(input, { control, startProxy: vi.fn().mockRejectedValue(failure) })).rejects.toBe(failure); + expect(control.revokeCredential).toHaveBeenCalledWith(input.tenantId, input.proxyIdentity); + }); + + it("rejects non-HTTPS or non-origin control URLs before issuing a credential", async () => { + const control = { provisionCredential: vi.fn(), revokeCredential: vi.fn() }; + await expect(provisionCredentialProxy({ ...input, controlOrigin: "http://factorize.example" }, { control, startProxy: vi.fn() })).rejects.toThrow("HTTPS origin"); + await expect(provisionCredentialProxy({ ...input, controlOrigin: "https://factorize.example/path" }, { control, startProxy: vi.fn() })).rejects.toThrow("HTTPS origin"); + await expect(provisionCredentialProxy({ ...input, browserOrigin: "http://proxy.example" }, { control, startProxy: vi.fn() })).rejects.toThrow("HTTPS origin"); + expect(control.provisionCredential).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/api/test/run-enrollment.postgres.test.ts b/packages/api/test/run-enrollment.postgres.test.ts index a23277e2..add65893 100644 --- a/packages/api/test/run-enrollment.postgres.test.ts +++ b/packages/api/test/run-enrollment.postgres.test.ts @@ -28,7 +28,7 @@ describe.skipIf(!url)("single-use run enrollments (PostgreSQL)", () => { const repository = new RunEnrollmentRepository(db); const service = new RunEnrollmentService(repository); await repository.setProxyState({ tenantId, proxyIdentity, bootEpoch }); - expect(await repository.authorizeRun(binding, 7)).toBe(true); + expect(await repository.authorizeRun(binding)).toBe(1); return { binding, repository, service, tenantId, runId, proxyIdentity, bootEpoch }; } async function enrollment(f: Awaited>, lifetime = 60_000) { @@ -52,7 +52,7 @@ describe.skipIf(!url)("single-use run enrollments (PostgreSQL)", () => { const persisted = (await db.pool.query("SELECT token_digest,consumed_at,grant_revision FROM app.credential_run_enrollments WHERE tenant_id=$1 AND id=$2", [f.tenantId, issued.enrollmentId])).rows[0]; expect(persisted.token_digest.toString("base64url")).not.toBe(issued.token); expect(persisted.consumed_at).toEqual(issued.now); - expect(persisted.grant_revision).toBe("7"); + expect(persisted.grant_revision).toBe("1"); }, 20_000); it("rejects expiry, disconnect replay, a changed proxy epoch, ended runs, cross-tenant use, and revoked or disabled grants", async () => { @@ -63,16 +63,22 @@ describe.skipIf(!url)("single-use run enrollments (PostgreSQL)", () => { const disconnected = await fixture(), lost = await enrollment(disconnected); const reconnected = await enrollment(disconnected); await expect(disconnected.service.authenticate(lost.token, disconnected.binding, lost.now)).rejects.toThrow(); - await expect(disconnected.service.authenticate(reconnected.token, disconnected.binding, reconnected.now)).resolves.toMatchObject({ grantRevision: 7 }); + await expect(disconnected.service.authenticate(reconnected.token, disconnected.binding, reconnected.now)).resolves.toMatchObject({ grantRevision: 1 }); await expect(disconnected.service.authenticate(reconnected.token, disconnected.binding, reconnected.now)).rejects.toThrow(); const revised = await fixture(), priorRevision = await enrollment(revised); - expect(await revised.repository.authorizeRun(revised.binding, 8)).toBe(true); + expect(await revised.repository.authorizeRun({ ...revised.binding, vmIdentity: `${revised.binding.vmIdentity}-replacement` })).toBe(2); await expect(revised.service.authenticate(priorRevision.token, revised.binding, priorRevision.now)).rejects.toThrow(); const restarted = await fixture(), oldEpoch = await enrollment(restarted); - await restarted.repository.setProxyState({ tenantId: restarted.tenantId, proxyIdentity: restarted.proxyIdentity, bootEpoch: crypto.randomUUID() }); + const newEpoch = crypto.randomUUID(); + await restarted.repository.setProxyState({ tenantId: restarted.tenantId, proxyIdentity: restarted.proxyIdentity, bootEpoch: newEpoch }); await expect(restarted.service.authenticate(oldEpoch.token, restarted.binding, oldEpoch.now)).rejects.toThrow(); + const restartedBinding = { ...restarted.binding, bootEpoch: newEpoch }; + expect(await restarted.repository.authorizeRun(restartedBinding)).toBe(2); + expect(await restarted.repository.authorizeRun(restartedBinding)).toBe(2); + const fresh = await restarted.service.enroll(restartedBinding, new Date(now.getTime() + 60_000), now); + await expect(restarted.service.authenticate(fresh.token, restartedBinding, now)).resolves.toMatchObject({ bootEpoch: newEpoch, grantRevision: 2 }); const ended = await fixture(), endedEnrollment = await enrollment(ended); await db.pool.query("UPDATE app.job_runs SET state='succeeded' WHERE tenant_id=$1 AND id=$2", [ended.tenantId, ended.runId]); @@ -84,6 +90,7 @@ describe.skipIf(!url)("single-use run enrollments (PostgreSQL)", () => { const revoked = await fixture(), revokedEnrollment = await enrollment(revoked); await revoked.repository.revokeRun(revoked.tenantId, revoked.runId); await expect(revoked.service.authenticate(revokedEnrollment.token, revoked.binding, revokedEnrollment.now)).rejects.toThrow(); + expect(await revoked.repository.authorizeRun(revoked.binding)).toBeNull(); const disabled = await fixture(), disabledEnrollment = await enrollment(disabled); await disabled.repository.setProxyState({ tenantId: disabled.tenantId, proxyIdentity: disabled.proxyIdentity, bootEpoch: disabled.bootEpoch, disabled: true }); diff --git a/packages/api/test/run-launch-block.test.ts b/packages/api/test/run-launch-block.test.ts new file mode 100644 index 00000000..5dc0670e --- /dev/null +++ b/packages/api/test/run-launch-block.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it } from "vitest"; +import { RunRepository } from "../src/postgres/run-repository"; + +describe("durable proxy-gated launch retry", () => { + it("persists a blocked pre-launch run without polling it as a live execution", async () => { + const statements: string[] = []; + const client = { query: async (sql: string) => { statements.push(sql); return { rowCount: 1, rows: [{ id: "run" }] }; } }; + const database = { transaction: async (fn: (client: any) => Promise) => fn(client), pool: { query: client.query } }; + await new RunRepository(database as any).blockLaunch({ tenantId: "tenant", id: "run" } as any, "proxy_locked"); + expect(statements.some(sql => sql.includes("state='blocked'") && sql.includes("launch_lease_expires_at=NULL"))).toBe(true); + expect(statements.some(sql => sql.includes("execution_handle IS NULL"))).toBe(true); + expect(statements.some(sql => sql.includes("'launch_blocked'"))).toBe(true); + }); + + it("only manually requeues a proxy-gated run that has no execution handle", async () => { + const statements: string[] = []; + const client = { query: async (sql: string) => { statements.push(sql); return { rowCount: 1, rows: [{ id: "run" }] }; } }; + const database = { transaction: async (fn: (client: any) => Promise) => fn(client), pool: { query: client.query } }; + expect(await new RunRepository(database as any).retryBlockedLaunch("tenant", "run")).toBe(true); + expect(statements.some(sql => sql.includes("state='blocked'") && sql.includes("failure_code='proxy_launch_blocked'") && sql.includes("execution_handle IS NULL"))).toBe(true); + expect(statements.some(sql => sql.includes("state='reserved'"))).toBe(true); + expect(statements.some(sql => sql.includes("'launch_retry_requested'"))).toBe(true); + expect(statements.some(sql => sql.includes("wake_hints"))).toBe(true); + }); +}); diff --git a/packages/api/test/run-proxy-launch.postgres.test.ts b/packages/api/test/run-proxy-launch.postgres.test.ts new file mode 100644 index 00000000..3bfbf500 --- /dev/null +++ b/packages/api/test/run-proxy-launch.postgres.test.ts @@ -0,0 +1,79 @@ +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { loadingFixture } from "./helpers/loading-fixture"; +import { RunScheduler } from "../src/postgres/run-scheduler"; +import { protectedApiFetch } from "../src/protected-api"; + +vi.mock("cloudflare:workers", () => ({ WorkerEntrypoint: class {}, DurableObject: class {} })); +vi.mock("@cloudflare/workers-oauth-provider", () => ({ OAuthProvider: class {}, AuthorizationError: class extends Error {}, getOAuthApi: vi.fn() })); + +const url = process.env.AUTH_TEST_DATABASE_URL; +describe.skipIf(!url)("proxy-gated scheduler lifecycle (PostgreSQL/API)", () => { + let fixture: Awaited>; + beforeAll(async () => { fixture = await loadingFixture(url!); }); + afterAll(async () => { await fixture?.cleanup(); }); + + it("keeps launches blocked for missing, disabled, revoked, stale, or locked proxy state until the owner retries", async () => { + const { db, env, auth, tenantId, runId } = fixture; + const proxyIdentity = `proxy-${crypto.randomUUID()}`; + await db.pool.query(`UPDATE app.runs SET execution_config=$3 + WHERE tenant_id=$1 AND id=$2`, [tenantId, runId, { + agentKind: "codex", connectionId: "local", executionProvider: "fly", executionType: "agent", + modelAccess: "portable", modelAccountId: crypto.randomUUID(), portableIntegrationIds: [], runScopes: [], + }]); + + const cases = [ + { label: "no registered proxy", state: "missing", reason: "proxy_disabled" }, + { label: "disabled proxy", state: "disabled", reason: "proxy_disabled" }, + { label: "revoked proxy", state: "revoked", reason: "proxy_disabled" }, + { label: "stale unlock heartbeat", state: "stale", reason: "proxy_locked" }, + { label: "locked proxy", state: "locked", reason: "proxy_locked" }, + ] as const; + for (const scenario of cases) { + await db.pool.query("DELETE FROM app.run_activity WHERE tenant_id=$1 AND run_id=$2", [tenantId, runId]); + await db.pool.query(`UPDATE app.job_runs SET state='reserved',launch_lease_expires_at=NULL,next_poll_at=NULL WHERE tenant_id=$1 AND id=$2`, [tenantId, runId]); + await db.pool.query(`UPDATE app.runs SET state='reserved',failure_source=NULL,failure_code=NULL,failure_reason=NULL,execution_handle=NULL WHERE tenant_id=$1 AND id=$2`, [tenantId, runId]); + if (scenario.state === "missing") { + await db.pool.query("DELETE FROM app.credential_proxy_states WHERE tenant_id=$1", [tenantId]); + } else { + const disabled = scenario.state === "disabled" ? "now()" : "NULL"; + const revoked = scenario.state === "revoked" ? "now()" : "NULL"; + const unlocked = scenario.state === "stale" ? "true" : "false"; + const heartbeat = scenario.state === "stale" ? "now()-interval '1 minute'" : "now()"; + await db.pool.query(`INSERT INTO app.credential_proxy_states(tenant_id,proxy_identity,boot_epoch,disabled_at,revoked_at,unlocked,updated_at) + VALUES ($1,$2,$3,${disabled},${revoked},${unlocked},${heartbeat}) + ON CONFLICT (tenant_id) DO UPDATE SET proxy_identity=excluded.proxy_identity,boot_epoch=excluded.boot_epoch, + disabled_at=excluded.disabled_at,revoked_at=excluded.revoked_at,unlocked=excluded.unlocked,updated_at=excluded.updated_at`, + [tenantId, proxyIdentity, crypto.randomUUID()]); + } + + await new RunScheduler(db, env).process(); + const blocked = (await db.pool.query(`SELECT jr.state job_state,r.state run_state,r.failure_code,r.failure_reason, + r.execution_handle,jr.launch_lease_expires_at + FROM app.job_runs jr JOIN app.runs r USING (tenant_id,id) WHERE jr.tenant_id=$1 AND jr.id=$2`, [tenantId, runId])).rows[0]; + expect(blocked, scenario.label).toMatchObject({ job_state: "blocked", run_state: "blocked", failure_code: "proxy_launch_blocked", execution_handle: null, launch_lease_expires_at: null }); + expect(blocked.failure_reason, scenario.label).toContain(scenario.reason); + expect((await db.pool.query("SELECT action FROM app.run_activity WHERE tenant_id=$1 AND run_id=$2 ORDER BY created_at", [tenantId, runId])).rows, scenario.label).toEqual([{ action: "launch_blocked" }]); + } + + await db.pool.query("UPDATE app.credential_proxy_states SET unlocked=true,updated_at=now(),disabled_at=NULL,revoked_at=NULL WHERE tenant_id=$1", [tenantId]); + const request = () => protectedApiFetch(new Request(`${env.APP_ORIGIN}/api/v1/runs/${runId}/retry-launch`, { + method: "POST", headers: { Origin: env.APP_ORIGIN }, + }), env, { ...auth, authMethod: "session", scopes: ["runs:cancel"] }, { waitUntil() {} } as any); + const retried = await request(); + expect(retried.status).toBe(200); + expect(await retried.json()).toEqual({ runId, retryRequested: true }); + expect((await db.pool.query(`SELECT jr.state job_state,r.state run_state,r.failure_code,r.execution_handle + FROM app.job_runs jr JOIN app.runs r USING (tenant_id,id) WHERE jr.tenant_id=$1 AND jr.id=$2`, [tenantId, runId])).rows[0]).toMatchObject({ job_state: "reserved", run_state: "reserved", failure_code: null, execution_handle: null }); + expect((await db.pool.query("SELECT action FROM app.run_activity WHERE tenant_id=$1 AND run_id=$2 ORDER BY created_at", [tenantId, runId])).rows.map((row: any) => row.action)).toEqual(["launch_blocked", "launch_retry_requested"]); + expect((await db.pool.query("SELECT key FROM app.wake_hints WHERE key='global'")).rowCount).toBe(1); + expect((await request()).status).toBe(409); + + // The explicit retry only requeues the run. If proxy state changes before + // the scheduler claims it, the launch gate must block it again. + await db.pool.query("UPDATE app.credential_proxy_states SET unlocked=false,updated_at=now() WHERE tenant_id=$1", [tenantId]); + await new RunScheduler(db, env).process(); + expect((await db.pool.query(`SELECT jr.state job_state,r.state run_state,r.failure_code,r.execution_handle + FROM app.job_runs jr JOIN app.runs r USING (tenant_id,id) WHERE jr.tenant_id=$1 AND jr.id=$2`, [tenantId, runId])).rows[0]).toMatchObject({ job_state: "blocked", run_state: "blocked", failure_code: "proxy_launch_blocked", execution_handle: null }); + expect((await db.pool.query("SELECT action FROM app.run_activity WHERE tenant_id=$1 AND run_id=$2 ORDER BY created_at", [tenantId, runId])).rows.map((row: any) => row.action)).toEqual(["launch_blocked", "launch_retry_requested", "launch_blocked"]); + }, 30_000); +}); diff --git a/packages/api/test/run-proxy-launch.test.ts b/packages/api/test/run-proxy-launch.test.ts index 7c23e64d..299f3e6e 100644 --- a/packages/api/test/run-proxy-launch.test.ts +++ b/packages/api/test/run-proxy-launch.test.ts @@ -4,10 +4,21 @@ import { RunRepository } from "../src/postgres/run-repository"; import { ExeVmBackend } from "../src/exe-vm-backend"; import { encrypt } from "../src/crypto"; import { runProxyName } from "../src/run-credentials"; +import { ProxyLaunchBlockedError } from "../src/proxy-launch-policy"; afterEach(() => vi.restoreAllMocks()); describe("control-plane run credential delivery", () => { + it("persists a pre-launch block when a credential-dependent job has no fresh unlocked proxy status", async () => { + const database = { pool: { query: vi.fn() } }; + const scheduler: any = new RunScheduler(database as any, {} as any); + vi.spyOn(scheduler, "proxyLaunchState").mockResolvedValue({ enabled: true, running: true, unlocked: false, pendingChanges: false }); + await expect(scheduler.launch({ + id: "00000000-0000-4000-8000-000000000001", tenantId: "tenant", + executionConfig: { executionProvider: "fly", executionType: "agent", modelAccess: "portable", modelAccountId: "account" }, + })).rejects.toEqual(new ProxyLaunchBlockedError("proxy_locked")); + }); + it.each(["codex", "claude", "pi", "code"])("keeps %s proxy credentials out of the guest environment and prompt", async agentKind => { const runId = "00000000-0000-4000-8000-000000000001", key = btoa("a".repeat(32)); let issuedScopes: unknown; @@ -18,6 +29,7 @@ describe("control-plane run credential delivery", () => { } } }; const env = { DATABASE: database, APP_ORIGIN: "https://app.test", CREDENTIAL_ENCRYPTION_KEY: key } as any; const scheduler: any = new RunScheduler(database as any, env); + vi.spyOn(scheduler, "proxyLaunchState").mockResolvedValue({ enabled: true, running: true, unlocked: true, pendingChanges: false }); const backend = new ExeVmBackend({ apiToken: "control-plane-token", agentKind: (agentKind === "code" ? "codex" : agentKind) as any, tags: [] }); vi.spyOn(scheduler, "backend").mockResolvedValue(backend); const launch = vi.spyOn(backend, "launch").mockResolvedValue({ handle: { backendKind: "exe-vm", id: "factorize-run" }, observation: { state: "running" }, capabilities: [], destinationUrl: "https://vm.test" }); diff --git a/packages/app/src/features/settings/integrations.tsx b/packages/app/src/features/settings/integrations.tsx index dfa08fbc..f8c72b9a 100644 --- a/packages/app/src/features/settings/integrations.tsx +++ b/packages/app/src/features/settings/integrations.tsx @@ -10,6 +10,7 @@ import { IntegrationForm } from "./integration-form"; import { messageOf } from "../auth/session"; import { UnlockKeySetup } from "./unlock-key-setup"; import { BearerIntegrations } from "./bearer-integrations"; +import { ModelAccounts } from "./model-accounts"; const linkClass = "inline-flex min-h-10 items-center rounded-lg border border-stone-200 px-3 py-2 text-sm font-semibold hover:bg-stone-100 dark:border-slate-700 dark:hover:bg-slate-800"; @@ -53,7 +54,7 @@ export function Integrations() { {status.data?.cloudflareTail.installations.map(c =>

Cloudflare Tail · {c.name}

Referenced by {c.referencedJobCount} {c.referencedJobCount === 1 ? "job" : "jobs"}

Connected{buttons("cloudflareTail", c.integrationId)}
{manage(`tail:${c.integrationId}`, )}
)} } - +

Available integrations

Connect a provider to add it to your workspace. Connected providers remain available for additional connections.

setOpenSetup(open ? "linear-new" : null)}>{status.data?.linear ? "Add another Linear account" : "Connect Linear"} setOpenSetup(open ? "clickup-new" : null)}>{status.data?.clickup ? "Add another ClickUp workspace" : "Connect ClickUp"} diff --git a/packages/app/src/features/settings/model-accounts.tsx b/packages/app/src/features/settings/model-accounts.tsx new file mode 100644 index 00000000..3283878f --- /dev/null +++ b/packages/app/src/features/settings/model-accounts.tsx @@ -0,0 +1,146 @@ +import { useEffect, useState } from "react"; +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { api } from "factorize-api-client"; +import { Button, Card, Input, Label } from "../../shared/ui"; +import { encryptModelAccountCacheRecord, verifyUnlockKey } from "../../lib/unlock-crypto"; +import { getUnlockVerification } from "../../lib/unlock-key"; +import { messageOf } from "../auth/session"; + +type DeviceState = { state: string; verificationUrl?: string; userCode?: string }; +type ModelAccount = { id: string; name: string; description: string | null; revision: number }; +const accountsQuery = { queryKey: ["settings", "model-accounts"], queryFn: async ({ signal }: { signal: AbortSignal }) => { + const { data, error } = await api.GET("/api/v1/model-accounts", { signal }); + if (!data || error) throw new Error(messageOf(error)); + return data; +} }; +const verificationQuery = { queryKey: ["unlock-key-verification"], queryFn: ({ signal }: { signal: AbortSignal }) => getUnlockVerification(signal), retry: false as const }; + +async function proxyAction(accountId: string, action: "authorize" | "status" | "cancel"): Promise { + const capability = await api.POST("/api/v1/model-accounts/{accountId}/browser-capability", { + params: { path: { accountId } }, body: { action }, + }); + if (!capability.data || capability.error) throw new Error(messageOf(capability.error)); + const response = await fetch(capability.data.endpoint, { + method: "POST", mode: "cors", credentials: "omit", cache: "no-store", + headers: { Authorization: `Bearer ${capability.data.capability}`, "Content-Type": "application/json" }, + body: "{}", + }); + let result: DeviceState; + try { result = await response.json() as DeviceState; } + catch { throw new Error("The credential proxy returned an invalid response."); } + if (!response.ok) { + const messages: Record = { 423: "Unlock the credential proxy before connecting a model account.", 503: "The credential proxy is unavailable. Try again shortly." }; + throw new Error(messages[response.status] ?? "The credential proxy could not complete this request."); + } + return result; +} + +export function ModelAccounts() { + const cache = useQueryClient(), accounts = useQuery(accountsQuery), verification = useQuery(verificationQuery); + const [active, setActive] = useState(null); + const [states, setStates] = useState>({}); + const [error, setError] = useState(""); + const [name, setName] = useState(""), [description, setDescription] = useState(""), [key, setKey] = useState(""); + + const create = useMutation({ retry: false, mutationFn: async () => { + const cleanName = name.trim(), cleanDescription = description.trim(); + if (!cleanName || cleanName.length > 120) throw new Error("Enter a name up to 120 characters."); + if (cleanDescription.length > 500) throw new Error("Description must be at most 500 characters."); + const setup = verification.data?.verification; + if (!setup || !verification.data) throw new Error("Set up an unlock key before adding a model account."); + if (!accounts.data) throw new Error("Model accounts are still loading. Try again shortly."); + try { await verifyUnlockKey(key, verification.data.ownerId, setup.generationId, JSON.parse(setup.encryptedVerification)); } + catch { throw new Error("Unlock key could not be verified for this account and generation."); } + const id = crypto.randomUUID(), expectedRevision = accounts.data.revision; + const encryptedPayload = await encryptModelAccountCacheRecord(new TextEncoder().encode("{}"), key, { + ownerId: verification.data.ownerId, recordId: id, recordType: "model", destination: "factorize", + keyGeneration: setup.generationId, revision: expectedRevision + 1, + }); + const { data, error } = await api.POST("/api/v1/model-accounts", { body: { + id, expectedRevision, idempotencyKey: crypto.randomUUID(), name: cleanName, + description: cleanDescription || null, encryptedPayload, + } }); + if (!data || error) throw new Error(messageOf(error)); + return data; + }, onSuccess: async () => { + setName(""); setDescription(""); setKey(""); setError(""); + await cache.invalidateQueries({ queryKey: accountsQuery.queryKey }); + }, onError: async reason => { + setKey(""); setError(reason.message); + if (reason.message.includes("changed") || reason.message.includes("revision")) await cache.invalidateQueries({ queryKey: accountsQuery.queryKey }); + } }); + + const remove = useMutation({ retry: false, mutationFn: async (account: ModelAccount) => { + if (!accounts.data) throw new Error("Model accounts are still loading. Try again shortly."); + const { data, error } = await api.DELETE("/api/v1/model-accounts/{accountId}", { + params: { path: { accountId: account.id } }, + body: { expectedRevision: accounts.data.revision, idempotencyKey: crypto.randomUUID() }, + }); + if (!data || error) throw new Error(messageOf(error)); + return data; + }, onSuccess: async () => { + setError(""); await cache.invalidateQueries({ queryKey: accountsQuery.queryKey }); + }, onError: async reason => { + setError(reason.message); await cache.invalidateQueries({ queryKey: accountsQuery.queryKey }); + } }); + + useEffect(() => { + if (!active) return; + let stopped = false; + const poll = async () => { + try { + const next = await proxyAction(active, "status"); + if (stopped) return; + setStates(current => ({ ...current, [active]: next })); + if (next.state !== "pending") setActive(null); + } catch (reason) { + if (!stopped) setError(reason instanceof Error ? reason.message : "Could not check model account authorization."); + } + }; + const timer = window.setInterval(poll, 2000); + return () => { stopped = true; window.clearInterval(timer); }; + }, [active]); + + const run = async (account: ModelAccount, action: "authorize" | "cancel") => { + setError(""); + try { + const next = await proxyAction(account.id, action); + setStates(current => ({ ...current, [account.id]: next })); + setActive(action === "authorize" && next.state === "pending" ? account.id : null); + } catch (reason) { + setError(reason instanceof Error ? reason.message : "Could not connect the model account."); + } + }; + + return
+

ChatGPT model accounts

Connect each account through Codex device authorization. Credentials stay in the credential proxy.

+ {accounts.isPending &&

Loading model accounts…

} + {accounts.error &&

{accounts.error.message}

} + {error &&

{error}

} + {!accounts.isPending && accounts.data?.accounts.length === 0 &&

No model accounts are configured yet.

} + {accounts.data?.accounts.map(account => { + const state = states[account.id] ?? { state: "idle" }; + return +

{account.name}

{account.description &&

{account.description}

} + {state.state === "pending" &&

Waiting for device authorization.

{state.userCode &&

Code: {state.userCode}

}{state.verificationUrl && OpenAI device authorization}
} + {state.state === "authorized" &&

ChatGPT account connected.

} + {state.state === "authorized-unsaved" &&

ChatGPT account connected, but its encrypted token cache could not be saved. Keep the proxy running and retry after checking its connection.

} + {state.state === "reauthentication-required" &&

Sign in again to reconnect this account.

} + {state.state === "failed" &&

Device authorization failed. You can try again.

} + {state.state === "canceled" &&

Device authorization canceled.

} +
+ {state.state === "pending" ? : } + +
; + })} +

Add a model account

+ {!verification.data?.verification && verification.data &&

Set up your unlock key above before adding a model account.

} +
{ event.preventDefault(); setError(""); create.mutate(); }}> + + + + +
+
+
; +} diff --git a/packages/app/src/features/settings/unlock-key-setup.tsx b/packages/app/src/features/settings/unlock-key-setup.tsx index 60dbdec2..2d8ebeb8 100644 --- a/packages/app/src/features/settings/unlock-key-setup.tsx +++ b/packages/app/src/features/settings/unlock-key-setup.tsx @@ -1,7 +1,7 @@ import { useState } from "react"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { Button, Card, Input, Label } from "../../shared/ui"; -import { createUnlockSetup, generateUnlockKey, getUnlockVerification, verifyUnlockKey } from "../../lib/unlock-key"; +import { createUnlockSetup, generateUnlockKey, getUnlockVerification, replaceUnlockKeyGeneration, verifyUnlockKey } from "../../lib/unlock-key"; export function UnlockKeySetup() { const cache = useQueryClient(); @@ -9,6 +9,7 @@ export function UnlockKeySetup() { const [pendingKey, setPendingKey] = useState(""); const [acknowledged, setAcknowledged] = useState(false); const [enteredKey, setEnteredKey] = useState(""); + const [replacementMode, setReplacementMode] = useState<"rotate" | "reset" | null>(null); const [verified, setVerified] = useState(false); const setup = useMutation({ retry: false, mutationFn: async () => { if (!verification.data || verification.data.verification || !pendingKey || !acknowledged) throw new Error("Save and acknowledge your unlock key before setup."); @@ -19,6 +20,20 @@ export function UnlockKeySetup() { if (!current || !verification.data) throw new Error("Set up an unlock key before verifying it."); await verifyUnlockKey(enteredKey, verification.data.ownerId, current.generationId, JSON.parse(current.encryptedVerification)); }, onSuccess: () => setVerified(true), onError: () => setVerified(false) }); + const replace = useMutation({ retry: false, mutationFn: async () => { + if (!replacementMode || !pendingKey || !acknowledged || (replacementMode === "rotate" && !enteredKey)) throw new Error("Generate and save the new key before continuing."); + return replaceUnlockKeyGeneration(replacementMode === "rotate" ? enteredKey : null, pendingKey, replacementMode); + }, onSuccess: async () => { + setPendingKey(""); setEnteredKey(""); setAcknowledged(false); setVerified(false); setReplacementMode(null); + await Promise.all([ + cache.invalidateQueries({ queryKey: ["unlock-key-verification"] }), + cache.invalidateQueries({ queryKey: ["settings", "bearer-integrations"] }), + cache.invalidateQueries({ queryKey: ["settings", "model-accounts"] }), + ]); + }, onError: async () => { + setPendingKey(""); setEnteredKey(""); setAcknowledged(false); setVerified(false); setReplacementMode(null); + await cache.invalidateQueries({ queryKey: ["unlock-key-verification"] }); + } }); const resetPending = () => { setPendingKey(""); setAcknowledged(false); }; return

Unlock key

@@ -39,6 +54,18 @@ export function UnlockKeySetup() { {verified &&

Unlock key verified for this account and generation.

} {check.error &&

Unlock key could not be verified for this account and generation.

} + {!replacementMode &&
+ + +
} + {replacementMode &&
+

{replacementMode === "rotate" ? "Save this new key before replacing the current key. All active integrations and pending changes will be re-encrypted." : "Save this new key before reset. All integrations and model accounts will be deleted; Fly connection and execution settings will be preserved."}

+ + {replacementMode === "rotate" && } + + {replace.error &&

{replace.error.message}

} +
+
} } {setup.error &&

{setup.error.message}

}
; diff --git a/packages/app/src/lib/unlock-crypto.ts b/packages/app/src/lib/unlock-crypto.ts index 900aabcf..b2279f98 100644 --- a/packages/app/src/lib/unlock-crypto.ts +++ b/packages/app/src/lib/unlock-crypto.ts @@ -91,6 +91,38 @@ export async function encryptRecord(plaintext: Uint8Array, key: string, binding: return { version: VERSION, algorithm: "A256GCM", nonce: encodeBase64Url(nonce), ciphertext: encodeBase64Url(new Uint8Array(ciphertext)) }; } +/** Encrypt an initial model-account cache with the same key-generation-bound AAD used by the proxy. */ +export async function encryptModelAccountCacheRecord(plaintext: Uint8Array, key: string, binding: RecordBinding & { keyGeneration: string }): Promise { + if (!(plaintext instanceof Uint8Array) || !binding || typeof binding.ownerId !== "string" || !binding.ownerId + || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(binding.recordId) + || typeof binding.keyGeneration !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(binding.keyGeneration) + || !Number.isSafeInteger(binding.revision) || binding.revision < 1) { + throw new TypeError("Malformed model-account encryption binding"); + } + const aad = encoder.encode(JSON.stringify(["factorize-record-v1", binding.ownerId, binding.recordId, "model", "factorize", binding.keyGeneration, binding.revision])); + const nonce = crypto.getRandomValues(new Uint8Array(NONCE_BYTES)); + const aesKey = await deriveAesKey(key, ["encrypt"]); + const ciphertext = await crypto.subtle.encrypt({ name: "AES-GCM", iv: toArrayBuffer(nonce), additionalData: toArrayBuffer(aad), tagLength: 128 }, aesKey, toArrayBuffer(plaintext)); + const encode = (bytes: Uint8Array) => btoa(Array.from(bytes, byte => String.fromCharCode(byte)).join("")); + return `${encode(nonce)}.${encode(new Uint8Array(ciphertext))}`; +} + +/** Decrypt a stored model-account cache locally for re-encryption during key rotation. */ +export async function decryptModelAccountCacheRecord(encryptedPayload: string, key: string, binding: RecordBinding & { keyGeneration: string }): Promise { + const [encodedNonce, encodedCiphertext, extra] = encryptedPayload.split("."); + if (!encodedNonce || !encodedCiphertext || extra !== undefined) throw new TypeError("Malformed model-account ciphertext"); + const nonce = Uint8Array.from(atob(encodedNonce), character => character.charCodeAt(0)); + const ciphertext = Uint8Array.from(atob(encodedCiphertext), character => character.charCodeAt(0)); + if (nonce.byteLength !== NONCE_BYTES || ciphertext.byteLength < 16) throw new TypeError("Malformed model-account ciphertext"); + if (!binding || typeof binding.ownerId !== "string" || !binding.ownerId + || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(binding.recordId) + || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(binding.keyGeneration) + || !Number.isSafeInteger(binding.revision) || binding.revision < 1) throw new TypeError("Malformed model-account encryption binding"); + const aad = encoder.encode(JSON.stringify(["factorize-record-v1", binding.ownerId, binding.recordId, "model", "factorize", binding.keyGeneration, binding.revision])); + const aesKey = await deriveAesKey(key, ["decrypt"]); + return new Uint8Array(await crypto.subtle.decrypt({ name: "AES-GCM", iv: toArrayBuffer(nonce), additionalData: toArrayBuffer(aad), tagLength: 128 }, aesKey, toArrayBuffer(ciphertext))); +} + /** Decrypt a supported envelope; malformed metadata, bindings and authentication fail closed. */ export async function decryptRecord(envelope: unknown, key: string, binding: RecordBinding): Promise { if (!envelope || typeof envelope !== "object" || Array.isArray(envelope)) throw new TypeError("Malformed encrypted envelope"); diff --git a/packages/app/src/lib/unlock-key.ts b/packages/app/src/lib/unlock-key.ts index 3af6b34d..384cfdef 100644 --- a/packages/app/src/lib/unlock-key.ts +++ b/packages/app/src/lib/unlock-key.ts @@ -1,5 +1,5 @@ import { api } from "factorize-api-client"; -import { createUnlockVerification, generateUnlockKey, verifyUnlockKey } from "./unlock-crypto"; +import { createUnlockVerification, decryptModelAccountCacheRecord, decryptRecord, encryptModelAccountCacheRecord, encryptRecord, generateUnlockKey, verifyUnlockKey } from "./unlock-crypto"; import { messageOf } from "../features/auth/session"; export async function getUnlockVerification(signal?: AbortSignal) { @@ -15,4 +15,64 @@ export async function createUnlockSetup(ownerId: string, key: string, generation return data; } +const base64ToUrl = (value: string) => btoa(atob(value).split("").map(c => c).join("")).replaceAll("+", "-").replaceAll("/", "_").replace(/=+$/u, ""); +const urlToBase64 = (value: string) => btoa(Array.from(Uint8Array.from(atob(value.replaceAll("-", "+").replaceAll("_", "/") + "=".repeat((4 - value.length % 4) % 4)), c => c.charCodeAt(0)), c => String.fromCharCode(c)).join("")); +function decodeStoredCiphertext(payload: string) { + const [nonce, ciphertext, extra] = payload.split("."); + if (!nonce || !ciphertext || extra !== undefined) throw new TypeError("Malformed stored integration ciphertext"); + return { version: 1 as const, algorithm: "A256GCM" as const, nonce: base64ToUrl(nonce), ciphertext: base64ToUrl(ciphertext) }; +} +function encodeStoredCiphertext(envelope: { nonce: string; ciphertext: string }) { return `${urlToBase64(envelope.nonce)}.${urlToBase64(envelope.ciphertext)}`; } + +export async function replaceUnlockKeyGeneration(oldKey: string | null, newKey: string, mode: "rotate" | "reset") { + const { data: snapshot, error: snapshotError } = await api.GET("/api/v1/unlock-key/rotation-snapshot"); + if (!snapshot || snapshotError) throw new Error(messageOf(snapshotError)); + if (mode === "rotate") { + const { data: current, error } = await api.GET("/api/v1/unlock-key/verification"); + if (!current || error || !current.verification || !oldKey) throw new Error("Enter the current unlock key to rotate it."); + await verifyUnlockKey(oldKey, current.ownerId, current.verification.generationId, JSON.parse(current.verification.encryptedVerification)); + if (current.verification.generationId !== snapshot.generationId) throw new Error("Unlock key generation changed. Reload and try again."); + } + const generationId = crypto.randomUUID(); + const encryptedVerification = JSON.stringify(await createUnlockVerification(newKey, snapshot.ownerId, generationId)); + const records = mode === "reset" ? [] : await Promise.all(snapshot.records.map(async record => { + if (record.deletedAt || record.encryptedPayload === null) return record; + let payload: string; + if (record.kind === "bearer") { + if (!record.origin) throw new Error("A bearer integration is missing its destination and cannot be rotated safely."); + const plain = await decryptRecord(decodeStoredCiphertext(record.encryptedPayload), oldKey!, { ownerId: snapshot.ownerId, recordId: record.id, recordType: "bearer", destination: record.origin, revision: record.revision }); + try { + payload = encodeStoredCiphertext(await encryptRecord(plain, newKey, { ownerId: snapshot.ownerId, recordId: record.id, recordType: "bearer", destination: record.origin, revision: record.revision })); + } finally { plain.fill(0); } + } else { + const plain = await decryptModelAccountCacheRecord(record.encryptedPayload, oldKey!, { ownerId: snapshot.ownerId, recordId: record.id, recordType: "model", destination: "factorize", keyGeneration: snapshot.generationId, revision: record.revision }); + try { payload = await encryptModelAccountCacheRecord(plain, newKey, { ownerId: snapshot.ownerId, recordId: record.id, recordType: "model", destination: "factorize", keyGeneration: generationId, revision: record.revision }); } + finally { plain.fill(0); } + } + return { ...record, encryptedPayload: payload }; + })); + const changes = mode === "reset" ? [] : await Promise.all(snapshot.changes.map(async change => { + if (change.encryptedPayload === null) return change; + let payload: string; + if (change.kind === "bearer") { + if (!change.origin) throw new Error("A bearer edit is missing its destination and cannot be rotated safely."); + const plain = await decryptRecord(decodeStoredCiphertext(change.encryptedPayload), oldKey!, { ownerId: snapshot.ownerId, recordId: change.id, recordType: "bearer", destination: change.origin, revision: change.revision }); + try { payload = encodeStoredCiphertext(await encryptRecord(plain, newKey, { ownerId: snapshot.ownerId, recordId: change.id, recordType: "bearer", destination: change.origin, revision: change.revision })); } + finally { plain.fill(0); } + } else { + if (!change.recordRevision) throw new Error("A model-account edit is missing its revision and cannot be rotated safely."); + const plain = await decryptModelAccountCacheRecord(change.encryptedPayload, oldKey!, { ownerId: snapshot.ownerId, recordId: change.id, recordType: "model", destination: "factorize", keyGeneration: snapshot.generationId, revision: change.recordRevision }); + try { payload = await encryptModelAccountCacheRecord(plain, newKey, { ownerId: snapshot.ownerId, recordId: change.id, recordType: "model", destination: "factorize", keyGeneration: generationId, revision: change.recordRevision }); } + finally { plain.fill(0); } + } + return { ...change, encryptedPayload: payload }; + })); + const { data, error } = await api.POST("/api/v1/unlock-key/rotation", { body: { + expectedGenerationId: snapshot.generationId, generationId, expectedRevision: snapshot.revision, + encryptedVerification, records, changes, + } }); + if (!data || error) throw new Error(messageOf(error)); + return data; +} + export { generateUnlockKey, verifyUnlockKey }; diff --git a/packages/app/test/jobs.browser.test.mjs b/packages/app/test/jobs.browser.test.mjs index a502b23b..3ce1b618 100644 --- a/packages/app/test/jobs.browser.test.mjs +++ b/packages/app/test/jobs.browser.test.mjs @@ -224,6 +224,15 @@ test("project graph does not zoom when the desktop wheel scrolls over it", async const graph = page.getByRole("region", { name: /1 jobs and 0 connections/ }); await graph.waitFor(); const viewport = graph.locator(".react-flow__viewport"); + await page.waitForFunction(async () => { + const element = document.querySelector('[role="region"] .react-flow__viewport'); + if (!element) return false; + const first = element.getAttribute("style"); + await new Promise(requestAnimationFrame); + const second = element.getAttribute("style"); + await new Promise(requestAnimationFrame); + return first === second && second === element.getAttribute("style"); + }); const initialTransform = await viewport.getAttribute("style"); const bounds = await graph.boundingBox(); await page.mouse.move(bounds.x + bounds.width / 2, bounds.y + bounds.height / 2); diff --git a/packages/app/test/settings.browser.test.mjs b/packages/app/test/settings.browser.test.mjs index bc3368e5..5717730d 100644 --- a/packages/app/test/settings.browser.test.mjs +++ b/packages/app/test/settings.browser.test.mjs @@ -18,6 +18,149 @@ test("integration setup tests credentials, submits typed fields, clears secrets await form.getByLabel("exe.dev account token").fill("private-input"); await form.getByRole("button", { name: "Test credentials" }).click(); await form.getByText("✓ Credentials verified.", { exact: true }).waitFor(); await form.getByLabel("VM tags").selectOption(["dev"]); await form.getByRole("button", { name: "Save integration", exact: true }).click(); await form.getByText("Integration saved", { exact: true }).waitFor(); assert.deepEqual(body.tags, ["dev"]); assert.equal(body.agentKind, "codex"); assert.equal(await form.getByLabel("exe.dev account token").inputValue(), ""); assert.equal(await page.evaluate(() => JSON.stringify(localStorage).includes("private-input")), false); await page.getByText("dev", { exact: true }).waitFor(); await context.close(); }); +test("model account settings issue short-lived capabilities and display the bounded device challenge", async () => { + const { context, page } = await setup(); + const accountId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", proxyOrigin = "https://proxy.example"; + await page.route("**/api/v1/providers/github/installations", route => route.fulfill({ json: [] })); + await page.route("**/api/v1/integrations", route => route.fulfill({ json: { linear: null, clickup: null, exe: null, exeConnections: [], ampConnections: [], cloudflareTail: { count: 0, installations: [] } } })); + await page.route("**/api/v1/model-accounts", async route => { + if (route.request().method() === "GET") return route.fulfill({ json: { revision: 1, accounts: [{ id: accountId, name: "Work account", description: null, revision: 1 }] } }); + const action = route.request().postDataJSON().action; + return route.fulfill({ json: { endpoint: `${proxyOrigin}/model-accounts/${accountId}/${action}`, capability: `one-use-${action}` } }); + }); + await page.route("**/api/v1/model-accounts/*/browser-capability", async route => { + const action = route.request().postDataJSON().action; + return route.fulfill({ json: { endpoint: `${proxyOrigin}/model-accounts/${accountId}/${action}`, capability: `one-use-${action}` } }); + }); + const proxyCalls = []; + await page.route(`${proxyOrigin}/model-accounts/**`, async route => { + const request = route.request(); + if (request.method() === "OPTIONS") return route.fulfill({ status: 204, headers: { "Access-Control-Allow-Origin": origin, "Access-Control-Allow-Methods": "POST", "Access-Control-Allow-Headers": "Authorization, Content-Type" } }); + proxyCalls.push({ path: new URL(request.url()).pathname, authorization: request.headers().authorization, body: request.postData() }); + const response = request.url().endsWith("/authorize") ? { state: "pending", verificationUrl: "https://auth.openai.com/codex/device", userCode: "ABCD-1234" } : { state: "canceled" }; + return route.fulfill({ json: response, headers: { "Access-Control-Allow-Origin": origin, "Cache-Control": "no-store" } }); + }); + await page.goto(origin + "/settings/integrations"); + const models = page.getByRole("region", { name: "ChatGPT model accounts" }); + await models.getByRole("button", { name: "Connect ChatGPT" }).click(); + await models.getByText("ABCD-1234", { exact: false }).waitFor(); + await models.getByRole("link", { name: "OpenAI device authorization" }).waitFor(); + await models.getByRole("button", { name: "Cancel authorization" }).click(); + await models.getByText("Device authorization canceled.", { exact: true }).waitFor(); + assert.deepEqual(proxyCalls.map(call => call.authorization), ["Bearer one-use-authorize", "Bearer one-use-cancel"]); + assert.deepEqual(proxyCalls.map(call => call.body), ["{}", "{}"]); + await context.close(); +}); +test("owner browser creates a bearer integration with encrypted input and clears both secret fields", async () => { + const { context, page } = await setup(); + await page.route("**/api/v1/providers/github/installations", route => route.fulfill({ json: [] })); + await page.route("**/api/v1/integrations", route => route.fulfill({ json: { linear: null, clickup: null, exe: null, exeConnections: [], ampConnections: [], cloudflareTail: { count: 0, installations: [] } } })); + let verification = null, saved = [], createBody; + await page.route("**/api/v1/unlock-key/verification", async route => { + if (route.request().method() === "GET") return route.fulfill({ json: { ownerId: "owner", verification } }); + const body = route.request().postDataJSON(); + assert.deepEqual(Object.keys(body).sort(), ["encryptedVerification", "generationId"]); + verification = { generationId: body.generationId, encryptedVerification: body.encryptedVerification }; + return route.fulfill({ json: { generationId: body.generationId } }); + }); + await page.route("**/api/v1/integrations/bearer", async route => { + if (route.request().method() === "GET") return route.fulfill({ json: { revision: saved.length, integrations: saved } }); + createBody = route.request().postDataJSON(); + saved = [{ id: createBody.id, name: createBody.name, origin: createBody.origin, description: createBody.description, revision: 1, encryptedPayload: createBody.encryptedPayload }]; + return route.fulfill({ status: 201, json: { revision: 1, duplicate: false } }); + }); + + await page.goto(origin + "/settings/integrations"); + await page.getByRole("button", { name: "Generate unlock key" }).click(); + const unlockKey = await page.getByLabel("New unlock key").inputValue(); + await page.getByRole("checkbox", { name: "I saved this key somewhere safe." }).check(); + await page.getByRole("button", { name: "I saved my key" }).click(); + await page.getByText("Unlock key verification is set up.", { exact: false }).waitFor(); + + const integrations = page.getByRole("region", { name: "Bearer integrations" }); + await integrations.getByLabel("Name").fill("Example service"); + await integrations.getByLabel("HTTPS origin").fill("https://api.example.test"); + await integrations.getByLabel("Description or help (optional)").fill("Fixture API"); + await integrations.getByLabel("Bearer token").fill("synthetic-upstream-secret"); + await integrations.getByLabel("Unlock key").fill(unlockKey); + await integrations.getByRole("button", { name: "Add integration" }).click(); + await integrations.getByText("Saved encrypted integration.", { exact: false }).waitFor(); + + assert.equal(createBody.name, "Example service"); + assert.equal(createBody.origin, "https://api.example.test"); + assert.equal(createBody.description, "Fixture API"); + assert.equal(JSON.stringify(createBody).includes("synthetic-upstream-secret"), false); + assert.equal(JSON.stringify(createBody).includes(unlockKey), false); + assert.match(createBody.encryptedPayload, /^[A-Za-z0-9+/]+=*\.[A-Za-z0-9+/]+=*$/u); + assert.equal(await integrations.getByLabel("Bearer token").inputValue(), ""); + assert.equal(await integrations.getByLabel("Unlock key").inputValue(), ""); + assert.equal(await page.evaluate(() => JSON.stringify(localStorage).includes("synthetic-upstream-secret")), false); + await context.close(); +}); +test("model account create/delete encrypts initial state and uses the current collection revision", async () => { + const { context, page } = await setup(); + await page.route("**/api/v1/providers/github/installations", route => route.fulfill({ json: [] })); + await page.route("**/api/v1/integrations", route => route.fulfill({ json: { linear: null, clickup: null, exe: null, exeConnections: [], ampConnections: [], cloudflareTail: { count: 0, installations: [] } } })); + let verification = null, revision = 0, account = null, createBody = null, deleteBodies = [], staleOnce = true; + await page.route("**/api/v1/unlock-key/verification", async route => { + if (route.request().method() === "GET") return route.fulfill({ json: { ownerId: "owner", verification } }); + const body = route.request().postDataJSON(); verification = { generationId: body.generationId, encryptedVerification: body.encryptedVerification }; + return route.fulfill({ json: { generationId: body.generationId } }); + }); + await page.route("**/api/v1/model-accounts", async route => { + if (route.request().method() === "GET") return route.fulfill({ json: { revision, accounts: account ? [account] : [] } }); + createBody = route.request().postDataJSON(); + assert.deepEqual(Object.keys(createBody).sort(), ["description", "encryptedPayload", "expectedRevision", "id", "idempotencyKey", "name"]); + assert.equal(createBody.expectedRevision, revision); + assert.equal(createBody.name, "Work account"); + assert.equal(createBody.encryptedPayload.includes("{}"), false); + assert.equal(createBody.encryptedPayload.includes("fzk1."), false); + assert.match(createBody.encryptedPayload, /^[A-Za-z0-9+/]+=*\.[A-Za-z0-9+/]+=*$/u); + revision++; account = { id: createBody.id, name: createBody.name, description: createBody.description, revision }; + return route.fulfill({ status: 201, json: { revision, duplicate: false } }); + }); + await page.route("**/api/v1/model-accounts/*", async route => { + assert.equal(route.request().method(), "DELETE"); + const body = route.request().postDataJSON(); deleteBodies.push(body); + assert.deepEqual(Object.keys(body).sort(), ["expectedRevision", "idempotencyKey"]); + if (staleOnce) { + staleOnce = false; revision++; + return route.fulfill({ status: 409, json: { error: { code: "stale_revision", message: "The model account collection changed." } } }); + } + assert.equal(body.expectedRevision, revision); + revision++; account = null; + return route.fulfill({ json: { id: createBody.id, deleted: true, revision, duplicate: false } }); + }); + + await page.goto(origin + "/settings/integrations"); + await page.getByRole("button", { name: "Generate unlock key" }).click(); + const unlockKey = await page.getByLabel("New unlock key").inputValue(); + await page.getByRole("checkbox", { name: "I saved this key somewhere safe." }).check(); + await page.getByRole("button", { name: "I saved my key" }).click(); + await page.getByText("Unlock key verification is set up.", { exact: false }).waitFor(); + + const models = page.getByRole("region", { name: "ChatGPT model accounts" }); + await models.getByLabel("Name").fill("Work account"); + await models.getByLabel("Description (optional)").fill("Personal account"); + await models.getByLabel("Unlock key").fill(unlockKey); + await models.getByRole("button", { name: "Add model account" }).click(); + await models.getByText("Work account", { exact: true }).waitFor(); + assert.equal(createBody.expectedRevision, 0); + assert.equal(createBody.description, "Personal account"); + + page.once("dialog", dialog => dialog.accept()); + const refreshAfterConflict = page.waitForResponse(response => response.url().includes("/api/v1/model-accounts") && response.request().method() === "GET" && response.status() === 200); + await models.getByRole("button", { name: "Delete account" }).click(); + await models.getByRole("alert").filter({ hasText: "collection changed" }).waitFor(); + await refreshAfterConflict; + page.once("dialog", dialog => dialog.accept()); + await models.getByRole("button", { name: "Delete account" }).click(); + await models.getByText("No model accounts are configured yet.", { exact: true }).waitFor(); + assert.equal(deleteBodies.length, 2); + assert.equal(deleteBodies[0].expectedRevision, 1); + assert.equal(deleteBodies[1].expectedRevision, 2); + await context.close(); +}); test("unlock-key setup requires acknowledgement, stores ciphertext only, verifies entered keys and survives refresh without a saved key", async () => { const { context, page } = await setup(); await page.route("**/api/v1/providers/github/installations", route => route.fulfill({ json: [] })); @@ -64,6 +207,58 @@ test("unlock-key setup requires acknowledgement, stores ciphertext only, verifie assert.equal(await page.evaluate(() => localStorage.length), 0); await context.close(); }); + +test("owner can rotate the unlock key or reset lost-key integrations without sending keys to the API", async () => { + const { context, page } = await setup(); + await page.route("**/api/v1/providers/github/installations", route => route.fulfill({ json: [] })); + await page.route("**/api/v1/integrations", route => route.fulfill({ json: { linear: null, clickup: null, exe: null, exeConnections: [], ampConnections: [], cloudflareTail: { count: 0, installations: [] } } })); + let generationId = "11111111-1111-4111-8111-111111111111", firstGenerationId, encryptedVerification = "", setupDone = false, rotationBody; + const verification = () => setupDone ? ({ generationId, encryptedVerification }) : null; + await page.route("**/api/v1/unlock-key/verification", async route => { + if (route.request().method() === "GET") return route.fulfill({ json: { ownerId: "owner", verification: verification() } }); + const body = route.request().postDataJSON(); + assert.equal("unlockKey" in body, false); + if (!setupDone) firstGenerationId = body.generationId; + generationId = body.generationId; encryptedVerification = body.encryptedVerification; setupDone = true; + return route.fulfill({ json: { generationId } }); + }); + await page.route("**/api/v1/unlock-key/rotation-snapshot", route => route.fulfill({ json: { ownerId: "owner", generationId, revision: 0, records: [], changes: [] } })); + await page.route("**/api/v1/unlock-key/rotation", async route => { + rotationBody = route.request().postDataJSON(); + assert.equal("unlockKey" in rotationBody, false); + assert.deepEqual(rotationBody.records, []); + assert.deepEqual(rotationBody.changes, []); + generationId = rotationBody.generationId; encryptedVerification = rotationBody.encryptedVerification; setupDone = true; + return route.fulfill({ json: { generationId, revision: 1, duplicate: false } }); + }); + + await page.goto(origin + "/settings/integrations"); + await page.getByRole("button", { name: "Generate unlock key" }).click(); + const initialKey = await page.getByLabel("New unlock key").inputValue(); + await page.getByRole("checkbox", { name: "I saved this key somewhere safe." }).check(); + await page.getByRole("button", { name: "I saved my key" }).click(); + await page.getByText("Unlock key verification is set up.", { exact: false }).waitFor(); + + page.once("dialog", dialog => dialog.accept()); + await page.getByRole("button", { name: "Reset integrations and model accounts" }).click(); + const resetKey = await page.getByLabel("New unlock key").inputValue(); + assert.notEqual(resetKey, initialKey); + await page.getByRole("checkbox", { name: "I saved the new key somewhere safe." }).check(); + await page.getByRole("button", { name: "Reset and replace key" }).click(); + await page.getByRole("button", { name: "Rotate unlock key" }).waitFor(); + assert.equal(rotationBody.expectedGenerationId, firstGenerationId); + assert.equal(JSON.stringify(rotationBody).includes(resetKey), false); + + await page.getByRole("region", { name: "Unlock key" }).getByLabel("Unlock key").fill(resetKey); + await page.getByRole("button", { name: "Rotate unlock key" }).click(); + const nextKey = await page.getByLabel("New unlock key").inputValue(); + await page.getByLabel("Current unlock key").fill(resetKey); + await page.getByRole("checkbox", { name: "I saved the new key somewhere safe." }).check(); + await page.getByRole("button", { name: "Replace unlock key" }).click(); + await page.getByRole("button", { name: "Rotate unlock key" }).waitFor(); + assert.equal(JSON.stringify(rotationBody).includes(nextKey), false); + await context.close(); +}); test("bearer integrations encrypt in the browser, clear secrets, handle conflicts and delete conditionally", async () => { const { context, page } = await setup(); await page.route("**/api/v1/providers/github/installations", route => route.fulfill({ json: [] })); diff --git a/packages/app/test/unlock-crypto.test.mjs b/packages/app/test/unlock-crypto.test.mjs index 2ac54691..af0ffb55 100644 --- a/packages/app/test/unlock-crypto.test.mjs +++ b/packages/app/test/unlock-crypto.test.mjs @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { createUnlockVerification, decryptRecord, encryptAfterUnlockVerification, encryptRecord, generateUnlockKey, verifyUnlockKey } from "../src/lib/unlock-crypto.ts"; +import { createUnlockVerification, decryptModelAccountCacheRecord, decryptRecord, encryptAfterUnlockVerification, encryptModelAccountCacheRecord, encryptRecord, generateUnlockKey, verifyUnlockKey } from "../src/lib/unlock-crypto.ts"; const key = "fzk1.AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8"; const binding = { ownerId: "owner-vector", recordId: "record-vector", recordType: "integration", destination: "https://api.example.test", revision: 7 }; @@ -25,6 +25,41 @@ test("fresh nonces make repeated encryptions differ", async () => { assert.notEqual(first.ciphertext, second.ciphertext); }); +test("initial model-account cache uses the proxy key-generation and record-revision binding", async () => { + const unlockKey = generateUnlockKey(), modelBinding = { + ownerId: "owner-a", recordId: "11111111-1111-4111-8111-111111111111", recordType: "model", + destination: "factorize", keyGeneration: "22222222-2222-4222-8222-222222222222", revision: 4, + }; + const encrypted = await encryptModelAccountCacheRecord(encoder.encode("{}"), unlockKey, modelBinding); + const [nonce, ciphertext] = encrypted.split(".").map(value => Uint8Array.from(atob(value), char => char.charCodeAt(0))); + const raw = Uint8Array.from(atob(unlockKey.slice(5).replaceAll("-", "+").replaceAll("_", "/") + "="), char => char.charCodeAt(0)); + try { + const material = await crypto.subtle.importKey("raw", raw, "HKDF", false, ["deriveKey"]); + const aes = await crypto.subtle.deriveKey({ name: "HKDF", hash: "SHA-256", salt: encoder.encode("factorize.unlock-key.v1"), info: encoder.encode("factorize.record.aes-256-gcm.v1") }, material, { name: "AES-GCM", length: 256 }, false, ["decrypt"]); + const aad = encoder.encode(JSON.stringify(["factorize-record-v1", modelBinding.ownerId, modelBinding.recordId, "model", "factorize", modelBinding.keyGeneration, modelBinding.revision])); + const plain = await crypto.subtle.decrypt({ name: "AES-GCM", iv: nonce, additionalData: aad, tagLength: 128 }, aes, ciphertext); + assert.equal(new TextDecoder().decode(plain), "{}"); + const staleAad = encoder.encode(JSON.stringify(["factorize-record-v1", modelBinding.ownerId, modelBinding.recordId, "model", "factorize", modelBinding.keyGeneration, modelBinding.revision + 1])); + await assert.rejects(crypto.subtle.decrypt({ name: "AES-GCM", iv: nonce, additionalData: staleAad, tagLength: 128 }, aes, ciphertext)); + } finally { raw.fill(0); } +}); + +test("model-account cache can be re-encrypted under a replacement generation while preserving its revision", async () => { + const oldKey = generateUnlockKey(), nextKey = generateUnlockKey(), binding = { + ownerId: "owner-a", recordId: "11111111-1111-4111-8111-111111111111", recordType: "model", + destination: "factorize", keyGeneration: "22222222-2222-4222-8222-222222222222", revision: 9, + }; + const oldPayload = await encryptModelAccountCacheRecord(encoder.encode('{"accessToken":"synthetic"}'), oldKey, binding); + const plaintext = await decryptModelAccountCacheRecord(oldPayload, oldKey, binding); + const nextGeneration = "33333333-3333-4333-8333-333333333333"; + try { + const nextPayload = await encryptModelAccountCacheRecord(plaintext, nextKey, { ...binding, keyGeneration: nextGeneration }); + assert.equal(text(await decryptModelAccountCacheRecord(nextPayload, nextKey, { ...binding, keyGeneration: nextGeneration })), '{"accessToken":"synthetic"}'); + await assert.rejects(decryptModelAccountCacheRecord(nextPayload, oldKey, { ...binding, keyGeneration: nextGeneration })); + await assert.rejects(decryptModelAccountCacheRecord(nextPayload, nextKey, { ...binding, keyGeneration: nextGeneration, revision: 10 })); + } finally { plaintext.fill(0); } +}); + test("wrong key and every authenticated binding change fail closed", async () => { await assert.rejects(decryptRecord(vector, "fzk1.AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE", binding)); for (const field of Object.keys(binding)) { diff --git a/packages/credential-proxy/Dockerfile b/packages/credential-proxy/Dockerfile new file mode 100644 index 00000000..315913bb --- /dev/null +++ b/packages/credential-proxy/Dockerfile @@ -0,0 +1,25 @@ +# syntax=docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e +FROM node:24.12.0-bookworm-slim@sha256:7326fb2dbdce998edd72140946851be64ef4a643e8715e138ca467e8e9d92c99 + +ARG SOURCE_REVISION=unknown +LABEL org.opencontainers.image.title="Factorize credential proxy" \ + org.opencontainers.image.source="https://github.com/GenerationSoftware/factorize" \ + org.opencontainers.image.revision="$SOURCE_REVISION" + +ENV NODE_ENV=production \ + FACTORIZE_PROXY_TUNNEL_PORT=9090 + +WORKDIR /opt/factorize/credential-proxy +COPY packages/credential-proxy/image/package.json packages/credential-proxy/image/package-lock.json /opt/codex/ +RUN npm ci --prefix /opt/codex --omit=dev --ignore-scripts \ + && ln -s /opt/codex/node_modules/.bin/codex /usr/local/bin/codex \ + && test "$(codex --version)" = "codex-cli 0.167.1" \ + && npm cache clean --force + +COPY packages/credential-proxy/src/ ./src/ +RUN mkdir -p /var/lib/factorize/codex-accounts /run/factorize-proxy \ + && chown -R node:node /opt/factorize /opt/codex /var/lib/factorize /run/factorize-proxy + +USER node +EXPOSE 9090 +ENTRYPOINT ["node", "/opt/factorize/credential-proxy/src/main.mjs"] diff --git a/packages/credential-proxy/image/package-lock.json b/packages/credential-proxy/image/package-lock.json new file mode 100644 index 00000000..170f488f --- /dev/null +++ b/packages/credential-proxy/image/package-lock.json @@ -0,0 +1,137 @@ +{ + "name": "factorize-credential-proxy-image-tools", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "factorize-credential-proxy-image-tools", + "version": "1.0.0", + "dependencies": { + "@openai/codex": "0.167.1" + } + }, + "node_modules/@openai/codex": { + "version": "0.167.1", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1.tgz", + "integrity": "sha512-39uApTRT9bkhE8tMXTIfZxaZ27oi7hPqWoF2Coyp+iQFAD3NVHgH838IYVUfJaX85leMSP2s4JOmVw0mMSCN5Q==", + "license": "Apache-2.0", + "bin": { + "codex": "bin/codex.js" + }, + "engines": { + "node": ">=16" + }, + "optionalDependencies": { + "@openai/codex-darwin-arm64": "npm:@openai/codex@0.167.1-darwin-arm64", + "@openai/codex-darwin-x64": "npm:@openai/codex@0.167.1-darwin-x64", + "@openai/codex-linux-arm64": "npm:@openai/codex@0.167.1-linux-arm64", + "@openai/codex-linux-x64": "npm:@openai/codex@0.167.1-linux-x64", + "@openai/codex-win32-arm64": "npm:@openai/codex@0.167.1-win32-arm64", + "@openai/codex-win32-x64": "npm:@openai/codex@0.167.1-win32-x64" + } + }, + "node_modules/@openai/codex-darwin-arm64": { + "name": "@openai/codex", + "version": "0.167.1-darwin-arm64", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1-darwin-arm64.tgz", + "integrity": "sha512-IMUByWGPhdbJFZzuk18iIdF/mfd21cPwalRTp5vGYIsOCyv3SJGRY5snbzZd0zO37WGuNjfUvH/62LvQHqNT2A==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16" + } + }, + "node_modules/@openai/codex-darwin-x64": { + "name": "@openai/codex", + "version": "0.167.1-darwin-x64", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1-darwin-x64.tgz", + "integrity": "sha512-SwJf87Qtuktk8qHSDsbNkjVjOerqQRCBgdrqw6gvAlF3tO8UizEp/Vs8qQVznNI1aJfFc6GoJLvcgBGVWVylDg==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16" + } + }, + "node_modules/@openai/codex-linux-arm64": { + "name": "@openai/codex", + "version": "0.167.1-linux-arm64", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1-linux-arm64.tgz", + "integrity": "sha512-z13jj4X4GDk7CTd1kXHkhXC8Gj8zomd1jYsU2//0KjevbLelPxowWLSTqIcR1rarJCtzveBHsS8gR7B9pAWB0g==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16" + } + }, + "node_modules/@openai/codex-linux-x64": { + "name": "@openai/codex", + "version": "0.167.1-linux-x64", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1-linux-x64.tgz", + "integrity": "sha512-qZteglyvMCj1QVymMyPyKdtw38HaXCv5gvw8WN+N0+JJstu3k/d8xiYj9ciL7OwFLhcujgP8bCGJgaREeqrHkw==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16" + } + }, + "node_modules/@openai/codex-win32-arm64": { + "name": "@openai/codex", + "version": "0.167.1-win32-arm64", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1-win32-arm64.tgz", + "integrity": "sha512-KFUMwSMBlQmabCSbpfv7AgouF+kNJE8iNw/nYv+IgHbHCY0v1pAzLFdt3BscYRUD+KNOeJqnp6n4xwCteA5OcQ==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16" + } + }, + "node_modules/@openai/codex-win32-x64": { + "name": "@openai/codex", + "version": "0.167.1-win32-x64", + "resolved": "https://registry.npmjs.org/@openai/codex/-/codex-0.167.1-win32-x64.tgz", + "integrity": "sha512-yvUA7s46paDIT5I3Phz8KcoGpSGlWR3pOS0npqQ3LQMpjtTOBWoLPRdjuoKVreuKIuOLDMxwWCID4nwojbbrSA==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16" + } + } + } +} diff --git a/packages/credential-proxy/image/package.json b/packages/credential-proxy/image/package.json new file mode 100644 index 00000000..0adb2472 --- /dev/null +++ b/packages/credential-proxy/image/package.json @@ -0,0 +1,8 @@ +{ + "name": "factorize-credential-proxy-image-tools", + "private": true, + "version": "1.0.0", + "dependencies": { + "@openai/codex": "0.167.1" + } +} diff --git a/packages/credential-proxy/package.json b/packages/credential-proxy/package.json index 20c1c5c9..dced1316 100644 --- a/packages/credential-proxy/package.json +++ b/packages/credential-proxy/package.json @@ -3,6 +3,15 @@ "version": "0.1.0", "private": true, "type": "module", + "files": [ + "src" + ], + "bin": { + "factorize-credential-proxy": "./src/main.mjs", + "factorize-proxy-startup": "./src/startup.mjs", + "factorize-proxy-certify-vm": "./src/vm-certificate.mjs", + "factorize-proxy-authorize-run": "./src/authorize-run.mjs" + }, "scripts": { "start": "node src/main.mjs", "test": "node --test test/*.test.mjs" diff --git a/packages/credential-proxy/src/authorize-run.mjs b/packages/credential-proxy/src/authorize-run.mjs new file mode 100644 index 00000000..b569a498 --- /dev/null +++ b/packages/credential-proxy/src/authorize-run.mjs @@ -0,0 +1,52 @@ +import { connect } from "node:net"; + +const SOCKET_PATH = "/run/factorize-proxy/bootstrap.sock"; +const MAX_REQUEST_BYTES = 4096; +const MAX_RESPONSE_BYTES = 2048; + +async function readRequest() { + let length = 0; + const chunks = []; + for await (const chunk of process.stdin) { + length += chunk.byteLength; + if (length > MAX_REQUEST_BYTES) throw new Error("invalid run authorization request"); + chunks.push(chunk); + } + if (!length || chunks.at(-1).at(-1) !== 10) throw new Error("invalid run authorization request"); + const value = JSON.parse(Buffer.concat(chunks).toString("utf8")); + if (!value || typeof value !== "object" || Array.isArray(value) + || Object.keys(value).sort().join(",") !== "credential,runId,vmIdentity" + || typeof value.credential !== "string" || typeof value.runId !== "string" || typeof value.vmIdentity !== "string") throw new Error("invalid run authorization request"); + return { type: "authorize-run", ...value }; +} + +async function send(request) { + const socket = connect(SOCKET_PATH); + try { + return await new Promise((resolve, reject) => { + let data = Buffer.alloc(0); + socket.on("data", chunk => { + data = Buffer.concat([data, chunk]); + if (data.length > MAX_RESPONSE_BYTES) { reject(new Error("invalid run authorization response")); socket.destroy(); return; } + const end = data.indexOf(10); + if (end < 0) return; + try { resolve(JSON.parse(data.subarray(0, end).toString("utf8"))); } + catch { reject(new Error("invalid run authorization response")); } + socket.end(); + }); + socket.once("error", () => reject(new Error("proxy run authorization service unavailable"))); + socket.setTimeout(7000, () => reject(new Error("proxy run authorization service timed out"))); + socket.once("connect", () => socket.write(`${JSON.stringify(request)}\n`)); + }); + } finally { socket.destroy(); } +} + +try { + const request = await readRequest(); + const result = await send(request); + if (result?.authorized !== true || result.runId !== request.runId || result.vmIdentity !== request.vmIdentity) throw new Error("run authorization was not confirmed"); + process.stdout.write("Run credential delivered to proxy memory\n"); +} catch { + process.stderr.write("Run credential delivery failed\n"); + process.exitCode = 1; +} diff --git a/packages/credential-proxy/src/bearer-forwarder.mjs b/packages/credential-proxy/src/bearer-forwarder.mjs new file mode 100644 index 00000000..37ea1854 --- /dev/null +++ b/packages/credential-proxy/src/bearer-forwarder.mjs @@ -0,0 +1,178 @@ +import https from "node:https"; +import { Transform } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { createOriginConnector } from "./destination-policy.mjs"; + +export const DEFAULT_FORWARD_LIMITS = Object.freeze({ + connectMs: 10_000, + headersMs: 60_000, + idleMs: 120_000, + requestMs: 15 * 60_000, + requestBytes: 32 * 1024 * 1024, + responseBytes: 64 * 1024 * 1024, + headerBytes: 32 * 1024, +}); + +const HOP_BY_HOP = new Set([ + "connection", "keep-alive", "proxy-authenticate", "proxy-authorization", + "te", "trailer", "transfer-encoding", "upgrade", "host", "authorization", + "content-length", "expect", +]); + +function safeHeaders(input, credential, maxBytes) { + const source = input instanceof Headers ? [...input.entries()] : Object.entries(input ?? {}); + const connectionTokens = new Set(); + for (const [name, value] of source) { + if (name.toLowerCase() === "connection") { + for (const token of String(value).split(",")) connectionTokens.add(token.trim().toLowerCase()); + } + } + const headers = Object.create(null); + let size = 0; + for (const [name, value] of source) { + const lower = name.toLowerCase(); + if (HOP_BY_HOP.has(lower) || lower.startsWith("x-forwarded-") || lower === "forwarded" || connectionTokens.has(lower)) continue; + const text = Array.isArray(value) ? value.join(", ") : String(value); + size += Buffer.byteLength(name) + Buffer.byteLength(text) + 4; + if (size > maxBytes) throw new RangeError("request headers exceed configured limit"); + headers[lower] = text; + } + headers.authorization = `Bearer ${credential}`; + return headers; +} + +function byteLimit(limit, label, onLimit) { + let bytes = 0; + return new Transform({ + transform(chunk, _encoding, callback) { + bytes += chunk.length; + if (bytes > limit) { + onLimit?.(); + callback(new RangeError(`${label} exceeds configured limit`)); + } else callback(null, chunk); + }, + }); +} + +function sendHttpsRequest({ socket, url, method, headers, body, limits, signal }) { + return new Promise((resolve, reject) => { + let settled = false; + let responseTimer; + let idleTimer; + let responseStream; + const agent = new https.Agent({ keepAlive: false }); + agent.createConnection = () => socket; + const finishError = (error) => { + if (settled) { + responseStream?.destroy(error); + return; + } + settled = true; + clearTimeout(responseTimer); + clearTimeout(idleTimer); + socket.destroy(error); + agent.destroy(); + reject(error); + }; + const request = https.request(url, { + method, + headers, + agent, + maxHeaderSize: limits.headerBytes, + signal, + }, (response) => { + clearTimeout(responseTimer); + let responseBytes = 0; + const resetIdle = () => { + clearTimeout(idleTimer); + idleTimer = setTimeout(() => responseStream.destroy(new Error("upstream response idle deadline exceeded")), limits.idleMs); + idleTimer.unref?.(); + }; + responseStream = new Transform({ + transform(chunk, _encoding, callback) { + responseBytes += chunk.length; + if (responseBytes > limits.responseBytes) return callback(new RangeError("upstream response exceeds configured limit")); + resetIdle(); + callback(null, chunk); + }, + }); + response.once("error", (error) => responseStream.destroy(error)); + responseStream.once("error", (error) => { + response.destroy(error); + request.destroy(error); + }); + responseStream.once("close", () => clearTimeout(idleTimer)); + responseStream.once("close", () => agent.destroy()); + if (response.statusCode >= 300 && response.statusCode < 400) { + response.resume(); + return finishError(new Error("upstream redirects are not permitted")); + } + resetIdle(); + response.pipe(responseStream); + settled = true; + resolve({ status: response.statusCode, headers: response.headers, body: responseStream }); + }); + request.once("error", finishError); + responseTimer = setTimeout(() => finishError(new Error("upstream response headers deadline exceeded")), limits.headersMs); + responseTimer.unref?.(); + if (!body) request.end(); + else { + const input = body[Symbol.asyncIterator] ? body : [body]; + pipeline(input, byteLimit(limits.requestBytes, "request body", () => request.destroy()), request, { signal }).catch(finishError); + } + }); +} + +/** + * Forward one request through a grant selected by authenticated connection state. + * The caller supplies the decrypted integration record; agent-controlled data + * cannot select either its credential or origin. + */ +export function createBearerForwarder({ connectorFactory = createOriginConnector, request = sendHttpsRequest, limits = DEFAULT_FORWARD_LIMITS } = {}) { + const active = new Set(); + return Object.freeze({ + async forward(integration, input, { signal } = {}) { + if (!integration || typeof integration.origin !== "string" || typeof integration.credential !== "string" || !integration.credential) { + throw new TypeError("integration grant is unavailable"); + } + if (active.size >= 16) throw new Error("concurrent request limit exceeded"); + const controller = new AbortController(); + const abort = () => controller.abort(signal?.reason); + signal?.addEventListener("abort", abort, { once: true }); + const timeout = setTimeout(() => controller.abort(new Error("request duration limit exceeded")), limits.requestMs); + timeout.unref?.(); + const key = {}; + active.add(key); + let released = false; + const release = () => { + if (released) return; + released = true; + clearTimeout(timeout); + signal?.removeEventListener("abort", abort); + active.delete(key); + }; + try { + const connect = connectorFactory(integration.origin, { timeoutMs: limits.connectMs }); + const { socket, url } = await connect(input.url); + if (controller.signal.aborted) { + socket.destroy(controller.signal.reason); + throw controller.signal.reason ?? new Error("request cancelled"); + } + const method = String(input.method ?? "GET").toUpperCase(); + if (!/^[A-Z!#$%&'*+.^_`|~-]+$/.test(method)) throw new TypeError("invalid HTTP method"); + const headers = safeHeaders(input.headers, integration.credential, limits.headerBytes); + const result = await request({ socket, url, method, headers, body: input.body, limits, signal: controller.signal }); + if (result?.body && typeof result.body.once === "function") { + result.body.once("end", release); + result.body.once("close", release); + result.body.once("error", release); + } else release(); + return result; + } catch (error) { + release(); + throw error; + } + }, + get activeRequests() { return active.size; }, + }); +} diff --git a/packages/credential-proxy/src/bootstrap-control.mjs b/packages/credential-proxy/src/bootstrap-control.mjs new file mode 100644 index 00000000..cfe6c94c --- /dev/null +++ b/packages/credential-proxy/src/bootstrap-control.mjs @@ -0,0 +1,110 @@ +import { createServer } from "node:net"; +import { chmod, mkdir, rm } from "node:fs/promises"; +import { dirname } from "node:path"; +import { registerProxyBoot } from "./control-client.mjs"; + +const MAX_BOOTSTRAP_BYTES = 4096; + +/** Local-only receiver for a provider exec channel; credentials never enter env or files. */ +export async function startProxyBootstrapControl({ runtime, socketPath = "/run/factorize-proxy/bootstrap.sock", register = registerProxyBoot, onRegistered = () => {}, issueVmCertificate, authorizeRun } = {}) { + if (!runtime || typeof register !== "function" || typeof onRegistered !== "function" + || (issueVmCertificate !== undefined && typeof issueVmCertificate !== "function") + || (authorizeRun !== undefined && typeof authorizeRun !== "function")) throw new TypeError("proxy bootstrap control requires a runtime, registrar, and registration callback"); + await mkdir(dirname(socketPath), { recursive: true, mode: 0o700 }); + await chmod(dirname(socketPath), 0o700); + await rm(socketPath, { force: true }); + const server = createServer((socket) => { + let data = Buffer.alloc(0); + let settled = false; + let processing = false; + const finish = (value) => { + if (settled) return; + settled = true; + socket.end(`${JSON.stringify(value)}\n`); + }; + socket.setTimeout(5000, () => finish({ error: "invalid bootstrap request" })); + socket.on("data", async (chunk) => { + if (settled || processing) return; + if (data.byteLength + chunk.byteLength > MAX_BOOTSTRAP_BYTES) return finish({ error: "invalid bootstrap request" }); + data = Buffer.concat([data, chunk]); + const end = data.indexOf(10); + if (end < 0) return; + if (end !== data.byteLength - 1) return finish({ error: "invalid bootstrap request" }); + let request; + try { request = JSON.parse(data.subarray(0, end).toString("utf8")); } + catch { return finish({ error: "invalid bootstrap request" }); } + if (request?.type === "issue-vm-certificate") { + if (!issueVmCertificate || request.bootEpoch !== runtime.bootEpoch + || typeof request.vmIdentity !== "string" || typeof request.csr !== "string" + || Object.keys(request).sort().join(",") !== "bootEpoch,csr,type,vmIdentity") { + return finish({ error: "invalid certificate request" }); + } + processing = true; + socket.pause(); + socket.removeAllListeners("data"); + try { + const issued = await issueVmCertificate({ vmIdentity: request.vmIdentity, csr: request.csr, bootEpoch: runtime.bootEpoch }); + if (typeof issued?.certificate !== "string" || typeof issued?.trustBundle !== "string") throw new Error("invalid certificate response"); + finish({ issued: true, bootEpoch: runtime.bootEpoch, vmIdentity: request.vmIdentity, certificate: issued.certificate, trustBundle: issued.trustBundle }); + } catch { + finish({ error: "certificate provisioning failed" }); + } finally { + request.csr = ""; + } + return; + } + if (request?.type === "authorize-run") { + if (!authorizeRun || Object.keys(request).sort().join(",") !== "credential,runId,type,vmIdentity" + || typeof request.runId !== "string" || typeof request.vmIdentity !== "string" || typeof request.credential !== "string") { + return finish({ error: "invalid run authorization" }); + } + processing = true; + socket.pause(); + socket.removeAllListeners("data"); + try { + await authorizeRun({ runId: request.runId, vmIdentity: request.vmIdentity, credential: request.credential }); + finish({ authorized: true, runId: request.runId, vmIdentity: request.vmIdentity }); + } catch { + finish({ error: "run authorization failed" }); + } finally { request.credential = ""; } + return; + } + if (request?.type !== "register-boot") return finish({ error: "invalid bootstrap request" }); + if (!request || typeof request !== "object" || Array.isArray(request) + || Object.keys(request).sort().join(",") !== "bootEpoch,browserOrigin,controlOrigin,credential,type" + || request.bootEpoch !== runtime.bootEpoch || typeof request.credential !== "string" + || typeof request.browserOrigin !== "string" + || !/^fzc_[A-Za-z0-9_-]{43}$/.test(request.credential)) { + return finish({ error: "invalid bootstrap request" }); + } + processing = true; + socket.pause(); + socket.removeAllListeners("data"); + let credential = request.credential; + try { + const registration = await register({ controlOrigin: request.controlOrigin, token: credential, bootEpoch: runtime.bootEpoch, browserOrigin: request.browserOrigin }); + // Keep the exchanged proxy credential inside the proxy process. The + // one-shot startup command receives only the public registration result. + await onRegistered(registration, { controlOrigin: request.controlOrigin, browserOrigin: request.browserOrigin, bootEpoch: runtime.bootEpoch }); + finish({ registered: true, proxyIdentity: registration.proxyIdentity, bootEpoch: registration.bootEpoch }); + } catch { + finish({ error: "proxy boot registration failed" }); + } finally { + credential = ""; + request.credential = ""; + } + }); + socket.on("error", () => {}); + }); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(socketPath, () => { server.removeListener("error", reject); resolve(); }); + }); + await chmod(socketPath, 0o600); + return Object.freeze({ + close: async () => { + await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + await rm(socketPath, { force: true }); + }, + }); +} diff --git a/packages/credential-proxy/src/browser-control.mjs b/packages/credential-proxy/src/browser-control.mjs new file mode 100644 index 00000000..930498b8 --- /dev/null +++ b/packages/credential-proxy/src/browser-control.mjs @@ -0,0 +1,165 @@ +import { createServer } from "node:http"; +import { createPublicKey, verify } from "node:crypto"; + +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const MAX_BODY_BYTES = 4096; + +function decodePart(value) { + if (!/^[A-Za-z0-9_-]+$/.test(value)) throw new Error("invalid token"); + return Buffer.from(value, "base64url"); +} + +function parseCapability(token, publicKey) { + if (typeof token !== "string" || token.length > 4096) throw new Error("invalid capability"); + const parts = token.split("."); + if (parts.length !== 3) throw new Error("invalid capability"); + const [headerPart, claimsPart, signaturePart] = parts; + const header = JSON.parse(decodePart(headerPart).toString("utf8")); + const claims = JSON.parse(decodePart(claimsPart).toString("utf8")); + if (header?.alg !== "EdDSA" || header?.typ !== "JWT" + || !verify(null, Buffer.from(`${headerPart}.${claimsPart}`), publicKey, decodePart(signaturePart))) { + throw new Error("invalid capability"); + } + return claims; +} + +function json(response, status, body, origin) { + const headers = { "Cache-Control": "no-store", "Content-Type": "application/json; charset=utf-8" }; + if (origin) { + headers["Access-Control-Allow-Origin"] = origin; + headers.Vary = "Origin"; + } + response.writeHead(status, headers).end(JSON.stringify(body)); +} + +function publicAuthorization(value) { + if (!value || typeof value !== "object") return { state: "idle" }; + const result = { state: value.state }; + const challenge = value.challenge; + if (challenge?.verificationUrl === "https://auth.openai.com/codex/device" + && typeof challenge.userCode === "string" && (/^[A-Z0-9]{4,6}-[A-Z0-9]{4,6}$/u.test(challenge.userCode) || /^[A-Z0-9]{8,12}$/u.test(challenge.userCode))) { + result.verificationUrl = challenge.verificationUrl; + result.userCode = challenge.userCode; + } + return result; +} + +async function readJson(request) { + const chunks = []; + let size = 0; + for await (const chunk of request) { + size += chunk.byteLength; + if (size > MAX_BODY_BYTES) throw Object.assign(new Error("request too large"), { status: 413 }); + chunks.push(chunk); + } + try { return JSON.parse(Buffer.concat(chunks).toString("utf8")); } + catch { throw Object.assign(new Error("invalid request"), { status: 400 }); } +} + +/** Private Fly service endpoint for owner-browser model-account operations. + * The Fly edge terminates public HTTPS; this listener must only be reachable + * through that service and never through the agent SSH integration channel. */ +export async function startBrowserControl({ + runtime, + lifecycle, + publicKey, + proxyId, + ownerId, + generation, + appOrigin, + host = "0.0.0.0", + port = 8080, + now = () => Math.floor(Date.now() / 1000), +} = {}) { + if (!runtime || !lifecycle || !publicKey || typeof proxyId !== "string" || !proxyId + || !UUID.test(ownerId ?? "") || !(Number.isSafeInteger(generation) && generation > 0 || UUID.test(generation ?? "")) + || !Number.isInteger(port) || port < 0 || port > 65535) { + throw new TypeError("browser control configuration is invalid"); + } + const expectedOrigin = new URL(appOrigin); + if (expectedOrigin.protocol !== "https:" || expectedOrigin.origin !== appOrigin) throw new TypeError("browser control origin must be canonical HTTPS"); + const verifyKey = publicKey?.type === "public" ? publicKey : createPublicKey(publicKey); + const consumed = new Map(); + const authorizations = new Map(); + + const server = createServer(async (request, response) => { + const origin = request.headers.origin; + if (origin !== appOrigin) return json(response, 403, { error: "origin_rejected" }); + if (request.method === "OPTIONS") { + if (request.headers["access-control-request-method"] !== "POST" + || request.headers["access-control-request-headers"]?.split(",").map(value => value.trim().toLowerCase()).sort().join(",") !== "authorization,content-type") { + return json(response, 403, { error: "preflight_rejected" }); + } + response.writeHead(204, { + "Access-Control-Allow-Origin": appOrigin, + "Access-Control-Allow-Methods": "POST", + "Access-Control-Allow-Headers": "Authorization, Content-Type", + "Access-Control-Max-Age": "60", + "Cache-Control": "no-store", + Vary: "Origin", + }).end(); + return; + } + if (request.method !== "POST") return json(response, 405, { error: "method_not_allowed" }, appOrigin); + const match = /^\/model-accounts\/([0-9a-f-]{36})\/(authorize|status|cancel)$/.exec(new URL(request.url ?? "/", "http://proxy.internal").pathname); + if (!match || !UUID.test(match[1])) return json(response, 404, { error: "not_found" }, appOrigin); + const [, accountId, operation] = match; + if (request.headers["content-type"]?.split(";", 1)[0]?.trim().toLowerCase() !== "application/json") { + return json(response, 415, { error: "unsupported_media_type" }, appOrigin); + } + const authorization = request.headers.authorization ?? ""; + if (!authorization.startsWith("Bearer ")) return json(response, 401, { error: "invalid_capability" }, appOrigin); + let claims; + try { claims = parseCapability(authorization.slice(7), verifyKey); } + catch { return json(response, 401, { error: "invalid_capability" }, appOrigin); } + const expectedAction = `model-account:${operation}`; + if (claims.iss !== "factorize" || claims.aud !== proxyId || claims.sub !== ownerId + || claims.action !== expectedAction || claims.accountId !== accountId || claims.origin !== appOrigin + || claims.generation !== generation || !Number.isSafeInteger(claims.exp) || claims.exp <= now() + || claims.exp > now() + 60 || typeof claims.jti !== "string" || !/^[A-Za-z0-9_-]{16,128}$/.test(claims.jti)) { + return json(response, 401, { error: "invalid_capability" }, appOrigin); + } + for (const [id, expiry] of consumed) if (expiry <= now()) consumed.delete(id); + if (consumed.has(claims.jti)) return json(response, 401, { error: "capability_replayed" }, appOrigin); + consumed.set(claims.jti, claims.exp); + + try { + const body = await readJson(request); + if (!body || typeof body !== "object" || Array.isArray(body) || Object.keys(body).length) { + return json(response, 400, { error: "invalid_request" }, appOrigin); + } + if (operation === "authorize") { + if (runtime.state !== "unlocked") return json(response, 423, { error: "proxy_locked" }, appOrigin); + const login = await lifecycle.deviceLogin(accountId); + const state = { state: "pending", challenge: login.challenge }; + authorizations.set(accountId, state); + Promise.resolve(login.completion).then(result => { + if (authorizations.get(accountId) === state) { + const next = result?.state === "saved" ? "authorized" : result?.state === "unsaved" ? "authorized-unsaved" : result?.state ?? "reauthentication-required"; + authorizations.set(accountId, { state: next }); + } + }, () => { + if (authorizations.get(accountId) === state) authorizations.set(accountId, { state: "failed" }); + }); + return json(response, 202, publicAuthorization(state), appOrigin); + } + if (operation === "status") return json(response, 200, publicAuthorization(authorizations.get(accountId)), appOrigin); + const result = await lifecycle.cancel(accountId); + authorizations.set(accountId, { state: "canceled" }); + return json(response, 200, { state: result?.state === "idle" ? "idle" : "canceled" }, appOrigin); + } catch (error) { + return json(response, Number.isInteger(error?.status) ? error.status : 400, + { error: "model_account_operation_failed" }, appOrigin); + } + }); + server.on("connect", (_request, socket) => socket.end("HTTP/1.1 405 Method Not Allowed\r\nConnection: close\r\n\r\n")); + server.on("upgrade", (_request, socket) => socket.destroy()); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(port, host, () => { server.removeListener("error", reject); resolve(); }); + }); + return Object.freeze({ + address: server.address(), + close: () => new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())), + }); +} diff --git a/packages/credential-proxy/src/codex-auth-helper.mjs b/packages/credential-proxy/src/codex-auth-helper.mjs new file mode 100644 index 00000000..8f888875 --- /dev/null +++ b/packages/credential-proxy/src/codex-auth-helper.mjs @@ -0,0 +1,163 @@ +import { mkdir, lstat } from "node:fs/promises"; +import { resolve, sep } from "node:path"; +import { spawn as spawnDefault } from "node:child_process"; + +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const PINNED_VERSION = "codex-cli 0.167.1"; +const METHODS = new Set(["device-login", "account-status", "refresh", "cancel", "logout"]); + +function accountHome(root, accountId) { + if (typeof accountId !== "string" || !UUID.test(accountId)) throw new TypeError("invalid model account id"); + const base = resolve(root); + const home = resolve(base, accountId); + if (!home.startsWith(`${base}${sep}`)) throw new TypeError("invalid model account id"); + return { base, home }; +} + +/** + * Narrow in-process adapter for Codex authentication commands. Callers must + * keep this object on the proxy's private control channel; it is not a job API. + * Each account gets its own normal Codex credential home. + */ +export function createCodexAuthHelper({ + codexPath = "/usr/local/bin/codex", + accountHomeRoot = "/var/lib/factorize/codex-accounts", + spawnProcess = spawnDefault, + expectedVersion = PINNED_VERSION, +} = {}) { + const logins = new Map(); + + async function prepareHome(accountId) { + const { base, home } = accountHome(accountHomeRoot, accountId); + await mkdir(base, { recursive: true, mode: 0o700 }); + await mkdir(home, { recursive: true, mode: 0o700 }); + const stat = await lstat(home); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Codex account home is not a private directory"); + return home; + } + + function run(accountId, args, { timeoutMs = 10_000, input, onStdout } = {}) { + return prepareHome(accountId).then(home => new Promise((resolveRun, rejectRun) => { + let stdout = "", stderr = "", settled = false; + const child = spawnProcess(codexPath, args, { + env: { PATH: process.env.PATH ?? "", CODEX_HOME: home, HOME: home }, + stdio: ["pipe", "pipe", "pipe"], + windowsHide: true, + }); + const finish = (error, result) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (error) rejectRun(error); + else resolveRun(result); + }; + const collect = (current, chunk, limit) => { + const next = current + chunk.toString("utf8"); + if (Buffer.byteLength(next) > limit) { + child.kill("SIGTERM"); + finish(new Error("Codex authentication helper output exceeded its limit")); + return current; + } + return next; + }; + child.stdout.on("data", chunk => { + stdout = collect(stdout, chunk, 16_384); + if (!settled) onStdout?.(chunk.toString("utf8")); + }); + child.stderr.on("data", chunk => { stderr = collect(stderr, chunk, 16_384); }); + child.once("error", () => finish(new Error("Codex authentication helper could not start"))); + child.once("close", (code, signal) => { + if (code === 0) finish(null, { code, stdout, stderr }); + else finish(new Error(signal ? `Codex authentication helper stopped (${signal})` : `Codex authentication helper failed (${code ?? "unknown"})`)); + }); + const timer = setTimeout(() => { + child.kill("SIGTERM"); + finish(new Error("Codex authentication helper timed out")); + }, timeoutMs); + timer.unref?.(); + if (input !== undefined) child.stdin.end(input); + else child.stdin.end(); + return child; + })); + } + + async function checkVersion(accountId) { + // `--version` is the only non-authentication command accepted by this helper. + const result = await run(accountId, ["--version"]); + if (result.stdout.trim() !== expectedVersion) throw new Error("Unsupported Codex version for model account authentication"); + } + + async function invoke(request) { + if (!request || typeof request !== "object" || Array.isArray(request) + || !METHODS.has(request.method) || Object.keys(request).some(key => !["method", "accountId"].includes(key))) { + throw new TypeError("unsupported Codex authentication operation"); + } + const { accountId } = request; + accountHome(accountHomeRoot, accountId); + await checkVersion(accountId); + if (request.method === "device-login") { + if (logins.has(accountId)) throw new Error("device authorization is already running for this account"); + const home = await prepareHome(accountId); + let child; + const challenge = {}; + const completion = new Promise((resolveRun, rejectRun) => { + child = spawnProcess(codexPath, ["login", "--device-auth"], { + env: { PATH: process.env.PATH ?? "", CODEX_HOME: home, HOME: home }, + stdio: ["ignore", "pipe", "pipe"], windowsHide: true, + }); + let stdout = ""; + const timer = setTimeout(() => { child.kill("SIGTERM"); rejectRun(new Error("Codex device authorization timed out")); }, 15 * 60_000); + timer.unref?.(); + child.stdout.on("data", chunk => { + stdout = (stdout + chunk.toString("utf8")).slice(-16_384); + const url = stdout.match(/https:\/\/auth\.openai\.com\/codex\/device(?:\b|\/)/u)?.[0]; + const code = stdout.match(/\b(?:user\s+)?code\s*:\s*([A-Z0-9]{4,6}-[A-Z0-9]{4,6}|[A-Z0-9]{8,12})\b/iu)?.[1]; + if (url) challenge.verificationUrl = "https://auth.openai.com/codex/device"; + if (code) challenge.userCode = code.toUpperCase(); + }); + child.stderr.on("data", chunk => { + // Retain no provider error text, which could include device details. + if (chunk.byteLength > 16_384) child.kill("SIGTERM"); + }); + child.once("error", () => { clearTimeout(timer); rejectRun(new Error("Codex authentication helper could not start")); }); + child.once("close", (code, signal) => { + clearTimeout(timer); + if (code === 0) resolveRun({ state: "authorized", output: stdout }); + else rejectRun(new Error(signal ? `Codex device authorization stopped (${signal})` : `Codex device authorization failed (${code ?? "unknown"})`)); + }); + }); + const entry = { child, completion }; + logins.set(accountId, entry); + completion.finally(() => { if (logins.get(accountId) === entry) logins.delete(accountId); }).catch(() => {}); + return { state: "pending", challenge, completion }; + } + if (request.method === "cancel") { + const login = logins.get(accountId); + if (!login) return { state: "idle" }; + login.child.kill("SIGTERM"); + logins.delete(accountId); + return { state: "canceled" }; + } + if (request.method === "logout") { + const login = logins.get(accountId); + if (login) { + login.child.kill("SIGTERM"); + logins.delete(accountId); + } + await run(accountId, ["logout"]); + return { state: "logged-out" }; + } + if (request.method === "account-status") { + const result = await run(accountId, ["login", "status"]); + return { state: result.stdout.trim() ? "authenticated" : "unauthenticated" }; + } + // Codex owns refresh semantics in its authentication manager. A status call + // exercises that manager without opening the agent command surface. + const result = await run(accountId, ["login", "status"]); + return { state: result.stdout.trim() ? "refreshed" : "reauthentication-required" }; + } + + return Object.freeze({ invoke }); +} + +export { PINNED_VERSION as CODEX_AUTH_HELPER_VERSION }; diff --git a/packages/credential-proxy/src/control-client.mjs b/packages/credential-proxy/src/control-client.mjs new file mode 100644 index 00000000..57e31c85 --- /dev/null +++ b/packages/credential-proxy/src/control-client.mjs @@ -0,0 +1,261 @@ +const bootEpochPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +/** Register one proxy process boot using a provisioning-scoped credential. The token is only held in this call's memory. */ +function validateOrigin(controlOrigin) { + const origin = new URL(controlOrigin); + if (origin.protocol !== "https:" || origin.pathname !== "/" || origin.username || origin.password || origin.search || origin.hash) throw new TypeError("proxy control origin must be an HTTPS origin"); + return origin; +} + +export async function registerProxyBoot({ controlOrigin, token, bootEpoch, browserOrigin, fetchImpl = fetch }) { + const origin = validateOrigin(controlOrigin); + if (!/^fzc_[A-Za-z0-9_-]{43}$/.test(token ?? "")) throw new TypeError("invalid proxy control credential"); + if (!bootEpochPattern.test(bootEpoch ?? "")) throw new TypeError("invalid proxy boot epoch"); + const browser = new URL(browserOrigin); + if (browser.protocol !== "https:" || browser.origin !== browserOrigin || browser.username || browser.password) throw new TypeError("invalid browser control origin"); + let credential = token; + try { + const response = await fetchImpl(new URL("/internal/credential-proxy/register-boot", origin), { + method: "POST", + headers: { "Content-Type": "application/json", Authorization: `Bearer ${credential}` }, + body: JSON.stringify({ bootEpoch, browserOrigin }), + signal: AbortSignal.timeout(10_000), + redirect: "error", + }); + if (!response.ok) throw new Error("proxy boot registration rejected"); + const result = await response.json(); + if (result?.registered !== true || result.bootEpoch !== bootEpoch || typeof result.proxyIdentity !== "string" + || !uuidPattern.test(result.tenantId ?? "") || !/^fzc_[A-Za-z0-9_-]{43}$/.test(result.controlCredential ?? "")) throw new Error("invalid proxy boot registration response"); + return { tenantId: result.tenantId, proxyIdentity: result.proxyIdentity, bootEpoch: result.bootEpoch, controlCredential: result.controlCredential }; + } finally { + credential = ""; + } +} + +async function proxyControlRequest({ controlOrigin, token, path, body, fetchImpl }) { + const origin = validateOrigin(controlOrigin); + if (!/^fzc_[A-Za-z0-9_-]{43}$/.test(token ?? "")) throw new TypeError("invalid proxy control credential"); + const response = await fetchImpl(new URL(path, origin), { + method: "POST", headers: { "Content-Type": "application/json", Authorization: `Bearer ${token}` }, + body: JSON.stringify(body), signal: AbortSignal.timeout(10_000), redirect: "error", + }); + if (!response.ok) { + let code; + try { code = (await response.clone().json())?.error; } catch { /* Keep error handling bounded to the HTTP status. */ } + const error = new Error("proxy control request rejected"); + error.name = "ProxyControlRequestError"; + error.status = response.status; + if (typeof code === "string") error.code = code; + throw error; + } + return response.json(); +} + +export async function issueProxyEnrollment(options) { + const result = await proxyControlRequest({ ...options, path: "/internal/credential-proxy/enroll" }); + if (typeof result?.token !== "string" || typeof result?.enrollmentId !== "string" || !Number.isSafeInteger(result?.grantRevision)) throw new Error("invalid proxy enrollment response"); + return result; +} + +export async function consumeProxyEnrollment(options) { + const result = await proxyControlRequest({ ...options, path: "/internal/credential-proxy/consume-enrollment" }); + if (result?.consumed !== true || typeof result?.binding?.enrollmentId !== "string") throw new Error("invalid consumed enrollment response"); + return result.binding; +} + +export async function reportProxyRuntimeStatus({ controlOrigin, token, bootEpoch, unlocked, fetchImpl = fetch } = {}) { + if (!bootEpochPattern.test(bootEpoch ?? "") || typeof unlocked !== "boolean") throw new TypeError("invalid proxy runtime status"); + const result = await proxyControlRequest({ controlOrigin, token, path: "/internal/credential-proxy/status", body: { bootEpoch, unlocked }, fetchImpl }); + if (result?.updated !== true) throw new Error("invalid proxy runtime status response"); + return result; +} + +export function startProxyRuntimeStatusReporter({ controlOrigin, token, bootEpoch, runtime, intervalMs = 10_000, fetchImpl = fetch } = {}) { + if (!runtime || typeof runtime.state !== "string" || !Number.isSafeInteger(intervalMs) || intervalMs < 1_000 || intervalMs > 60_000) { + throw new TypeError("invalid proxy runtime status reporter"); + } + let closed = false, pending = Promise.resolve(); + const report = () => { + if (closed) return; + pending = pending.then(() => reportProxyRuntimeStatus({ controlOrigin, token, bootEpoch, unlocked: runtime.state === "unlocked", fetchImpl })).catch(() => {}); + }; + report(); + const timer = setInterval(report, intervalMs); + timer.unref?.(); + return { async close() { closed = true; clearInterval(timer); await pending; } }; +} + +/** Read the current user-encrypted integration records selected by an active run. + * The control plane decrypts only Factorize's outer encryption layer. */ +export async function readProxyIntegrationSnapshot({ controlOrigin, token, binding, fetchImpl = fetch } = {}) { + if (!binding || !uuidPattern.test(binding.runId ?? "") || !Number.isSafeInteger(binding.grantRevision) || binding.grantRevision < 0) { + throw new TypeError("invalid proxy integration snapshot binding"); + } + const result = await proxyControlRequest({ + controlOrigin, + token, + path: "/internal/credential-proxy/integration-snapshot", + body: { runId: binding.runId, grantRevision: binding.grantRevision }, + fetchImpl, + }); + const modelAccount = result?.modelAccount; + if (typeof result?.ownerId !== "string" || !Array.isArray(result.records) + || result.records.some(record => !record || typeof record.id !== "string" || record.kind !== "bearer" + || !(record.deletedAt != null && record.encryptedPayload === null + || typeof record.origin === "string" && typeof record.encryptedPayload === "string") || !Number.isSafeInteger(record.revision)) + || !(modelAccount === null || (modelAccount && typeof modelAccount.id === "string" && modelAccount.kind === "model" + && modelAccount.origin === null && typeof modelAccount.encryptedPayload === "string" && Number.isSafeInteger(modelAccount.revision)))) { + throw new Error("invalid proxy integration snapshot response"); + } + return result; +} + +/** Concrete proxy-side adapters for revision-bound encrypted model caches. */ +export function createModelAccountPersistenceAdapter({ controlOrigin, controlCredential, ownerId, fetchImpl = fetch } = {}) { + if (typeof ownerId !== "string" || !ownerId || !/^fzc_[A-Za-z0-9_-]{43}$/.test(controlCredential ?? "")) { + throw new TypeError("model account persistence requires the registered proxy identity"); + } + const read = async accountId => { + if (!uuidPattern.test(accountId ?? "")) throw new TypeError("invalid model account id"); + const result = await proxyControlRequest({ controlOrigin, token: controlCredential, path: "/internal/credential-proxy/model-account-cache", body: { operation: "read", accountId }, fetchImpl }); + const account = result?.account; + if (result?.ownerId !== ownerId || !account || account.id !== accountId || account.kind !== "model" + || account.origin !== null || typeof account.encryptedPayload !== "string" + || !Number.isSafeInteger(account.revision) || !uuidPattern.test(account.keyGeneration ?? "")) { + throw new Error("invalid model account persistence response"); + } + return { ...account, ownerId }; + }; + return Object.freeze({ + async listAccounts() { + const result = await proxyControlRequest({ controlOrigin, token: controlCredential, path: "/internal/credential-proxy/model-account-cache", body: { operation: "list" }, fetchImpl }); + if (result?.ownerId !== ownerId || !Array.isArray(result.accounts) + || !uuidPattern.test(result.keyGeneration ?? "") + || result.accounts.some(account => !account || !uuidPattern.test(account.id ?? "") || account.kind !== "model" + || account.origin !== null || !Number.isSafeInteger(account.revision))) { + throw new Error("invalid model account list response"); + } + return result.accounts.map(account => ({ ...account, ownerId, keyGeneration: result.keyGeneration })); + }, + getAccount: read, + async getEncryptedCache(accountId) { + const account = await read(accountId); + return { accountId, ownerId, keyGeneration: account.keyGeneration, revision: account.revision, encryptedPayload: account.encryptedPayload }; + }, + async persistEncrypted({ id, expectedRevision, keyGeneration, encryptedPayload, idempotencyKey }) { + if (!uuidPattern.test(id ?? "") || !uuidPattern.test(idempotencyKey ?? "") || !Number.isSafeInteger(expectedRevision) + || !uuidPattern.test(keyGeneration ?? "") || typeof encryptedPayload !== "string" || encryptedPayload.length > 65_536) throw new TypeError("invalid encrypted model account update"); + const result = await proxyControlRequest({ controlOrigin, token: controlCredential, path: "/internal/credential-proxy/model-account-cache", + body: { operation: "save", accountId: id, expectedRevision, keyGeneration, encryptedPayload, idempotencyKey }, fetchImpl }); + if (result?.accountId !== id || result.revision !== expectedRevision + 1 || typeof result.duplicate !== "boolean") { + throw new Error("invalid model account persistence response"); + } + return result; + }, + }); +} + +/** Read durable revocations for the registered proxy boot. Replaying the full + * set is intentional: it closes runs revoked while this proxy was offline and + * runtime.revokeRun is idempotent. */ +export async function readProxyRunRevocations({ controlOrigin, token, bootEpoch, fetchImpl = fetch } = {}) { + return (await readProxyControlState({ controlOrigin, token, bootEpoch, fetchImpl })).revokedRunIds; +} + +export async function readProxyControlState({ controlOrigin, token, bootEpoch, fetchImpl = fetch } = {}) { + if (!bootEpochPattern.test(bootEpoch ?? "")) throw new TypeError("invalid proxy boot epoch"); + const result = await proxyControlRequest({ + controlOrigin, + token, + path: "/internal/credential-proxy/revocations", + body: { bootEpoch }, + fetchImpl, + }); + if (!Array.isArray(result?.revokedRunIds) || result.revokedRunIds.some(id => !uuidPattern.test(id))) { + throw new Error("invalid proxy revocation response"); + } + return { revokedRunIds: result.revokedRunIds, lockProxy: result.lockProxy === true }; +} + +/** Poll durable control-plane state. A failed request leaves established + * connections running and the next bounded poll retries when communication + * recovers; it does not add an authorization lease or expiry. */ +export function startProxyRunRevocationMonitor({ controlOrigin, controlCredential, bootEpoch, runtime, onRevoked = () => {}, intervalMs = 5_000, fetchImpl = fetch } = {}) { + if (!runtime || typeof runtime.revokeRun !== "function" || !Number.isInteger(intervalMs) || intervalMs < 100 || intervalMs > 60_000) { + throw new TypeError("invalid proxy revocation monitor configuration"); + } + let stopped = false; + let timer; + let polling; + const poll = async () => { + if (stopped || polling) return; + polling = (async () => { + try { + const state = await readProxyControlState({ controlOrigin, token: controlCredential, bootEpoch, fetchImpl }); + for (const runId of state.revokedRunIds) { await runtime.revokeRun(runId); await onRevoked(runId); } + if (state.lockProxy && runtime.state !== "locked") await runtime.lock(); + } catch { + // Keep existing connections alive during an outage and retry on schedule. + } finally { + polling = undefined; + if (!stopped) timer = setTimeout(poll, intervalMs); + } + })(); + }; + void poll(); + return Object.freeze({ + async close() { + stopped = true; + clearTimeout(timer); + await polling; + }, + }); +} + +const uuidPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +/** + * Create the proxy-side half of enrollment. The job VM supplies only its run + * ID over the already authenticated TLS connection; the proxy derives VM and + * boot identity locally, then issues and consumes a fresh one-use enrollment + * using its private control credential. Integration secrets stay in the + * proxy's local grant resolver and never cross this control API. + */ +export function createProxyEnrollmentAuthenticator({ controlOrigin, controlCredential, bootEpoch, resolveGrants, resolveRunCredential, fetchImpl = fetch } = {}) { + if (!/^fzc_[A-Za-z0-9_-]{43}$/.test(controlCredential ?? "")) throw new TypeError("invalid proxy control credential"); + if (!bootEpochPattern.test(bootEpoch ?? "")) throw new TypeError("invalid proxy boot epoch"); + if (typeof resolveGrants !== "function" || typeof resolveRunCredential !== "function") throw new TypeError("proxy enrollment requires local grant and run credential resolvers"); + validateOrigin(controlOrigin); + let credential = controlCredential; + return async function authenticateRun({ runId, vmIdentity, bootEpoch: requestedBootEpoch }) { + if (!uuidPattern.test(runId ?? "") || typeof vmIdentity !== "string" || !vmIdentity.trim() || vmIdentity.length > 255 + || requestedBootEpoch !== bootEpoch) throw new Error("invalid proxy enrollment identity"); + const runCredential = resolveRunCredential(runId, vmIdentity); + if (!/^fzt_run_[A-Za-z0-9_-]+_[A-Za-z0-9_-]{43}$/.test(runCredential ?? "")) throw new Error("run credential is unavailable for this VM"); + const options = { controlOrigin, token: credential, fetchImpl }; + const requested = { runId, vmIdentity, runCredential }; + const enrollment = await issueProxyEnrollment({ ...options, body: requested }); + const binding = await consumeProxyEnrollment({ ...options, body: { runId, vmIdentity, enrollmentToken: enrollment.token } }); + if (binding.runId !== runId || binding.vmIdentity !== vmIdentity || binding.bootEpoch !== bootEpoch + || typeof binding.tenantId !== "string" || typeof binding.proxyIdentity !== "string" + || binding.enrollmentId !== enrollment.enrollmentId || binding.grantRevision !== enrollment.grantRevision) { + throw new Error("consumed enrollment binding did not match the authenticated connection"); + } + const grants = await resolveGrants(binding); + if (!Array.isArray(grants)) throw new Error("proxy grants are unavailable"); + return { + authenticated: true, + enrollmentConsumed: true, + connectionId: binding.enrollmentId, + runId: binding.runId, + vmId: binding.vmIdentity, + vmIdentity: binding.vmIdentity, + bootEpoch: binding.bootEpoch, + tenantId: binding.tenantId, + proxyIdentity: binding.proxyIdentity, + grantRevision: binding.grantRevision, + grants, + refreshGrants: () => resolveGrants(binding), + }; + }; +} diff --git a/packages/credential-proxy/src/decrypted-grants.mjs b/packages/credential-proxy/src/decrypted-grants.mjs new file mode 100644 index 00000000..dd765af7 --- /dev/null +++ b/packages/credential-proxy/src/decrypted-grants.mjs @@ -0,0 +1,132 @@ +import { webcrypto } from "node:crypto"; +import { readProxyIntegrationSnapshot } from "./control-client.mjs"; + +const KEY_PREFIX = "fzk1."; +const KEY_BYTES = 32; +const NONCE_BYTES = 12; +const SALT = new TextEncoder().encode("factorize.unlock-key.v1"); +const INFO = new TextEncoder().encode("factorize.record.aes-256-gcm.v1"); +const encoder = new TextEncoder(); +const decoder = new TextDecoder("utf-8", { fatal: true }); + +function decodeBase64Url(value, label) { + if (typeof value !== "string" || !/^[A-Za-z0-9_-]+$/.test(value)) throw new TypeError(`malformed ${label}`); + const bytes = Buffer.from(value, "base64url"); + if (bytes.toString("base64url") !== value) throw new TypeError(`malformed ${label}`); + return new Uint8Array(bytes); +} + +function decodeStoredCiphertext(value) { + if (typeof value !== "string" || value.length > 65_536) throw new TypeError("malformed integration ciphertext"); + const parts = value.split("."); + if (parts.length !== 2 || !parts.every(part => /^[A-Za-z0-9+/]+={0,2}$/.test(part))) throw new TypeError("malformed integration ciphertext"); + const nonce = Buffer.from(parts[0], "base64"); + const ciphertext = Buffer.from(parts[1], "base64"); + if (nonce.toString("base64") !== parts[0] || ciphertext.toString("base64") !== parts[1] + || nonce.byteLength !== NONCE_BYTES || ciphertext.byteLength < 16) throw new TypeError("malformed integration ciphertext"); + return { nonce: new Uint8Array(nonce), ciphertext: new Uint8Array(ciphertext) }; +} + +function recordAad(ownerId, record) { + if (typeof ownerId !== "string" || !ownerId || typeof record?.id !== "string" || !record.id + || typeof record.origin !== "string" || !record.origin || !Number.isSafeInteger(record.revision) || record.revision < 1) { + throw new TypeError("malformed integration record binding"); + } + return encoder.encode(JSON.stringify(["factorize-record-v1", ownerId, record.id, "bearer", record.origin, record.revision])); +} + +async function deriveKey(key) { + if (typeof key !== "string" || !key.startsWith(KEY_PREFIX)) throw new TypeError("malformed unlock key"); + const raw = decodeBase64Url(key.slice(KEY_PREFIX.length), "unlock key"); + if (raw.byteLength !== KEY_BYTES) throw new TypeError("malformed unlock key"); + try { + const material = await webcrypto.subtle.importKey("raw", raw, "HKDF", false, ["deriveKey"]); + return await webcrypto.subtle.deriveKey({ name: "HKDF", hash: "SHA-256", salt: SALT, info: INFO }, material, + { name: "AES-GCM", length: 256 }, false, ["decrypt"]); + } finally { + raw.fill(0); + } +} + +function validateOrigin(origin) { + let parsed; + try { parsed = new URL(origin); } catch { throw new TypeError("invalid integration origin"); } + if (parsed.protocol !== "https:" || parsed.origin !== origin || parsed.username || parsed.password) throw new TypeError("invalid integration origin"); + return parsed.origin; +} + +/** + * Resolve only the selected bearer records from a current, server-decrypted + * snapshot. The API supplies ciphertext after removing its own encryption + * layer; this helper never sends the user's unlock key or plaintext upstream. + */ +export function createDecryptedGrantResolver({ readSnapshot, getUnlockKey, resolveIntegrationIds } = {}) { + if (typeof readSnapshot !== "function" || typeof getUnlockKey !== "function" + || (resolveIntegrationIds !== undefined && typeof resolveIntegrationIds !== "function")) throw new TypeError("grant resolver requires snapshot and unlock-key readers"); + return async function resolveGrants(binding) { + if (!binding || typeof binding.tenantId !== "string" || !binding.tenantId + || (binding.integrationIds !== undefined && !Array.isArray(binding.integrationIds))) throw new TypeError("invalid run integration selection"); + const requestedIds = binding.integrationIds ?? await resolveIntegrationIds?.(binding); + if (requestedIds !== undefined && (!Array.isArray(requestedIds) || requestedIds.some(id => typeof id !== "string" || !id) + || new Set(requestedIds).size !== requestedIds.length)) throw new TypeError("invalid run integration selection"); + if (requestedIds?.length === 0) return []; + const unlockKey = await getUnlockKey(); + if (typeof unlockKey !== "string" || !unlockKey) throw new Error("proxy is locked"); + const snapshot = await readSnapshot(binding); + if (!snapshot || !Array.isArray(snapshot.records) || typeof snapshot.ownerId !== "string" || !snapshot.ownerId) throw new Error("integration snapshot is unavailable"); + // Production snapshots are already filtered by the control plane using the + // active run's immutable selection and grant revision. Local callers may + // still provide IDs explicitly for additional narrowing. + const integrationIds = requestedIds ?? snapshot.records.map(record => record?.id); + if (!Array.isArray(integrationIds) || integrationIds.some(id => typeof id !== "string" || !id) + || new Set(integrationIds).size !== integrationIds.length) throw new TypeError("invalid run integration selection"); + if (integrationIds.length === 0) { + if (snapshot.records.length) throw new Error("integration snapshot does not match the run selection"); + return []; + } + const selected = new Set(integrationIds); + const selectedRecords = snapshot.records.filter(record => selected.has(record?.id) && record.kind === "bearer"); + const selectionWasNarrowedLocally = binding.integrationIds !== undefined || resolveIntegrationIds !== undefined; + if (selectedRecords.length !== selected.size || (!selectionWasNarrowedLocally && snapshot.records.length !== selectedRecords.length)) throw new Error("selected integration is unavailable"); + const aesKey = await deriveKey(unlockKey); + const grants = []; + for (const record of selectedRecords) { + if (record.deletedAt != null) { + if (record.encryptedPayload !== null) throw new Error("deleted integration snapshot contains ciphertext"); + continue; + } + if (typeof record.encryptedPayload !== "string") throw new Error("selected integration is unavailable"); + const origin = validateOrigin(record.origin); + const { nonce, ciphertext } = decodeStoredCiphertext(record.encryptedPayload); + const plaintext = new Uint8Array(await webcrypto.subtle.decrypt({ name: "AES-GCM", iv: nonce, + additionalData: recordAad(snapshot.ownerId, record), tagLength: 128 }, aesKey, ciphertext)); + try { + const credential = decoder.decode(plaintext); + if (!credential || credential.length > 16_384 || /[\u0000-\u001f\u007f]/.test(credential)) throw new TypeError("invalid bearer credential"); + grants.push(Object.freeze({ + id: record.id, + name: typeof record.name === "string" && record.name ? record.name : record.id, + description: record.description ?? null, + origin, + credential, + })); + } finally { + plaintext.fill(0); + nonce.fill(0); + ciphertext.fill(0); + } + } + return grants; + }; +} + +/** Compose the boot-registered control client, current snapshot reader, and + * process-local unlock key into the resolver used by authenticated tunnels. */ +export function createProxyIntegrationGrantResolver({ runtime, controlOrigin, controlCredential, fetchImpl = fetch } = {}) { + if (!runtime || typeof runtime.getUnlockKey !== "function") throw new TypeError("proxy runtime does not expose its unlocked key to the grant resolver"); + if (typeof controlOrigin !== "string" || typeof controlCredential !== "string") throw new TypeError("proxy control registration is required"); + return createDecryptedGrantResolver({ + readSnapshot: binding => readProxyIntegrationSnapshot({ controlOrigin, token: controlCredential, binding, fetchImpl }), + getUnlockKey: () => runtime.getUnlockKey(), + }); +} diff --git a/packages/credential-proxy/src/destination-policy.mjs b/packages/credential-proxy/src/destination-policy.mjs index 0b71dbbb..e0bbba9e 100644 --- a/packages/credential-proxy/src/destination-policy.mjs +++ b/packages/credential-proxy/src/destination-policy.mjs @@ -109,7 +109,7 @@ async function resolveAll(hostname) { return lookup(hostname, { all: true, verbatim: true }); } -function openTlsSocket({ hostname, port, address }) { +function openTlsSocket({ hostname, port, address, timeoutMs = 10_000 }) { return new Promise((resolve, reject) => { const socket = tlsConnect({ host: address, @@ -117,9 +117,12 @@ function openTlsSocket({ hostname, port, address }) { servername: isIP(hostname) ? undefined : hostname, checkServerIdentity: (_server, certificate) => checkServerIdentity(hostname, certificate), }); - const failed = (error) => { socket.destroy(); reject(error); }; + const timer = setTimeout(() => failed(new Error("upstream connect deadline exceeded")), timeoutMs); + timer.unref?.(); + const failed = (error) => { clearTimeout(timer); socket.destroy(); reject(error); }; socket.once("error", failed); socket.once("secureConnect", () => { + clearTimeout(timer); socket.removeListener("error", failed); resolve(socket); }); @@ -132,7 +135,7 @@ function openTlsSocket({ hostname, port, address }) { * so a second DNS lookup cannot change the destination between validation and * connection. Paths, queries, methods, and bodies remain available to callers. */ -export function createOriginConnector(configuredOrigin, { resolve = resolveAll, connect = openTlsSocket } = {}) { +export function createOriginConnector(configuredOrigin, { resolve = resolveAll, connect = openTlsSocket, timeoutMs = 10_000 } = {}) { const configured = parseHttpsUrl(configuredOrigin, "configured origin"); if (configured.url.pathname !== "/" || configured.url.search || configured.url.hash) { throw new TypeError("configured origin must contain only an HTTPS origin"); @@ -148,7 +151,7 @@ export function createOriginConnector(configuredOrigin, { resolve = resolveAll, if (!answer || !isPublicAddress(answer.address)) throw new Error("origin resolved to a non-public address"); } const address = answers[0].address; - const socket = await connect({ hostname: target.hostname, port: target.port, address, servername: target.hostname }); + const socket = await connect({ hostname: target.hostname, port: target.port, address, servername: target.hostname, timeoutMs }); const rawPeerAddress = socket?.remoteAddress ?? ""; const peerAddress = normalizeAddress(rawPeerAddress); if (!peerAddress || peerAddress !== normalizeAddress(address) || !isPublicAddress(rawPeerAddress)) { diff --git a/packages/credential-proxy/src/main.mjs b/packages/credential-proxy/src/main.mjs index 20c32377..714823cb 100644 --- a/packages/credential-proxy/src/main.mjs +++ b/packages/credential-proxy/src/main.mjs @@ -1,7 +1,103 @@ import { createInterface } from "node:readline"; +import { randomUUID } from "node:crypto"; import { createProxyRuntime } from "./runtime.mjs"; +import { registerProxyBoot, startProxyRunRevocationMonitor, startProxyRuntimeStatusReporter } from "./control-client.mjs"; +import { startProxyBootstrapControl } from "./bootstrap-control.mjs"; +import { createProxyIntegrationGrantResolver } from "./decrypted-grants.mjs"; +import { startAuthenticatedProxyTunnel } from "./tunnel-transport.mjs"; +import { createProxyTlsAuthority } from "./tls-identity.mjs"; +import { createCodexAuthHelper } from "./codex-auth-helper.mjs"; +import { createModelAccountLifecycle } from "./model-account-lifecycle.mjs"; +import { createModelAccountPersistenceAdapter } from "./control-client.mjs"; +import { decryptModelAccountCache, encryptModelAccountCache } from "./model-account-crypto.mjs"; +import { startBrowserControl } from "./browser-control.mjs"; +import { createProxyRunCredentialStore } from "./run-credential-store.mjs"; -const runtime = createProxyRuntime(); +const epochIndex = process.argv.indexOf("--boot-epoch"); +const bootEpoch = epochIndex < 0 ? randomUUID() : process.argv[epochIndex + 1]; +const runtime = createProxyRuntime({ bootEpoch }); +const runCredentials = createProxyRunCredentialStore(); +const tunnelPort = Number(process.env.FACTORIZE_PROXY_TUNNEL_PORT ?? "9090"); +if (!Number.isInteger(tunnelPort) || tunnelPort < 1 || tunnelPort > 65535) throw new Error("invalid proxy tunnel port"); +const tlsAuthority = await createProxyTlsAuthority({ + serverName: process.env.FACTORIZE_PROXY_TUNNEL_SERVER_NAME ?? "localhost", + bootEpoch: runtime.bootEpoch, +}); +let controlCredential; +let controlOrigin; +let tunnel; +let revocationMonitor; +let statusReporter; +let modelAccounts; +let modelAccountLifecycle; +let browserControl; + +async function activateRegistration(registration, origin, browserOrigin) { + if (tunnel) throw new Error("proxy boot is already registered"); + const persistence = createModelAccountPersistenceAdapter({ + controlOrigin: origin, + controlCredential: registration.controlCredential, + ownerId: registration.tenantId, + }); + modelAccounts = persistence; + modelAccountLifecycle = createModelAccountLifecycle({ + authHelper: createCodexAuthHelper(), + getUnlockKey: () => runtime.getUnlockKey(), + getAccount: persistence.getAccount, + getEncryptedCache: persistence.getEncryptedCache, + persistEncrypted: persistence.persistEncrypted, + encryptCache: encryptModelAccountCache, + decryptCache: decryptModelAccountCache, + }); + if (!process.env.FACTORIZE_PROXY_BROWSER_CAPABILITY_PUBLIC_KEY) throw new Error("proxy browser capability verification key is unavailable"); + browserControl = await startBrowserControl({ + runtime, + lifecycle: modelAccountLifecycle, + publicKey: process.env.FACTORIZE_PROXY_BROWSER_CAPABILITY_PUBLIC_KEY, + proxyId: registration.proxyIdentity, + ownerId: registration.tenantId, + generation: registration.bootEpoch, + appOrigin: process.env.FACTORIZE_APP_ORIGIN ?? "https://app.factorize.sh", + }); + const resolveGrants = createProxyIntegrationGrantResolver({ runtime, controlOrigin: origin, controlCredential: registration.controlCredential }); + const started = await startAuthenticatedProxyTunnel({ + tls: tlsAuthority.tls, + runtime, + controlOrigin: origin, + controlCredential: registration.controlCredential, + resolveGrants, + resolveRunCredential: (runId, vmIdentity) => runCredentials.resolve(runId, vmIdentity), + host: process.env.FACTORIZE_PROXY_TUNNEL_HOST ?? "0.0.0.0", + port: tunnelPort, + }); + controlCredential = registration.controlCredential; + controlOrigin = origin; + tunnel = started; + revocationMonitor = startProxyRunRevocationMonitor({ + controlOrigin: origin, + controlCredential: registration.controlCredential, + bootEpoch: runtime.bootEpoch, + runtime, + onRevoked: runId => runCredentials.revoke(runId), + }); + statusReporter = startProxyRuntimeStatusReporter({ + controlOrigin: origin, + token: registration.controlCredential, + bootEpoch: runtime.bootEpoch, + runtime, + }); +} + +const bootstrapControl = await startProxyBootstrapControl({ + runtime, + authorizeRun: input => runCredentials.authorize(input), + socketPath: process.env.FACTORIZE_PROXY_BOOTSTRAP_SOCKET, + onRegistered: (registration, request) => activateRegistration(registration, request.controlOrigin, request.browserOrigin), + issueVmCertificate: ({ vmIdentity, csr }) => { + if (!tunnel) throw new Error("proxy tunnel is not registered"); + return tlsAuthority.signVmCsr({ vmIdentity, csr }); + }, +}); const lines = createInterface({ input: process.stdin, crlfDelay: Infinity }); let closing = false; let shutdownDeadline; @@ -10,10 +106,27 @@ function respond(value) { process.stdout.write(`${JSON.stringify(value)}\n`); } -function stop() { +function respondStatus() { + respond({ type: "status", state: runtime.state, proxyControlRegistered: Boolean(controlCredential && controlOrigin) }); +} + +async function stop() { if (closing) return; closing = true; - runtime.shutdown(); + controlCredential = undefined; + controlOrigin = undefined; + runCredentials.clear(); + await revocationMonitor?.close(); + revocationMonitor = undefined; + await statusReporter?.close(); + statusReporter = undefined; + await browserControl?.close().catch(() => {}); + browserControl = undefined; + await runtime.shutdown(); + await tunnel?.close().catch(() => {}); + tunnel = undefined; + await tlsAuthority.close(); + await bootstrapControl.close(); lines.close(); clearTimeout(shutdownDeadline); process.exit(0); @@ -23,20 +136,35 @@ process.once("SIGTERM", stop); process.once("SIGINT", stop); shutdownDeadline = setTimeout(() => process.exit(1), 2_000); shutdownDeadline.unref(); -respond({ type: "ready", state: runtime.state }); +respond({ type: "ready", state: runtime.state, bootEpoch: runtime.bootEpoch }); -lines.on("line", (line) => { +lines.on("line", async (line) => { if (closing) return; let command; try { command = JSON.parse(line); - if (command?.type === "status") respond({ type: "status", state: runtime.state }); + if (command?.type === "status") respondStatus(); else if (command?.type === "unlock") { - runtime.unlock(command.key); - respond({ type: "status", state: runtime.state }); + await runtime.unlockAndPrepare(command.key, async () => { + if (modelAccountLifecycle && modelAccounts) { + const accounts = await modelAccounts.listAccounts(); + await modelAccountLifecycle.reconcileAll(accounts); + } + }); + respondStatus(); } else if (command?.type === "lock") { - runtime.lock(); - respond({ type: "status", state: runtime.state }); + await runtime.lock(); + respondStatus(); + } else if (command?.type === "register-boot") { + let credential = command.credential; + command.credential = undefined; + try { + const registration = await registerProxyBoot({ controlOrigin: command.controlOrigin, browserOrigin: command.browserOrigin, token: credential, bootEpoch: runtime.bootEpoch }); + await activateRegistration(registration, command.controlOrigin, command.browserOrigin); + respond({ type: "boot-registered", proxyIdentity: registration.proxyIdentity, bootEpoch: registration.bootEpoch }); + } catch { + respond({ type: "error", error: "proxy boot registration failed" }); + } finally { credential = ""; } } else if (command?.type === "stop") stop(); else respond({ type: "error", error: "unsupported command" }); } catch { diff --git a/packages/credential-proxy/src/model-account-crypto.mjs b/packages/credential-proxy/src/model-account-crypto.mjs new file mode 100644 index 00000000..4602c480 --- /dev/null +++ b/packages/credential-proxy/src/model-account-crypto.mjs @@ -0,0 +1,61 @@ +import { webcrypto } from "node:crypto"; + +const KEY_PREFIX = "fzk1."; +const KEY_BYTES = 32; +const NONCE_BYTES = 12; +const SALT = new TextEncoder().encode("factorize.unlock-key.v1"); +const INFO = new TextEncoder().encode("factorize.record.aes-256-gcm.v1"); +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +function bindingData(binding) { + if (!binding || typeof binding.ownerId !== "string" || !binding.ownerId + || !UUID.test(binding.accountId ?? "") || !UUID.test(binding.keyGeneration ?? "") + || !Number.isSafeInteger(binding.revision) || binding.revision < 1) { + throw new TypeError("invalid model account encryption binding"); + } + return new TextEncoder().encode(JSON.stringify([ + "factorize-record-v1", binding.ownerId, binding.accountId, "model", "factorize", binding.keyGeneration, binding.revision, + ])); +} + +async function encryptionKey(unlockKey, usage) { + if (typeof unlockKey !== "string" || !unlockKey.startsWith(KEY_PREFIX)) throw new TypeError("malformed unlock key"); + const encoded = unlockKey.slice(KEY_PREFIX.length); + if (!/^[A-Za-z0-9_-]{43}$/.test(encoded)) throw new TypeError("malformed unlock key"); + const raw = Buffer.from(encoded, "base64url"); + if (raw.byteLength !== KEY_BYTES || raw.toString("base64url") !== encoded) throw new TypeError("malformed unlock key"); + try { + const material = await webcrypto.subtle.importKey("raw", raw, "HKDF", false, ["deriveKey"]); + return await webcrypto.subtle.deriveKey({ name: "HKDF", hash: "SHA-256", salt: SALT, info: INFO }, material, + { name: "AES-GCM", length: 256 }, false, [usage]); + } finally { raw.fill(0); } +} + +function decodeStored(value, label) { + if (typeof value !== "string" || !/^[A-Za-z0-9+/]+={0,2}$/.test(value)) throw new TypeError(`malformed ${label}`); + const bytes = Buffer.from(value, "base64"); + if (bytes.toString("base64") !== value) throw new TypeError(`malformed ${label}`); + return new Uint8Array(bytes); +} + +/** Encrypt a Codex auth cache for persistence. The cache and unlock key remain in the proxy process. */ +export async function encryptModelAccountCache(plaintext, unlockKey, binding) { + if (!(plaintext instanceof Uint8Array) || plaintext.byteLength === 0 || plaintext.byteLength > 65_536) throw new TypeError("invalid model account cache"); + const aad = bindingData(binding), nonce = webcrypto.getRandomValues(new Uint8Array(NONCE_BYTES)); + const key = await encryptionKey(unlockKey, "encrypt"); + const ciphertext = new Uint8Array(await webcrypto.subtle.encrypt({ name: "AES-GCM", iv: nonce, additionalData: aad, tagLength: 128 }, key, plaintext)); + return `${Buffer.from(nonce).toString("base64")}.${Buffer.from(ciphertext).toString("base64")}`; +} + +/** Decrypt only after the selected account, owner, key generation, and revision are verified. */ +export async function decryptModelAccountCache(encryptedPayload, unlockKey, binding) { + if (typeof encryptedPayload !== "string" || encryptedPayload.length > 65_536) throw new TypeError("malformed model account ciphertext"); + const parts = encryptedPayload.split("."); + if (parts.length !== 2) throw new TypeError("malformed model account ciphertext"); + const nonce = decodeStored(parts[0], "model account nonce"), ciphertext = decodeStored(parts[1], "model account ciphertext"); + if (nonce.byteLength !== NONCE_BYTES || ciphertext.byteLength < 16) throw new TypeError("malformed model account ciphertext"); + const aad = bindingData(binding), key = await encryptionKey(unlockKey, "decrypt"); + try { + return new Uint8Array(await webcrypto.subtle.decrypt({ name: "AES-GCM", iv: nonce, additionalData: aad, tagLength: 128 }, key, ciphertext)); + } finally { nonce.fill(0); ciphertext.fill(0); } +} diff --git a/packages/credential-proxy/src/model-account-lifecycle.mjs b/packages/credential-proxy/src/model-account-lifecycle.mjs new file mode 100644 index 00000000..565abde8 --- /dev/null +++ b/packages/credential-proxy/src/model-account-lifecycle.mjs @@ -0,0 +1,282 @@ +import { mkdir, readFile, readdir, rename, rm, writeFile } from "node:fs/promises"; +import { randomUUID } from "node:crypto"; +import { join, resolve, sep } from "node:path"; + +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +function cachePath(root, accountId) { + if (!UUID.test(accountId ?? "")) throw new TypeError("invalid model account id"); + const base = resolve(root), path = resolve(base, accountId, "auth.json"); + if (!path.startsWith(`${base}${sep}`)) throw new TypeError("invalid model account id"); + return path; +} + +/** + * Keep Codex token refresh and encrypted persistence inside the proxy. The + * persistence callback receives only user-encrypted auth-cache bytes; token + * JSON and the unlock key never leave this process in plaintext. + */ +export function createModelAccountLifecycle({ + authHelper, + accountHomeRoot = "/var/lib/factorize/codex-accounts", + getUnlockKey, + getAccount, + encryptCache, + persistEncrypted, + onUnsaved = () => {}, + retryDelaysMs = [100, 500, 2_000], + staleRevisionRetries = 3, + sleep = ms => new Promise(resolveSleep => setTimeout(resolveSleep, ms)), + removeCache = rm, + readCache = readFile, + writeCache = writeFile, + makeDirectory = mkdir, + readDirectory = readdir, + replaceFile = rename, + getEncryptedCache, + decryptCache, +} = {}) { + if (!authHelper || typeof authHelper.invoke !== "function" || typeof getUnlockKey !== "function" + || typeof getAccount !== "function" || typeof encryptCache !== "function" || typeof persistEncrypted !== "function" + || !Array.isArray(retryDelaysMs) || retryDelaysMs.length > 5 + || retryDelaysMs.some(ms => !Number.isSafeInteger(ms) || ms < 0 || ms > 30_000) + || !Number.isSafeInteger(staleRevisionRetries) || staleRevisionRetries < 0 || staleRevisionRetries > 5) { + throw new TypeError("model account lifecycle configuration is invalid"); + } + if ((getEncryptedCache === undefined) !== (decryptCache === undefined) + || (getEncryptedCache !== undefined && typeof getEncryptedCache !== "function") + || (decryptCache !== undefined && typeof decryptCache !== "function")) { + throw new TypeError("model account cache restoration configuration is invalid"); + } + + const generations = new Map(); + const queues = new Map(); + const unsaved = new Set(); + const currentGeneration = id => generations.get(id) ?? 0; + + function serialize(id, operation) { + const previous = queues.get(id) ?? Promise.resolve(); + const next = previous.catch(() => {}).then(operation); + queues.set(id, next); + return next.finally(() => { if (queues.get(id) === next) queues.delete(id); }); + } + + async function persist(id, generation) { + let account = await getAccount(id); + if (!account || account.id !== id || account.kind !== "model" || !Number.isSafeInteger(account.revision) + || typeof account.ownerId !== "string" || !account.ownerId || typeof account.keyGeneration !== "string" || !account.keyGeneration) { + throw new Error("model account persistence binding is unavailable"); + } + const unlockKey = await getUnlockKey(); + if (typeof unlockKey !== "string" || !unlockKey) throw new Error("proxy is locked"); + const plain = await readCache(cachePath(accountHomeRoot, id)); + if (!Buffer.isBuffer(plain) || plain.byteLength === 0 || plain.byteLength > 65_536) throw new Error("Codex token cache is missing or exceeds its limit"); + let staleRetries = 0; + let attempts = 0; + try { + for (;;) { + let encrypted; + try { + encrypted = await encryptCache(new Uint8Array(plain), unlockKey, { + ownerId: account.ownerId, + recordId: id, + recordType: "model", + destination: "factorize", + revision: account.revision + 1, + keyGeneration: account.keyGeneration, + }); + } catch (error) { + throw error; + } + if (currentGeneration(id) !== generation) throw new Error("model account changed before token persistence"); + const request = { id, expectedRevision: account.revision, keyGeneration: account.keyGeneration, encryptedPayload: encrypted, idempotencyKey: randomUUID() }; + let retryWithLatestRevision = false; + for (let attempt = 0; ; attempt++) { + if (currentGeneration(id) !== generation) throw new Error("model account changed before token persistence"); + attempts++; + try { + await persistEncrypted(request); + unsaved.delete(id); + onUnsaved({ accountId: id, unsaved: false }); + return { state: "saved" }; + } catch (error) { + if (error?.code === "stale_revision") { + if (staleRetries >= staleRevisionRetries) break; + let latest; + try { latest = await getAccount(id); } + catch { throw new Error("model account changed before token persistence"); } + if (!latest || latest.id !== id || latest.kind !== "model" || !Number.isSafeInteger(latest.revision) + || latest.revision <= account.revision || latest.ownerId !== account.ownerId + || latest.keyGeneration !== account.keyGeneration) { + throw new Error("model account changed before token persistence"); + } + account = latest; + staleRetries++; + retryWithLatestRevision = true; + break; + } + if (attempt >= retryDelaysMs.length) { + unsaved.add(id); + onUnsaved({ accountId: id, unsaved: true }); + return { state: "unsaved", attempts }; + } + await sleep(retryDelaysMs[attempt]); + if (currentGeneration(id) !== generation) throw new Error("model account changed before token persistence"); + let latest; + try { latest = await getAccount(id); } + catch { throw new Error("model account changed before token persistence"); } + if (!latest || latest.id !== id || latest.kind !== "model" || latest.ownerId !== account.ownerId + || latest.keyGeneration !== account.keyGeneration) { + throw new Error("model account changed before token persistence"); + } + } + } + if (!retryWithLatestRevision) { + unsaved.add(id); + onUnsaved({ accountId: id, unsaved: true }); + return { state: "unsaved", attempts }; + } + } + } finally { + plain.fill(0); + } + } + + async function refresh(id) { + return serialize(id, async () => { + const generation = currentGeneration(id); + const result = await authHelper.invoke({ method: "refresh", accountId: id }); + if (result.state === "reauthentication-required") return result; + return persist(id, generation); + }); + } + + async function deviceLogin(id) { + return serialize(id, async () => { + const generation = currentGeneration(id); + const result = await authHelper.invoke({ method: "device-login", accountId: id }); + if (result.state !== "pending" || !result.completion || typeof result.completion.then !== "function") throw new Error("Codex device authorization did not start"); + const completion = Promise.resolve(result.completion).then(async outcome => { + if (currentGeneration(id) !== generation) return { state: "deleted" }; + if (outcome?.state !== "authorized") return { state: outcome?.state ?? "reauthentication-required" }; + return persist(id, generation); + }); + return { state: "pending", challenge: result.challenge, completion }; + }); + } + + async function cancel(id) { + return serialize(id, async () => authHelper.invoke({ method: "cancel", accountId: id })); + } + + async function deleteAccount(id) { + if (!UUID.test(id ?? "")) throw new TypeError("invalid model account id"); + // Invalidate in-flight save retries immediately; cleanup then waits behind + // any already running helper process before deleting its cache. + generations.set(id, currentGeneration(id) + 1); + return serialize(id, async () => { + await authHelper.invoke({ method: "cancel", accountId: id }).catch(() => {}); + await authHelper.invoke({ method: "logout", accountId: id }).catch(() => {}); + await removeCache(resolve(accountHomeRoot, id), { recursive: true, force: true }); + unsaved.delete(id); + onUnsaved({ accountId: id, unsaved: false }); + return { state: "deleted" }; + }); + } + + async function restore(id) { + if (!getEncryptedCache || !decryptCache) throw new Error("model account cache restoration is unavailable"); + const account = await getAccount(id); + if (!account || account.id !== id || account.kind !== "model" || !Number.isSafeInteger(account.revision) + || typeof account.ownerId !== "string" || !account.ownerId || typeof account.keyGeneration !== "string" || !account.keyGeneration) { + await removeCache(resolve(accountHomeRoot, id), { recursive: true, force: true }); + return { state: "reauthentication-required" }; + } + const encrypted = await getEncryptedCache(id); + if (!encrypted || encrypted.accountId !== id || encrypted.ownerId !== account.ownerId + || encrypted.keyGeneration !== account.keyGeneration || !Number.isSafeInteger(encrypted.revision) + || typeof encrypted.encryptedPayload !== "string") { + await removeCache(resolve(accountHomeRoot, id), { recursive: true, force: true }); + return { state: "reauthentication-required" }; + } + const unlockKey = await getUnlockKey(); + if (typeof unlockKey !== "string" || !unlockKey) throw new Error("proxy is locked"); + let plain; + try { + plain = await decryptCache(encrypted.encryptedPayload, unlockKey, { + ownerId: account.ownerId, + recordId: id, + recordType: "model", + destination: "factorize", + revision: encrypted.revision, + keyGeneration: account.keyGeneration, + }); + if (!(plain instanceof Uint8Array) || plain.byteLength === 0 || plain.byteLength > 65_536) { + throw new Error("restored Codex token cache is invalid"); + } + const destination = cachePath(accountHomeRoot, id); + await makeDirectory(resolve(accountHomeRoot, id), { recursive: true, mode: 0o700 }); + const temporary = `${destination}.${randomUUID()}.tmp`; + try { + await writeCache(temporary, plain, { mode: 0o600, flag: "wx" }); + await replaceFile(temporary, destination); + } finally { + await rm(temporary, { force: true }); + } + } finally { + plain?.fill(0); + } + return { state: "restored", revision: encrypted.revision }; + } + + async function reconcile(id) { + return serialize(id, async () => { + if (!await getAccount(id)) { + await removeCache(resolve(accountHomeRoot, id), { recursive: true, force: true }); + unsaved.delete(id); + return { state: "deleted" }; + } + if (getEncryptedCache) { + const restored = await restore(id); + if (restored.state !== "restored") return restored; + } + const result = await authHelper.invoke({ method: "account-status", accountId: id }); + return result.state === "authenticated" ? persist(id, currentGeneration(id)) : { state: "reauthentication-required" }; + }); + } + + return Object.freeze({ + deviceLogin, + cancel, + refresh, + deleteAccount, + restore, + reconcile, + isUnsaved: id => unsaved.has(id), + /** Reconcile a complete server snapshot after unlock. A reset removes every + * prior model account from that snapshot, so stale local Codex homes must + * be logged out and removed before any current account is admitted. */ + reconcileAll: async accounts => { + if (!Array.isArray(accounts) || accounts.some(account => !account || !UUID.test(account.id ?? "") || account.kind !== "model") + || new Set(accounts.map(account => account.id)).size !== accounts.length) { + throw new TypeError("model account snapshot is invalid"); + } + const activeIds = new Set(accounts.map(account => account.id)); + const entries = await readDirectory(accountHomeRoot, { withFileTypes: true }).catch(error => { + if (error?.code === "ENOENT") return []; + throw error; + }); + const removed = []; + for (const entry of entries) { + if (!UUID.test(entry.name) || activeIds.has(entry.name)) continue; + await deleteAccount(entry.name); + removed.push(entry.name); + } + const states = []; + for (const account of accounts) states.push({ accountId: account.id, ...(await reconcile(account.id)) }); + return { removed, accounts: states }; + }, + }); +} + +export { cachePath as modelAccountCachePath }; diff --git a/packages/credential-proxy/src/proxy-listener.mjs b/packages/credential-proxy/src/proxy-listener.mjs new file mode 100644 index 00000000..b1ce1861 --- /dev/null +++ b/packages/credential-proxy/src/proxy-listener.mjs @@ -0,0 +1,168 @@ +import { createServer } from "node:http"; +import { pipeline } from "node:stream/promises"; +import { createBearerForwarder } from "./bearer-forwarder.mjs"; + +const RESPONSE_HOP_BY_HOP = new Set([ + "connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", + "trailer", "transfer-encoding", "upgrade", +]); + +function grantTable(grants) { + if (!Array.isArray(grants)) throw new TypeError("authenticated connection grants must be an array"); + const table = new Map(); + for (const grant of grants) { + if (!grant || typeof grant.id !== "string" || !grant.id || table.has(grant.id) + || typeof grant.origin !== "string" || typeof grant.credential !== "string" || !grant.credential) { + throw new TypeError("authenticated connection contains an invalid integration grant"); + } + table.set(grant.id, Object.freeze({ + id: grant.id, + name: typeof grant.name === "string" && grant.name ? grant.name : grant.id, + description: typeof grant.description === "string" ? grant.description : null, + origin: grant.origin, + credential: grant.credential, + })); + } + return table; +} + +function requestTarget(origin, suffix, search) { + const base = new URL(origin); + if (base.protocol !== "https:" || base.pathname !== "/" || base.search || base.hash) { + throw new TypeError("integration grant origin must be an HTTPS origin"); + } + const path = suffix || "/"; + if (!path.startsWith("/") || path.startsWith("//") || path.includes("\\")) { + throw new TypeError("invalid integration request path"); + } + return `${base.origin}${path}${search}`; +} + +/** + * Start one loopback HTTP endpoint for a single already-authenticated tunnel + * connection. The connection's consumed grants are held in this closure and + * can never be selected or supplied by an HTTP caller. + */ +export async function createAuthenticatedProxyListener({ connection, forwarder = createBearerForwarder(), host = "127.0.0.1", port = 0 } = {}) { + if (!connection || connection.authenticated !== true || connection.enrollmentConsumed !== true + || !connection.connectionId || !connection.runId || !connection.vmId) { + throw new TypeError("proxy listener requires an authenticated consumed connection"); + } + let grants = grantTable(connection.grants); + let unlocked = true; + let lockedMetadata = []; + const controllers = new Set(); + let closed = false; + const server = createServer(async (request, response) => { + const controller = new AbortController(); + controllers.add(controller); + const cancel = () => { + if (!response.writableEnded) controller.abort(new Error("tunnel request cancelled")); + }; + request.once("aborted", cancel); + response.once("close", cancel); + try { + const parsed = new URL(request.url ?? "/", "http://proxy.loopback"); + if (parsed.pathname === "/integrations") { + if (request.method !== "GET") { + response.writeHead(405, { allow: "GET", "cache-control": "no-store" }).end(); + return; + } + const visibleGrants = unlocked ? [...grants.values()] : lockedMetadata; + const integrations = visibleGrants.map(grant => { + const path = `/integrations/${encodeURIComponent(grant.id)}/`; + return { + id: grant.id, + source: "portable", + name: grant.name, + description: grant.description, + endpoint: path, + example: `curl "$FACTORIZE_PROXY_URL${path}"`, + availability: unlocked ? "available" : "locked", + }; + }); + response.writeHead(200, { "content-type": "application/json; charset=utf-8", "cache-control": "no-store" }); + response.end(JSON.stringify({ state: unlocked ? "unlocked" : "locked", integrations })); + return; + } + if (!unlocked) { + response.writeHead(423, { "cache-control": "no-store" }).end(); + return; + } + const match = /^\/integrations\/([^/]+)(\/.*)?$/.exec(parsed.pathname); + if (!match) { + response.writeHead(404).end(); + return; + } + let integrationId; + try { integrationId = decodeURIComponent(match[1]); } catch { + response.writeHead(400).end(); + return; + } + if (!integrationId || integrationId.includes("/") || integrationId.includes("\\")) { + response.writeHead(400).end(); + return; + } + const integration = grants.get(integrationId); + if (!integration) { + response.writeHead(403).end(); + return; + } + const target = requestTarget(integration.origin, match[2] ?? "/", parsed.search); + const result = await forwarder.forward(integration, { + url: target, + method: request.method, + headers: request.headers, + body: request, + }, { signal: controller.signal }); + const headers = Object.create(null); + for (const [name, value] of Object.entries(result.headers ?? {})) { + if (value !== undefined && !RESPONSE_HOP_BY_HOP.has(name.toLowerCase())) headers[name] = value; + } + response.writeHead(result.status, headers); + await pipeline(result.body, response, { signal: controller.signal }); + } catch (error) { + if (!response.headersSent && !response.destroyed) { + response.writeHead(controller.signal.aborted ? 499 : 502, { "content-type": "text/plain; charset=utf-8" }); + response.end(controller.signal.aborted ? "request cancelled" : "upstream request failed"); + } else if (!response.destroyed) response.destroy(error); + } finally { + controllers.delete(controller); + request.removeListener("aborted", cancel); + response.removeListener("close", cancel); + } + }); + server.on("connect", (_request, socket) => { socket.end("HTTP/1.1 405 Method Not Allowed\r\nConnection: close\r\n\r\n"); }); + server.on("upgrade", (_request, socket) => socket.destroy()); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(port, host, () => { + server.removeListener("error", reject); + resolve(); + }); + }); + return Object.freeze({ + connectionId: connection.connectionId, + address: server.address(), + lock() { + if (closed || !unlocked) return; + unlocked = false; + lockedMetadata = [...grants.values()].map(({ id, name, description }) => ({ id, name, description })); + grants = new Map(); + for (const controller of controllers) controller.abort(new Error("proxy locked")); + }, + replaceGrants(nextGrants) { + if (closed) return; + grants = grantTable(nextGrants); + unlocked = true; + lockedMetadata = []; + }, + close: () => { + if (closed) return Promise.resolve(); + closed = true; + grants = new Map(); + for (const controller of controllers) controller.abort(new Error("authenticated tunnel disconnected")); + return new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + }, + }); +} diff --git a/packages/credential-proxy/src/run-credential-store.mjs b/packages/credential-proxy/src/run-credential-store.mjs new file mode 100644 index 00000000..fd9b7840 --- /dev/null +++ b/packages/credential-proxy/src/run-credential-store.mjs @@ -0,0 +1,32 @@ +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +const RUN_CREDENTIAL = /^fzt_run_[A-Za-z0-9_-]+_[A-Za-z0-9_-]{43}$/; +const VM_IDENTITY = /^[A-Za-z0-9:._-]{1,255}$/; + +/** Scoped run credentials are delivered over the provider's private exec path + * and retained only in this proxy process. They are never written to disk. */ +export function createProxyRunCredentialStore() { + const credentials = new Map(); + const revoked = new Set(); + return Object.freeze({ + authorize({ runId, vmIdentity, credential } = {}) { + if (!UUID.test(runId ?? "") || !VM_IDENTITY.test(vmIdentity ?? "") || !RUN_CREDENTIAL.test(credential ?? "")) { + throw new TypeError("invalid run credential authorization"); + } + if (revoked.has(runId)) throw new Error("run authorization has been revoked"); + credentials.set(runId, Object.freeze({ vmIdentity, credential })); + return true; + }, + resolve(runId, vmIdentity) { + const entry = credentials.get(runId); + return !revoked.has(runId) && entry?.vmIdentity === vmIdentity ? entry.credential : undefined; + }, + revoke(runId) { + credentials.delete(runId); + revoked.add(runId); + }, + clear() { + credentials.clear(); + revoked.clear(); + }, + }); +} diff --git a/packages/credential-proxy/src/runtime.mjs b/packages/credential-proxy/src/runtime.mjs index e85401c7..dfe18926 100644 --- a/packages/credential-proxy/src/runtime.mjs +++ b/packages/credential-proxy/src/runtime.mjs @@ -1,37 +1,171 @@ +import { createAuthenticatedProxyListener } from "./proxy-listener.mjs"; +import { randomUUID } from "node:crypto"; + /** - * Internal adapter contracts for the follow-up record, enrollment and forwarding - * slices. This runtime deliberately does not call or implement these adapters. - * - * @typedef {object} ProxyAdapters - * @property {(identity: object) => Promise} loadRecords - * @property {(request: object) => Promise} createEnrollment - * @property {(request: object) => Promise} forwardRequest + * Process-local runtime for decrypted records and authorized tunnel listeners. + * The transport adapter must pass a connection only after it has consumed and + * authenticated its enrollment against the current run and proxy boot epoch. */ -export function createProxyRuntime() { +export function createProxyRuntime({ listenerFactory = createAuthenticatedProxyListener, bootEpoch = randomUUID(), grantRefreshIntervalMs = 5_000 } = {}) { + if (typeof bootEpoch !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(bootEpoch)) { + throw new TypeError("proxy boot epoch must be a UUID v4"); + } let unlockMaterial; let state = "locked"; let shutdown = false; + let locking = false; + const listeners = new Set(); + const listenerRuns = new Map(); + const listenerConnections = new Map(); + const grantRefreshTimers = new Map(); + const revokedRuns = new Set(); + + async function closeListeners() { + const current = [...listeners]; + listeners.clear(); + listenerRuns.clear(); + listenerConnections.clear(); + for (const timer of grantRefreshTimers.values()) clearInterval(timer); + grantRefreshTimers.clear(); + await Promise.allSettled(current.map((listener) => listener.close())); + } + + async function refreshActiveGrants() { + const current = [...listeners]; + await Promise.allSettled(current.map(async listener => { + const connection = listenerConnections.get(listener); + if (!connection || typeof connection.refreshGrants !== "function" || typeof listener.replaceGrants !== "function") return; + const grants = await connection.refreshGrants(); + if (state === "unlocked" && !shutdown && !revokedRuns.has(connection.runId) && listeners.has(listener)) listener.replaceGrants(grants); + })); + } return Object.freeze({ + // A fresh process identity invalidates enrollments from every prior boot. + bootEpoch, get state() { return state; }, unlock(key) { if (shutdown) throw new Error("runtime is shutting down"); + if (locking) throw new Error("proxy is locking"); if (typeof key !== "string" || key.length === 0) throw new TypeError("unlock key must be a non-empty string"); unlockMaterial?.fill(0); unlockMaterial = Buffer.from(key, "utf8"); state = "unlocked"; + void refreshActiveGrants(); }, - lock() { + /** Keep access locked until the caller has reconciled the current encrypted + * snapshot. The preparation callback can read the transient key, but no + * agent connection can be admitted while it is running. */ + async unlockAndPrepare(key, prepare) { + if (shutdown) throw new Error("runtime is shutting down"); + if (locking) throw new Error("proxy is locking"); + if (typeof key !== "string" || key.length === 0) throw new TypeError("unlock key must be a non-empty string"); + if (typeof prepare !== "function") throw new TypeError("unlock preparation callback is required"); + unlockMaterial?.fill(0); + unlockMaterial = Buffer.from(key, "utf8"); + state = "preparing"; + try { + await prepare(); + if (shutdown || locking || state !== "preparing") throw new Error("proxy unlock was interrupted"); + state = "unlocked"; + await refreshActiveGrants(); + } catch (error) { + unlockMaterial?.fill(0); + unlockMaterial = undefined; + if (!shutdown && !locking) state = "locked"; + throw error; + } + }, + /** Return a transient copy only to the local grant resolver. */ + getUnlockKey() { + if (shutdown || locking || !["unlocked", "preparing"].includes(state) || !unlockMaterial) throw new Error("proxy is locked"); + return unlockMaterial.toString("utf8"); + }, + async lock() { + locking = true; + state = "locked"; unlockMaterial?.fill(0); unlockMaterial = undefined; - if (!shutdown) state = "locked"; + try { + for (const listener of [...listeners]) { + if (typeof listener.lock === "function") listener.lock(); + else { + await listener.close().catch(() => {}); + listeners.delete(listener); + listenerRuns.delete(listener); + listenerConnections.delete(listener); + clearInterval(grantRefreshTimers.get(listener)); + grantRefreshTimers.delete(listener); + } + } + } + finally { locking = false; } + }, + async acceptAuthenticatedConnection(connection) { + if (shutdown) throw new Error("runtime is shutting down"); + if (state !== "unlocked") throw new Error("proxy is locked"); + if (connection?.bootEpoch !== bootEpoch) throw new Error("enrollment belongs to a different proxy boot"); + if (typeof connection.runId !== "string" || !connection.runId) throw new TypeError("authenticated connection requires a run identity"); + if (revokedRuns.has(connection.runId)) throw new Error("run grants have been revoked"); + // The transport adapter calls this only after atomically consuming and + // authenticating enrollment. Grants are the consumed, current run state. + const listener = await listenerFactory({ connection, isUnlocked: () => state === "unlocked" }); + if (state !== "unlocked" || shutdown || revokedRuns.has(connection.runId)) { + await listener.close(); + throw new Error(revokedRuns.has(connection.runId) ? "run grants have been revoked" : "proxy locked while accepting the connection"); + } + listeners.add(listener); + // Keep the run identity beside the listener so a control-plane revocation + // can tear down every established connection without retaining grants. + listenerRuns.set(listener, connection.runId); + listenerConnections.set(listener, connection); + if (typeof connection.refreshGrants === "function" && typeof listener.replaceGrants === "function") { + if (!Number.isSafeInteger(grantRefreshIntervalMs) || grantRefreshIntervalMs < 100 || grantRefreshIntervalMs > 60_000) { + await listener.close(); + listeners.delete(listener); + listenerRuns.delete(listener); + listenerConnections.delete(listener); + throw new TypeError("grant refresh interval is out of bounds"); + } + let refreshing = false; + const refresh = async () => { + if (refreshing || shutdown || locking || state !== "unlocked" || revokedRuns.has(connection.runId)) return; + refreshing = true; + try { + const next = await connection.refreshGrants(); + if (state === "unlocked" && !shutdown && !revokedRuns.has(connection.runId)) listener.replaceGrants(next); + } catch { + // A control-plane outage does not expire established grants. Retry + // on the next interval; durable revocation has its own monitor. + } finally { refreshing = false; } + }; + void refresh(); + const timer = setInterval(refresh, grantRefreshIntervalMs); + timer.unref?.(); + grantRefreshTimers.set(listener, timer); + } + return listener; + }, + async disconnect(connectionId) { + const matching = [...listeners].filter((listener) => listener.connectionId === connectionId); + await Promise.allSettled(matching.map((listener) => listener.close())); + for (const listener of matching) { listeners.delete(listener); listenerRuns.delete(listener); listenerConnections.delete(listener); clearInterval(grantRefreshTimers.get(listener)); grantRefreshTimers.delete(listener); } + }, + async revokeRun(runId) { + if (typeof runId !== "string" || !runId) throw new TypeError("run identity is required for revocation"); + revokedRuns.add(runId); + const matching = [...listeners].filter((listener) => listenerRuns.get(listener) === runId); + await Promise.allSettled(matching.map((listener) => listener.close())); + for (const listener of matching) { listeners.delete(listener); listenerRuns.delete(listener); listenerConnections.delete(listener); clearInterval(grantRefreshTimers.get(listener)); grantRefreshTimers.delete(listener); } + return matching.length; }, - shutdown() { - this.lock(); + async shutdown() { shutdown = true; + await this.lock(); + await closeListeners(); state = "stopped"; }, }); diff --git a/packages/credential-proxy/src/startup.mjs b/packages/credential-proxy/src/startup.mjs new file mode 100644 index 00000000..1eb7d163 --- /dev/null +++ b/packages/credential-proxy/src/startup.mjs @@ -0,0 +1,46 @@ +import { connect } from "node:net"; +import { once } from "node:events"; + +const SOCKET_PATH = "/run/factorize-proxy/bootstrap.sock"; +const MAX_BYTES = 4096; + +async function readRequest() { + let bytes = 0; + const chunks = []; + for await (const chunk of process.stdin) { + bytes += chunk.byteLength; + if (bytes > MAX_BYTES) throw new Error("invalid bootstrap request"); + chunks.push(chunk); + } + const data = Buffer.concat(chunks); + if (!data.byteLength || data[data.byteLength - 1] !== 10) throw new Error("invalid bootstrap request"); + return data; +} + +async function main() { + const request = await readRequest(); + const socket = connect(SOCKET_PATH); + const result = new Promise((resolve, reject) => { + let data = Buffer.alloc(0); + socket.on("data", chunk => { + data = Buffer.concat([data, chunk]); + const end = data.indexOf(10); + if (end >= 0) { + try { resolve(JSON.parse(data.subarray(0, end).toString("utf8"))); } + catch { reject(new Error("invalid bootstrap response")); } + socket.end(); + } + }); + socket.once("error", () => reject(new Error("proxy startup control unavailable"))); + socket.setTimeout(7000, () => reject(new Error("proxy startup control timed out"))); + }); + socket.write(request); + const response = await result; + if (response.registered !== true) throw new Error("proxy boot registration failed"); + process.stdout.write("Credential proxy started\n"); +} + +main().catch(() => { + process.stderr.write("Credential proxy startup failed\n"); + process.exitCode = 1; +}); diff --git a/packages/credential-proxy/src/tls-identity.mjs b/packages/credential-proxy/src/tls-identity.mjs new file mode 100644 index 00000000..933fde50 --- /dev/null +++ b/packages/credential-proxy/src/tls-identity.mjs @@ -0,0 +1,87 @@ +import { execFile as execFileCallback } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { chmod, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; + +const execFile = promisify(execFileCallback); +const SAFE_VM_IDENTITY = /^[A-Za-z0-9:._-]{1,255}$/; +const SAFE_DNS_NAME = /^(?=.{1,253}$)(?:[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)(?:\.(?:[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?))*$/; + +async function openssl(args) { + try { + await execFile("openssl", args, { timeout: 10_000, maxBuffer: 64 * 1024, windowsHide: true }); + } catch { + // Never include command output or key material in errors. + throw new Error("proxy TLS identity operation failed"); + } +} + +function pem(bytes) { + return Buffer.from(bytes).toString("utf8"); +} + +/** + * Creates a per-process private CA and proxy server identity on a memory-backed + * filesystem. The CA private key never leaves this process or tmpfs. Job VMs + * generate their own private keys and submit CSRs; only their signed public + * certificate and the public trust root are returned for secure distribution. + */ +export async function createProxyTlsAuthority({ serverName, bootEpoch } = {}) { + if (typeof serverName !== "string" || !SAFE_DNS_NAME.test(serverName)) throw new TypeError("proxy TLS server name is invalid"); + if (typeof bootEpoch !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(bootEpoch)) throw new TypeError("proxy TLS boot epoch is invalid"); + const directory = await mkdtemp(join("/dev/shm", "factorize-proxy-tls-")); + const caKeyPath = join(directory, "ca-key.pem"); + const caCertPath = join(directory, "ca-cert.pem"); + const serverKeyPath = join(directory, "server-key.pem"); + const serverCertPath = join(directory, "server-cert.pem"); + try { + await openssl(["genpkey", "-algorithm", "EC", "-pkeyopt", "ec_paramgen_curve:P-256", "-out", caKeyPath]); + await chmod(caKeyPath, 0o600); + await openssl(["req", "-new", "-x509", "-key", caKeyPath, "-sha256", "-days", "1", "-subj", `/CN=factorize-proxy-${bootEpoch}`, "-addext", "basicConstraints=critical,CA:TRUE,pathlen:0", "-addext", "keyUsage=critical,keyCertSign,cRLSign", "-out", caCertPath]); + await openssl(["genpkey", "-algorithm", "EC", "-pkeyopt", "ec_paramgen_curve:P-256", "-out", serverKeyPath]); + await chmod(serverKeyPath, 0o600); + await openssl(["req", "-new", "-key", serverKeyPath, "-subj", `/CN=${serverName}`, "-out", join(directory, "server.csr")]); + await writeFile(join(directory, "server.ext"), `basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyAgreement\nextendedKeyUsage=serverAuth\nsubjectAltName=DNS:${serverName}\n`, { mode: 0o600 }); + await openssl(["x509", "-req", "-in", join(directory, "server.csr"), "-CA", caCertPath, "-CAkey", caKeyPath, "-CAcreateserial", "-days", "1", "-sha256", "-extfile", join(directory, "server.ext"), "-out", serverCertPath]); + const ca = await readFile(caCertPath); + const key = await readFile(serverKeyPath); + const cert = await readFile(serverCertPath); + + let closed = false; + return Object.freeze({ + tls: Object.freeze({ key, cert, ca }), + trustBundle: pem(ca), + async signVmCsr({ vmIdentity, csr } = {}) { + if (closed) throw new Error("proxy TLS identity authority is closed"); + if (typeof vmIdentity !== "string" || !SAFE_VM_IDENTITY.test(vmIdentity) + || typeof csr !== "string" || csr.length < 100 || csr.length > 16_384 + || !csr.startsWith("-----BEGIN CERTIFICATE REQUEST-----\n") + || !csr.trimEnd().endsWith("-----END CERTIFICATE REQUEST-----")) { + throw new TypeError("invalid VM certificate request"); + } + const identityUri = `urn:factorize:vm:${Buffer.from(vmIdentity, "utf8").toString("base64url")}`; + const csrPath = join(directory, `vm-${randomUUID()}.csr`); + const certPath = join(directory, `vm-${randomUUID()}.pem`); + const extensionPath = join(directory, `vm-${randomUUID()}.ext`); + await writeFile(csrPath, csr, { mode: 0o600 }); + await writeFile(extensionPath, `basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyAgreement\nextendedKeyUsage=clientAuth\nsubjectAltName=URI:${identityUri}\n`, { mode: 0o600 }); + try { + await openssl(["req", "-in", csrPath, "-verify", "-noout"]); + await openssl(["x509", "-req", "-in", csrPath, "-CA", caCertPath, "-CAkey", caKeyPath, "-CAcreateserial", "-days", "1", "-sha256", "-subj", `/CN=${vmIdentity}`, "-extfile", extensionPath, "-out", certPath]); + return Object.freeze({ certificate: pem(await readFile(certPath)), trustBundle: pem(ca) }); + } finally { + await Promise.all([rm(csrPath, { force: true }), rm(certPath, { force: true }), rm(extensionPath, { force: true })]); + } + }, + async close() { + if (closed) return; + closed = true; + await rm(directory, { recursive: true, force: true }); + }, + }); + } catch (error) { + await rm(directory, { recursive: true, force: true }); + throw error; + } +} diff --git a/packages/credential-proxy/src/tunnel-transport.mjs b/packages/credential-proxy/src/tunnel-transport.mjs new file mode 100644 index 00000000..367014c6 --- /dev/null +++ b/packages/credential-proxy/src/tunnel-transport.mjs @@ -0,0 +1,232 @@ +import { connect as connectTls, createServer as createTlsServer } from "node:tls"; +import { connect as connectTcp, createServer as createTcpServer } from "node:net"; +import { once } from "node:events"; +import { createProxyEnrollmentAuthenticator } from "./control-client.mjs"; + +const HANDSHAKE_LIMIT = 4096; +const HANDSHAKE_TIMEOUT_MS = 10_000; +const CONTROL_RETRY_WINDOW_MS = 5 * 60_000; +const CONTROL_RETRY_MAX_MS = 30_000; +const CONTROL_RETRY_SLOW_MS = 60_000; + +export function peerIdentity(socket) { + const certificate = socket.getPeerCertificate(); + if (!socket.authorized || !certificate?.subject?.CN || typeof certificate.subjectaltname !== "string") { + throw new Error("tunnel client certificate is not authorized"); + } + const identities = certificate.subjectaltname.split(/,\s*/).filter(value => value.startsWith("URI:")); + if (identities.length !== 1) throw new Error("tunnel client certificate must contain one VM identity URI"); + const match = /^URI:urn:factorize:vm:([A-Za-z0-9_-]+)$/.exec(identities[0]); + if (!match) throw new Error("tunnel client certificate has an invalid VM identity URI"); + const identityBytes = Buffer.from(match[1], "base64url"); + const identity = identityBytes.toString("utf8"); + if (!identity || identityBytes.toString("base64url") !== match[1] + || identity.length > 255 || /[\u0000-\u001f\u007f]/.test(identity) + || certificate.subject.CN !== identity) { + throw new Error("tunnel client certificate identity is invalid or inconsistent"); + } + return identity; +} + +/** + * Accept only a TLS client with a trusted VM certificate, consume its one-use + * enrollment, and bridge that stream to the connection-scoped HTTP listener. + * The bridge does not expose a shell or arbitrary destination forwarding. + */ +export function createAuthenticatedTunnelServer({ tls, runtime, authenticateEnrollment, authenticateRun, host = "127.0.0.1", port = 0 } = {}) { + if (!tls || !runtime || (typeof authenticateEnrollment !== "function" && typeof authenticateRun !== "function")) throw new TypeError("tunnel server requires TLS, runtime and enrollment authentication"); + const connections = new Set(); + const server = createTlsServer({ ...tls, requestCert: true, rejectUnauthorized: true }, (socket) => { + connections.add(socket); + let handshake = Buffer.alloc(0); + let settled = false; + const timeout = setTimeout(() => socket.destroy(new Error("tunnel enrollment timed out")), HANDSHAKE_TIMEOUT_MS); + timeout.unref(); + const reject = () => { + if (settled) return; + settled = true; + clearTimeout(timeout); + socket.destroy(); + }; + socket.on("data", async (chunk) => { + if (settled) return; + handshake = Buffer.concat([handshake, chunk]); + const end = handshake.indexOf("\n"); + if (end < 0) return handshake.length > HANDSHAKE_LIMIT ? reject() : undefined; + if (end > HANDSHAKE_LIMIT) return reject(); + settled = true; + clearTimeout(timeout); + socket.pause(); + const line = handshake.subarray(0, end).toString("utf8").replace(/\r$/, ""); + const rest = handshake.subarray(end + 1); + const match = /^FACTORIZE-TUNNEL\/1 ([A-Za-z0-9_-]{32,128})$/.exec(line); + const runMatch = /^FACTORIZE-TUNNEL\/3 ([0-9a-f-]{36})$/i.exec(line); + if ((!match || typeof authenticateEnrollment !== "function") && (!runMatch || typeof authenticateRun !== "function")) return reject(); + let listener; + let target; + try { + const vmIdentity = peerIdentity(socket); + const connection = runMatch + ? await authenticateRun({ runId: runMatch[1], vmIdentity, bootEpoch: runtime.bootEpoch }) + : await authenticateEnrollment({ token: match[1], vmIdentity, bootEpoch: runtime.bootEpoch }); + // The production control plane returns the consumed enrollment binding. + // Compare its VM and boot identities with the authenticated TLS peer and + // this process so an adapter cannot accidentally attach another run's + // grants to the tunnel. + if (!connection || connection.authenticated !== true || connection.enrollmentConsumed !== true + || connection.vmIdentity !== peerIdentity(socket) || connection.bootEpoch !== runtime.bootEpoch + || typeof connection.connectionId !== "string" || connection.connectionId.length === 0) { + throw new Error("enrollment binding does not match the authenticated tunnel"); + } + listener = await runtime.acceptAuthenticatedConnection(connection); + const address = listener.address; + target = connectTcp(address.port, address.address); + await once(target, "connect"); + if (rest.length) target.write(rest); + socket.write("FACTORIZE-TUNNEL/1 ACCEPTED\n"); + socket.pipe(target); + target.pipe(socket); + socket.resume(); + const close = () => { + socket.unpipe(target); + target.unpipe(socket); + target.destroy(); + void runtime.disconnect(connection.connectionId); + }; + socket.once("close", close); + target.once("close", close); + } catch { + listener && await runtime.disconnect(listener.connectionId); + socket.destroy(); + } + }); + socket.once("close", () => { clearTimeout(timeout); connections.delete(socket); }); + socket.once("error", () => {}); + }); + server.listen(port, host); + return { + server, + listening: once(server, "listening"), + address: () => server.address(), + close: async () => { + for (const socket of connections) socket.destroy(); + await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + }, + }; +} + +/** Start the proxy's production-shaped mTLS listener using its boot-registered + * control credential and process-local decrypted integration grants. */ +export async function startAuthenticatedProxyTunnel({ + tls, runtime, controlOrigin, controlCredential, resolveGrants, resolveRunCredential, fetchImpl = fetch, host = "127.0.0.1", port = 0, +} = {}) { + const authenticateRun = createProxyEnrollmentAuthenticator({ + controlOrigin, controlCredential, bootEpoch: runtime?.bootEpoch, resolveGrants, resolveRunCredential, fetchImpl, + }); + const tunnel = createAuthenticatedTunnelServer({ tls, runtime, authenticateRun, host, port }); + try { + await tunnel.listening; + return tunnel; + } catch (error) { + await tunnel.close().catch(() => {}); + throw error; + } +} + +/** Start a loopback-only HTTP tunnel helper. Each accepted stream gets fresh enrollment. */ +export async function startVmTunnelHelper({ tls, enrollment, runId, proxyHost, proxyPort, host = "127.0.0.1", port = 0, + now = Date.now, random = Math.random, wait = delay } = {}) { + const runBasedEnrollment = typeof runId === "string" && /^[0-9a-f-]{36}$/i.test(runId); + if (!tls || (!runBasedEnrollment && typeof enrollment !== "function") + || typeof proxyHost !== "string" || !Number.isInteger(proxyPort)) { + throw new TypeError("VM tunnel helper requires TLS, enrollment and proxy address"); + } + const sockets = new Set(); + const server = createTcpServer((local) => { + sockets.add(local); + local.pause(); + void (async () => { + const startedAt = now(); + let attempt = 0; + while (!local.destroyed) { + let remote; + try { + const grant = runBasedEnrollment ? undefined : await enrollment(); + if (local.destroyed) return; + if (!runBasedEnrollment && (!grant || typeof grant.token !== "string" || !/^[A-Za-z0-9_-]{32,128}$/.test(grant.token))) throw new Error("fresh enrollment unavailable"); + remote = connectTls({ ...tls, host: proxyHost, port: proxyPort, servername: tls.servername ?? proxyHost, rejectUnauthorized: true }); + sockets.add(remote); + const connectTimer = setTimeout(() => remote.destroy(new Error("proxy TLS connection timed out")), HANDSHAKE_TIMEOUT_MS); + try { await once(remote, "secureConnect"); } + finally { clearTimeout(connectTimer); } + remote.write(runBasedEnrollment ? `FACTORIZE-TUNNEL/3 ${runId}\n` : `FACTORIZE-TUNNEL/1 ${grant.token}\n`); + await waitForTunnelAcceptance(remote); + local.pipe(remote); + remote.pipe(local); + local.resume(); + remote.once("close", () => { sockets.delete(remote); local.destroy(); }); + local.once("close", () => remote.destroy()); + return; + } catch { + remote?.destroy(); + if (remote) sockets.delete(remote); + if (local.destroyed) return; + const elapsed = now() - startedAt; + const base = elapsed < CONTROL_RETRY_WINDOW_MS + ? Math.min(1000 * (2 ** Math.min(attempt, 5)), CONTROL_RETRY_MAX_MS) + : CONTROL_RETRY_SLOW_MS; + const jittered = Math.round(base * (0.5 + random())); + const delayMs = Math.max(1, elapsed < CONTROL_RETRY_WINDOW_MS ? Math.min(CONTROL_RETRY_MAX_MS, jittered) : jittered); + attempt++; + try { await wait(delayMs); } catch { local.destroy(); return; } + } + } + })(); + const remove = () => { sockets.delete(local); }; + local.once("close", remove); + local.once("error", remove); + }); + server.listen(port, host); + await once(server, "listening"); + const address = server.address(); + return Object.freeze({ + address, + close: async () => { + for (const socket of sockets) socket.destroy(); + await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + }, + }); +} + +function delay(milliseconds) { + return new Promise(resolve => setTimeout(resolve, milliseconds)); +} + +function waitForTunnelAcceptance(socket) { + return new Promise((resolve, reject) => { + let response = Buffer.alloc(0); + const cleanup = () => { + clearTimeout(timer); + socket.removeListener("data", onData); + socket.removeListener("error", onError); + socket.removeListener("close", onClose); + }; + const onError = error => { cleanup(); reject(error); }; + const onClose = () => { cleanup(); reject(new Error("proxy closed before accepting the tunnel")); }; + const onData = chunk => { + response = Buffer.concat([response, chunk]); + if (response.length > HANDSHAKE_LIMIT) { cleanup(); reject(new Error("invalid proxy tunnel response")); return; } + const end = response.indexOf(10); + if (end < 0) return; + const accepted = response.subarray(0, end).toString("utf8") === "FACTORIZE-TUNNEL/1 ACCEPTED"; + const trailingData = response.subarray(end + 1); + cleanup(); + if (!accepted || trailingData.length) reject(new Error("proxy tunnel was rejected")); + else resolve(); + }; + const timer = setTimeout(() => { cleanup(); socket.destroy(); reject(new Error("proxy tunnel acceptance timed out")); }, HANDSHAKE_TIMEOUT_MS); + socket.on("data", onData); + socket.once("error", onError); + socket.once("close", onClose); + }); +} diff --git a/packages/credential-proxy/src/vm-certificate.mjs b/packages/credential-proxy/src/vm-certificate.mjs new file mode 100644 index 00000000..a06a743a --- /dev/null +++ b/packages/credential-proxy/src/vm-certificate.mjs @@ -0,0 +1,61 @@ +import { connect } from "node:net"; + +const SOCKET_PATH = "/run/factorize-proxy/bootstrap.sock"; +const MAX_REQUEST_BYTES = 20 * 1024; +const MAX_RESPONSE_BYTES = 40 * 1024; + +async function readRequest() { + let length = 0; + const chunks = []; + for await (const chunk of process.stdin) { + length += chunk.byteLength; + if (length > MAX_REQUEST_BYTES) throw new Error("invalid certificate request"); + chunks.push(chunk); + } + if (!length || chunks.at(-1).at(-1) !== 10) throw new Error("invalid certificate request"); + const value = JSON.parse(Buffer.concat(chunks).toString("utf8")); + if (!value || typeof value !== "object" || Array.isArray(value) + || Object.keys(value).sort().join(",") !== "bootEpoch,csr,vmIdentity" + || typeof value.bootEpoch !== "string" || typeof value.csr !== "string" || typeof value.vmIdentity !== "string") { + throw new Error("invalid certificate request"); + } + return { type: "issue-vm-certificate", ...value }; +} + +async function send(request) { + const socket = connect(SOCKET_PATH); + try { + return await new Promise((resolve, reject) => { + let data = Buffer.alloc(0); + socket.on("data", chunk => { + data = Buffer.concat([data, chunk]); + if (data.length > MAX_RESPONSE_BYTES) { + reject(new Error("invalid certificate response")); + socket.destroy(); + return; + } + const end = data.indexOf(10); + if (end < 0) return; + try { resolve(JSON.parse(data.subarray(0, end).toString("utf8"))); } + catch { reject(new Error("invalid certificate response")); } + socket.end(); + }); + socket.once("error", () => reject(new Error("proxy certificate service unavailable"))); + socket.setTimeout(7000, () => reject(new Error("proxy certificate service timed out"))); + socket.once("connect", () => socket.write(`${JSON.stringify(request)}\n`)); + }); + } finally { + socket.destroy(); + } +} + +try { + const request = await readRequest(); + const result = await send(request); + if (result?.issued !== true || result.bootEpoch !== request.bootEpoch || result.vmIdentity !== request.vmIdentity + || typeof result.certificate !== "string" || typeof result.trustBundle !== "string") throw new Error("VM certificate was not issued"); + process.stdout.write(`${JSON.stringify({ bootEpoch: result.bootEpoch, vmIdentity: result.vmIdentity, certificate: result.certificate, trustBundle: result.trustBundle })}\n`); +} catch { + process.stderr.write("VM certificate provisioning failed\n"); + process.exitCode = 1; +} diff --git a/packages/credential-proxy/test/bearer-forwarder.test.mjs b/packages/credential-proxy/test/bearer-forwarder.test.mjs new file mode 100644 index 00000000..9284b49a --- /dev/null +++ b/packages/credential-proxy/test/bearer-forwarder.test.mjs @@ -0,0 +1,79 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { Readable } from "node:stream"; +import { createBearerForwarder } from "../src/bearer-forwarder.mjs"; + +const integration = { id: "integration-1", origin: "https://api.example.test", credential: "synthetic-token" }; + +test("forwards only through the granted origin and replaces unsafe routing and authorization headers", async () => { + const connections = []; + const calls = []; + const forwarder = createBearerForwarder({ + connectorFactory: origin => async url => { + connections.push({ origin, url }); + return { socket: { fixture: true }, url: new URL(url) }; + }, + request: async request => { calls.push(request); return { status: 201, headers: {}, body: Readable.from([Buffer.from("created")]) }; }, + }); + const result = await forwarder.forward(integration, { + method: "post", + url: "https://api.example.test/v1/items?source=job", + headers: { + Authorization: "Bearer attacker-token", + Host: "attacker.example", + Connection: "keep-alive, x-routing-hint", + "X-Routing-Hint": "attacker", + "X-Forwarded-Host": "attacker.example", + "Content-Type": "application/json", + }, + body: Buffer.from('{"ok":true}'), + }); + assert.equal(result.status, 201); + assert.deepEqual(Buffer.concat(await Array.fromAsync(result.body)), Buffer.from("created")); + assert.deepEqual(connections, [{ origin: integration.origin, url: "https://api.example.test/v1/items?source=job" }]); + assert.equal(calls[0].method, "POST"); + assert.equal(calls[0].headers.authorization, "Bearer synthetic-token"); + assert.equal(calls[0].headers.host, undefined); + assert.equal(calls[0].headers["x-routing-hint"], undefined); + assert.equal(calls[0].headers["x-forwarded-host"], undefined); + assert.equal(calls[0].headers["content-type"], "application/json"); +}); + +test("rejects requests outside the grant origin without sending them upstream", async () => { + let sent = false; + const forwarder = createBearerForwarder({ + connectorFactory: () => async () => { throw new TypeError("request URL does not match the configured HTTPS origin"); }, + request: async () => { sent = true; }, + }); + await assert.rejects(forwarder.forward(integration, { url: "https://169.254.169.254/latest/meta-data" }), /configured HTTPS origin/); + assert.equal(sent, false); + assert.equal(forwarder.activeRequests, 0); +}); + +test("applies an explicit per-proxy concurrent request bound", async () => { + let release; + const gate = new Promise(resolve => { release = resolve; }); + const forwarder = createBearerForwarder({ + connectorFactory: () => async url => ({ socket: {}, url: new URL(url) }), + request: async () => gate, + limits: { requestMs: 10_000 }, + }); + const started = []; + for (let i = 0; i < 16; i++) started.push(forwarder.forward(integration, { url: "https://api.example.test/" })); + await new Promise(resolve => setImmediate(resolve)); + await assert.rejects(forwarder.forward(integration, { url: "https://api.example.test/" }), /concurrent request limit/); + release({ status: 200 }); + await Promise.all(started); + assert.equal(forwarder.activeRequests, 0); +}); + +test("rejects oversized caller headers before upstream dispatch", async () => { + let sent = false; + const forwarder = createBearerForwarder({ + connectorFactory: () => async url => ({ socket: {}, url: new URL(url) }), + request: async () => { sent = true; }, + limits: { headerBytes: 20 }, + }); + await assert.rejects(forwarder.forward(integration, { url: "https://api.example.test/", headers: { "x-large": "x".repeat(50) } }), /headers exceed/); + assert.equal(sent, false); +}); diff --git a/packages/credential-proxy/test/bootstrap-control.test.mjs b/packages/credential-proxy/test/bootstrap-control.test.mjs new file mode 100644 index 00000000..f81d0e83 --- /dev/null +++ b/packages/credential-proxy/test/bootstrap-control.test.mjs @@ -0,0 +1,97 @@ +import { strict as assert } from "node:assert"; +import { afterEach, test } from "node:test"; +import { connect } from "node:net"; +import { mkdtemp, rm, stat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { once } from "node:events"; +import { startProxyBootstrapControl } from "../src/bootstrap-control.mjs"; + +const epoch = "11111111-1111-4111-8111-111111111111"; +const token = `fzc_${"a".repeat(43)}`; +let directory; +let service; +afterEach(async () => { + await service?.close(); service = undefined; + if (directory) await rm(directory, { recursive: true, force: true }); + directory = undefined; +}); + +async function fixture(register, onRegistered, issueVmCertificate, authorizeRun) { + directory = await mkdtemp(join(tmpdir(), "factorize-proxy-startup-")); + const socketPath = join(directory, "control.sock"); + const runtime = { bootEpoch: epoch }; + service = await startProxyBootstrapControl({ runtime, socketPath, register, onRegistered, issueVmCertificate, authorizeRun }); + return socketPath; +} + +async function request(socketPath, payload) { + const socket = connect(socketPath); + let response = ""; + socket.on("data", chunk => { response += chunk.toString(); }); + await once(socket, "connect"); + socket.end(`${JSON.stringify(payload)}\n`); + await once(socket, "close"); + return JSON.parse(response); +} + +test("local startup control redeems the one-use credential and leaves a private socket", async () => { + const calls = []; + const registered = []; + const socketPath = await fixture( + async value => { calls.push(value); return { proxyIdentity: "fly:app:machine", bootEpoch: epoch, controlCredential: token }; }, + (value, request) => registered.push({ value, request }), + ); + const result = await request(socketPath, { type: "register-boot", controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", bootEpoch: epoch, credential: token }); + assert.deepEqual(result, { registered: true, proxyIdentity: "fly:app:machine", bootEpoch: epoch }); + assert.deepEqual(calls, [{ controlOrigin: "https://factorize.example", token, bootEpoch: epoch, browserOrigin: "https://proxy.example" }]); + assert.deepEqual(registered, [{ + value: { proxyIdentity: "fly:app:machine", bootEpoch: epoch, controlCredential: token }, + request: { controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", bootEpoch: epoch }, + }]); + assert.equal(JSON.stringify(result).includes(token), false, "startup response never returns the proxy control credential"); + assert.equal((await stat(socketPath)).mode & 0o777, 0o600); +}); + +test("local startup control rejects run credentials, epoch substitution, and extra fields", async () => { + const calls = []; + const socketPath = await fixture(async value => { calls.push(value); return {}; }); + for (const payload of [ + { type: "register-boot", controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", bootEpoch: epoch, credential: `fzt_run_${"b".repeat(32)}` }, + { type: "register-boot", controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", bootEpoch: "22222222-2222-4222-8222-222222222222", credential: token }, + { type: "register-boot", controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", bootEpoch: epoch, credential: token, tenantId: "attacker" }, + ]) assert.deepEqual(await request(socketPath, payload), { error: "invalid bootstrap request" }); + assert.deepEqual(calls, []); +}); + +test("local provider channel returns only a boot-bound VM certificate and public trust bundle", async () => { + const csr = `-----BEGIN CERTIFICATE REQUEST-----\n${"A".repeat(120)}\n-----END CERTIFICATE REQUEST-----`; + let registered = false; + const socketPath = await fixture( + async () => ({ proxyIdentity: "fly:app:machine", bootEpoch: epoch, controlCredential: token }), + async () => { registered = true; }, + async request => { + assert.equal(registered, true); + return { certificate: `cert-for:${request.vmIdentity}`, trustBundle: "public-ca" }; + }, + ); + const payload = { type: "issue-vm-certificate", bootEpoch: epoch, vmIdentity: "fly:job-1", csr }; + assert.deepEqual(await request(socketPath, payload), { error: "certificate provisioning failed" }); + await request(socketPath, { type: "register-boot", controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", bootEpoch: epoch, credential: token }); + assert.deepEqual(await request(socketPath, payload), { + issued: true, bootEpoch: epoch, vmIdentity: "fly:job-1", certificate: "cert-for:fly:job-1", trustBundle: "public-ca", + }); + assert.equal(JSON.stringify(await request(socketPath, payload)).includes(token), false); +}); + +test("provider exec delivers a scoped run credential directly to proxy memory and returns no credential", async () => { + const received = []; + const socketPath = await fixture(async () => ({}), undefined, undefined, async value => received.push(value)); + const runId = "33333333-3333-4333-8333-333333333333"; + const credential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + const requestBody = { runId, vmIdentity: "fly:app:machine-1", credential }; + const result = await request(socketPath, { type: "authorize-run", ...requestBody }); + assert.deepEqual(result, { authorized: true, runId, vmIdentity: requestBody.vmIdentity }); + assert.deepEqual(received, [requestBody]); + assert.equal(JSON.stringify(result).includes(credential), false); +}); diff --git a/packages/credential-proxy/test/browser-control.test.mjs b/packages/credential-proxy/test/browser-control.test.mjs new file mode 100644 index 00000000..918c6995 --- /dev/null +++ b/packages/credential-proxy/test/browser-control.test.mjs @@ -0,0 +1,76 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { generateKeyPairSync, sign } from "node:crypto"; +import { startBrowserControl } from "../src/browser-control.mjs"; + +const ownerId = "11111111-1111-4111-8111-111111111111"; +const accountId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; +const proxyId = "fly:factorize-proxy:machine-1"; +const generation = 4; +const appOrigin = "https://factorize.example"; +const { publicKey, privateKey } = generateKeyPairSync("ed25519"); + +function capability({ action, exp = 1_900_000_030, aud = proxyId, sub = ownerId, origin = appOrigin, gen = generation, account = accountId, jti = `nonce-${crypto.randomUUID()}` }) { + const head = Buffer.from(JSON.stringify({ alg: "EdDSA", typ: "JWT" })).toString("base64url"); + const claims = Buffer.from(JSON.stringify({ iss: "factorize", aud, sub, action, origin, generation: gen, exp, accountId: account, jti })).toString("base64url"); + const input = `${head}.${claims}`; + return `${input}.${sign(null, Buffer.from(input), privateKey).toString("base64url")}`; +} + +test("private browser control requires exact origin and one-use owner/proxy/action capabilities", async t => { + let now = 1_900_000_000; + const calls = []; + const runtime = { state: "unlocked" }; + const lifecycle = { + async deviceLogin(id) { calls.push(["authorize", id]); return { state: "pending", challenge: { verificationUrl: "https://auth.openai.com/codex/device", userCode: "ABCD-1234" }, completion: Promise.resolve({ state: "saved" }) }; }, + async cancel(id) { calls.push(["cancel", id]); return { state: "canceled" }; }, + }; + const server = await startBrowserControl({ runtime, lifecycle, publicKey, proxyId, ownerId, generation, appOrigin, port: 0, host: "127.0.0.1", now: () => now }); + t.after(() => server.close()); + const origin = `http://127.0.0.1:${server.address.port}`; + const post = (action, token, path = `/model-accounts/${accountId}/${action.split(":")[1]}`, requestOrigin = appOrigin) => fetch(origin + path, { + method: "POST", headers: { Origin: requestOrigin, "Content-Type": "application/json", Authorization: `Bearer ${token}` }, body: "{}", + }); + + const preflight = await fetch(origin + `/model-accounts/${accountId}/authorize`, { + method: "OPTIONS", headers: { Origin: appOrigin, "Access-Control-Request-Method": "POST", "Access-Control-Request-Headers": "content-type,authorization" }, + }); + assert.equal(preflight.status, 204); + assert.equal(preflight.headers.get("access-control-allow-origin"), appOrigin); + const foreign = await fetch(origin + `/model-accounts/${accountId}/authorize`, { method: "OPTIONS", headers: { Origin: "https://attacker.example", "Access-Control-Request-Method": "POST", "Access-Control-Request-Headers": "authorization,content-type" } }); + assert.equal(foreign.status, 403); + assert.equal(foreign.headers.get("access-control-allow-origin"), null); + + const authToken = capability({ action: "model-account:authorize" }); + const started = await post("model-account:authorize", authToken); + assert.equal(started.status, 202); + assert.deepEqual(await started.json(), { state: "pending", verificationUrl: "https://auth.openai.com/codex/device", userCode: "ABCD-1234" }); + assert.deepEqual(calls[0], ["authorize", accountId]); + const replay = await post("model-account:authorize", authToken); + assert.equal(replay.status, 401); + + const status = await post("model-account:status", capability({ action: "model-account:status" })); + assert.deepEqual(await status.json(), { state: "authorized" }); + const wrongAudience = await post("model-account:cancel", capability({ action: "model-account:cancel", aud: "fly:other:machine" })); + assert.equal(wrongAudience.status, 401); + const expired = await post("model-account:cancel", capability({ action: "model-account:cancel", exp: now - 1 })); + assert.equal(expired.status, 401); + const wrongOrigin = await post("model-account:cancel", capability({ action: "model-account:cancel" }), undefined, "https://attacker.example"); + assert.equal(wrongOrigin.status, 403); + + const canceled = await post("model-account:cancel", capability({ action: "model-account:cancel" })); + assert.deepEqual(await canceled.json(), { state: "canceled" }); + assert.deepEqual(calls.at(-1), ["cancel", accountId]); + now += 61; +}); + +test("authorization is refused while locked and expiry remains bounded to one minute", async t => { + const runtime = { state: "locked" }; + const lifecycle = { async deviceLogin() { assert.fail("locked proxy must not start authorization"); }, async cancel() { return { state: "idle" }; } }; + const server = await startBrowserControl({ runtime, lifecycle, publicKey, proxyId, ownerId, generation, appOrigin, port: 0, host: "127.0.0.1", now: () => 1_900_000_000 }); + t.after(() => server.close()); + const response = await fetch(`http://127.0.0.1:${server.address.port}/model-accounts/${accountId}/authorize`, { + method: "POST", headers: { Origin: appOrigin, "Content-Type": "application/json", Authorization: `Bearer ${capability({ action: "model-account:authorize", exp: 1_900_000_061 })}` }, body: "{}", + }); + assert.equal(response.status, 401); +}); diff --git a/packages/credential-proxy/test/codex-auth-helper.test.mjs b/packages/credential-proxy/test/codex-auth-helper.test.mjs new file mode 100644 index 00000000..9528f76f --- /dev/null +++ b/packages/credential-proxy/test/codex-auth-helper.test.mjs @@ -0,0 +1,87 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { EventEmitter } from "node:events"; +import { PassThrough } from "node:stream"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createCodexAuthHelper, CODEX_AUTH_HELPER_VERSION } from "../src/codex-auth-helper.mjs"; + +const accountA = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; +const accountB = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"; + +function fakeCodex() { + const calls = [], children = []; + const spawnProcess = (command, args, options) => { + calls.push({ command, args, options }); + const child = new EventEmitter(); + child.stdout = new PassThrough(); + child.stderr = new PassThrough(); + child.stdin = new PassThrough(); + child.kill = signal => { + child.killed = signal; + queueMicrotask(() => child.emit("close", null, signal)); + return true; + }; + children.push(child); + if (args[0] !== "login" || args[1] !== "--device-auth") { + queueMicrotask(() => { + child.stdout.end(args[0] === "--version" ? `${CODEX_AUTH_HELPER_VERSION}\n` : "authenticated\n"); + child.stderr.end(); + child.emit("close", 0, null); + }); + } + return child; + }; + return { calls, children, spawnProcess }; +} + +test("Codex auth helper constrains commands and isolates account credential homes", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-codex-auth-")); + try { + const fake = fakeCodex(); + const helper = createCodexAuthHelper({ codexPath: "/opt/codex", accountHomeRoot: directory, spawnProcess: fake.spawnProcess }); + assert.deepEqual(await helper.invoke({ method: "account-status", accountId: accountA }), { state: "authenticated" }); + assert.deepEqual(await helper.invoke({ method: "refresh", accountId: accountB }), { state: "refreshed" }); + assert.deepEqual(fake.calls.map(call => call.args), [ + ["--version"], ["login", "status"], ["--version"], ["login", "status"], + ]); + assert.ok(fake.calls.every(call => call.command === "/opt/codex")); + assert.equal(fake.calls[0].options.env.CODEX_HOME, join(directory, accountA)); + assert.equal(fake.calls[3].options.env.CODEX_HOME, join(directory, accountB)); + assert.equal(fake.calls[0].options.env.HOME, join(directory, accountA)); + assert.equal("OPENAI_API_KEY" in fake.calls[0].options.env, false); + await assert.rejects(helper.invoke({ method: "exec", accountId: accountA, args: ["--dangerously-bypass-approvals-and-sandbox"] }), /unsupported Codex authentication operation/); + await assert.rejects(helper.invoke({ method: "logout", accountId: "../../outside" }), /invalid model account id/); + } finally { await rm(directory, { recursive: true, force: true }); } +}); + +test("device authorization runs as a separate cancellable login process", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-codex-auth-")); + try { + const fake = fakeCodex(); + const helper = createCodexAuthHelper({ accountHomeRoot: directory, spawnProcess: fake.spawnProcess }); + const result = await helper.invoke({ method: "device-login", accountId: accountA }); + assert.equal(result.state, "pending"); + assert.deepEqual(result.challenge, {}); + assert.deepEqual(fake.calls.map(call => call.args), [["--version"], ["login", "--device-auth"]]); + assert.equal(fake.calls[1].options.env.CODEX_HOME, join(directory, accountA)); + fake.children[1].stdout.write("Open this URL in your browser: https://auth.openai.com/codex/device\nEnter user code: abcd-1234\n"); + assert.deepEqual(result.challenge, { verificationUrl: "https://auth.openai.com/codex/device", userCode: "ABCD-1234" }); + assert.deepEqual(await helper.invoke({ method: "cancel", accountId: accountA }), { state: "canceled" }); + await assert.rejects(result.completion, /stopped/); + assert.deepEqual(await helper.invoke({ method: "cancel", accountId: accountA }), { state: "idle" }); + } finally { await rm(directory, { recursive: true, force: true }); } +}); + +test("logout terminates pending device login and invokes only Codex logout", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-codex-auth-")); + try { + const fake = fakeCodex(); + const helper = createCodexAuthHelper({ accountHomeRoot: directory, spawnProcess: fake.spawnProcess }); + const result = await helper.invoke({ method: "device-login", accountId: accountA }); + assert.deepEqual(await helper.invoke({ method: "logout", accountId: accountA }), { state: "logged-out" }); + await assert.rejects(result.completion, /stopped/); + assert.deepEqual(fake.calls.map(call => call.args), [["--version"], ["login", "--device-auth"], ["--version"], ["logout"]]); + } finally { await rm(directory, { recursive: true, force: true }); } +}); diff --git a/packages/credential-proxy/test/control-client.test.mjs b/packages/credential-proxy/test/control-client.test.mjs new file mode 100644 index 00000000..1e425643 --- /dev/null +++ b/packages/credential-proxy/test/control-client.test.mjs @@ -0,0 +1,203 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { consumeProxyEnrollment, createModelAccountPersistenceAdapter, createProxyEnrollmentAuthenticator, issueProxyEnrollment, readProxyIntegrationSnapshot, readProxyRunRevocations, registerProxyBoot, reportProxyRuntimeStatus, startProxyRuntimeStatusReporter, startProxyRunRevocationMonitor } from "../src/control-client.mjs"; + +const token = `fzc_${"A".repeat(43)}`; +const bootEpoch = "22222222-2222-4222-8222-222222222222"; +const runId = "33333333-3333-4333-8333-333333333333"; +const runCredential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + +test("proxy runtime status reports the current boot and runtime unlock state", async () => { + const requests = []; + await reportProxyRuntimeStatus({ controlOrigin: "https://factorize.example", token, bootEpoch, unlocked: true, fetchImpl: async (url, init) => { + requests.push({ url: String(url), init, body: JSON.parse(init.body) }); + return Response.json({ updated: true }); + } }); + assert.equal(requests[0].url, "https://factorize.example/internal/credential-proxy/status"); + assert.equal(requests[0].init.headers.Authorization, `Bearer ${token}`); + assert.deepEqual(requests[0].body, { bootEpoch, unlocked: true }); + await assert.rejects(reportProxyRuntimeStatus({ controlOrigin: "https://factorize.example", token, bootEpoch: "invalid", unlocked: false }), /runtime status/); +}); + +test("runtime status reporter periodically refreshes authenticated status", async () => { + const requests = []; + const runtime = { state: "locked" }; + const monitor = startProxyRuntimeStatusReporter({ controlOrigin: "https://factorize.example", token, bootEpoch, runtime, intervalMs: 1_000, fetchImpl: async (url, init) => { + requests.push({ url: String(url), body: JSON.parse(init.body) }); + return Response.json({ updated: true }); + } }); + await new Promise(resolve => setTimeout(resolve, 5)); + runtime.state = "unlocked"; + await new Promise(resolve => setTimeout(resolve, 5)); + await monitor.close(); + assert.equal(requests[0].body.unlocked, false); + assert.ok(requests.every(request => request.url.endsWith("/internal/credential-proxy/status"))); +}); + +test("boot registration uses a one-call control credential, HTTPS, exact path and fresh process epoch", async () => { + let request; + const result = await registerProxyBoot({ controlOrigin: "https://factorize.example", browserOrigin: "https://proxy.example", token, bootEpoch, fetchImpl: async (url, init) => { + request = { url: String(url), init }; + return Response.json({ registered: true, tenantId: "11111111-1111-4111-8111-111111111111", proxyIdentity: "proxy-fixture", bootEpoch, controlCredential: `fzc_${"c".repeat(43)}` }); + } }); + assert.deepEqual(result, { tenantId: "11111111-1111-4111-8111-111111111111", proxyIdentity: "proxy-fixture", bootEpoch, controlCredential: `fzc_${"c".repeat(43)}` }); + assert.equal(request.url, "https://factorize.example/internal/credential-proxy/register-boot"); + assert.equal(request.init.headers.Authorization, `Bearer ${token}`); + assert.equal(request.init.redirect, "error"); + assert.deepEqual(JSON.parse(request.init.body), { bootEpoch, browserOrigin: "https://proxy.example" }); +}); + +test("proxy snapshot reader requests only the bound run revision and validates encrypted records", async () => { + let request; + const snapshot = { ownerId: "owner-1", records: [{ id: "integration-1", kind: "bearer", origin: "https://api.example", revision: 4, encryptedPayload: "ciphertext" }], modelAccount: { id: "model-1", kind: "model", origin: null, revision: 5, encryptedPayload: "model-ciphertext" } }; + const result = await readProxyIntegrationSnapshot({ controlOrigin: "https://factorize.example", token, binding: { runId, grantRevision: 7 }, fetchImpl: async (url, init) => { + request = { url: String(url), init, body: JSON.parse(init.body) }; + return Response.json(snapshot); + } }); + assert.deepEqual(result, snapshot); + assert.equal(request.url, "https://factorize.example/internal/credential-proxy/integration-snapshot"); + assert.deepEqual(request.body, { runId, grantRevision: 7 }); + assert.equal(request.init.headers.Authorization, `Bearer ${token}`); + await assert.rejects(readProxyIntegrationSnapshot({ controlOrigin: "https://factorize.example", token, binding: { runId, grantRevision: 7 }, fetchImpl: async () => Response.json({ ownerId: "owner-1", records: [{ ...snapshot.records[0], kind: "model" }] }) }), /invalid proxy integration snapshot response/); + await assert.rejects(readProxyIntegrationSnapshot({ controlOrigin: "https://factorize.example", token, binding: { runId, grantRevision: 7 }, fetchImpl: async () => Response.json({ ...snapshot, modelAccount: { ...snapshot.modelAccount, kind: "bearer" } }) }), /invalid proxy integration snapshot response/); +}); + +test("model account persistence adapter lists owner-bound account revisions over proxy authentication", async () => { + const ownerId = "11111111-1111-4111-8111-111111111111"; + let request; + const keyGeneration = "22222222-2222-4222-8222-222222222222"; + const accountId = "33333333-3333-4333-8333-333333333333"; + const adapter = createModelAccountPersistenceAdapter({ + controlOrigin: "https://factorize.example", controlCredential: token, ownerId, + fetchImpl: async (url, init) => { + request = { url: String(url), init, body: JSON.parse(init.body) }; + if (request.body.operation === "save") return Response.json({ accountId, revision: 5, collectionRevision: 9, duplicate: false }); + if (request.body.operation === "read") return Response.json({ ownerId, account: { id: accountId, kind: "model", origin: null, revision: 4, encryptedPayload: "ciphertext", keyGeneration } }); + return Response.json({ ownerId, keyGeneration, accounts: [{ id: accountId, kind: "model", origin: null, revision: 4 }] }); + }, + }); + assert.deepEqual(await adapter.listAccounts(), [{ id: accountId, kind: "model", origin: null, revision: 4, ownerId, keyGeneration }]); + assert.equal(request.url, "https://factorize.example/internal/credential-proxy/model-account-cache"); + assert.deepEqual(request.body, { operation: "list" }); + assert.equal(request.init.headers.Authorization, `Bearer ${token}`); + await adapter.persistEncrypted({ id: accountId, expectedRevision: 4, keyGeneration, encryptedPayload: "ciphertext", idempotencyKey: "44444444-4444-4444-8444-444444444444" }); + assert.deepEqual(request.body, { operation: "save", accountId, expectedRevision: 4, keyGeneration, encryptedPayload: "ciphertext", idempotencyKey: "44444444-4444-4444-8444-444444444444" }); + await assert.rejects(adapter.persistEncrypted({ id: accountId, expectedRevision: 4, encryptedPayload: "ciphertext", idempotencyKey: "44444444-4444-4444-8444-444444444444" }), /invalid encrypted model account update/); + await assert.rejects(adapter.listAccounts && createModelAccountPersistenceAdapter({ + controlOrigin: "https://factorize.example", controlCredential: token, ownerId, + fetchImpl: async () => Response.json({ ownerId: "22222222-2222-4222-8222-222222222222", keyGeneration: "22222222-2222-4222-8222-222222222222", accounts: [] }), + }).listAccounts(), /invalid model account list response/); +}); + +test("proxy revocation monitor applies durable run revocations and tolerates an outage", async () => { + const requests = []; + const revoked = []; + let failFirst = true; + const fetchImpl = async (url, init) => { + requests.push({ url: String(url), init, body: JSON.parse(init.body) }); + if (failFirst) { failFirst = false; throw new Error("offline fixture"); } + return Response.json({ revokedRunIds: [runId] }); + }; + const runtime = { revokeRun: async id => { revoked.push(id); } }; + const monitor = startProxyRunRevocationMonitor({ + controlOrigin: "https://factorize.example", controlCredential: token, bootEpoch, runtime, intervalMs: 100, fetchImpl, + }); + try { + const deadline = Date.now() + 1_000; + while (!revoked.length && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 10)); + assert.deepEqual(revoked, [runId]); + assert.equal(requests.length, 2, "a failed poll leaves the proxy live and retries through the control plane"); + assert.ok(requests.every(request => request.url === "https://factorize.example/internal/credential-proxy/revocations")); + assert.ok(requests.every(request => request.init.headers.Authorization === `Bearer ${token}`)); + assert.ok(requests.every(request => request.body.bootEpoch === bootEpoch)); + } finally { + await monitor.close(); + } + await assert.rejects(readProxyRunRevocations({ controlOrigin: "https://factorize.example", token, bootEpoch, fetchImpl: async () => Response.json({ revokedRunIds: ["invalid"] }) }), /invalid proxy revocation response/); +}); + +test("boot registration rejects unsafe origins, invalid credentials and mismatched response bindings", async () => { + await assert.rejects(registerProxyBoot({ controlOrigin: "http://factorize.example", token, bootEpoch, browserOrigin: "https://proxy.example" }), /HTTPS origin/); + await assert.rejects(registerProxyBoot({ controlOrigin: "https://factorize.example", token: "fzt_run_token", bootEpoch, browserOrigin: "https://proxy.example" }), /credential/); + await assert.rejects(registerProxyBoot({ controlOrigin: "https://factorize.example", token, bootEpoch, browserOrigin: "https://proxy.example", fetchImpl: async () => Response.json({ registered: true, proxyIdentity: "proxy-fixture", bootEpoch: "33333333-3333-4333-8333-333333333333" }) }), /invalid proxy boot registration response/); +}); + +test("proxy control client requests and consumes a one-use enrollment over authenticated HTTPS", async () => { + const controlToken = `fzc_${"c".repeat(43)}`; + const requests = []; + const fetchImpl = async (url, init) => { + requests.push({ url: String(url), init }); + return requests.length === 1 + ? Response.json({ enrollmentId: "enrollment-1", token: "e".repeat(43), grantRevision: 2 }) + : Response.json({ consumed: true, binding: { enrollmentId: "enrollment-1" } }); + }; + const common = { controlOrigin: "https://factorize.example", token: controlToken, fetchImpl }; + const enrollment = await issueProxyEnrollment({ ...common, body: { runId: "33333333-3333-4333-8333-333333333333", vmIdentity: "fly:vm-1" } }); + assert.equal(enrollment.token, "e".repeat(43)); + const consumed = await consumeProxyEnrollment({ ...common, body: { runId: "33333333-3333-4333-8333-333333333333", vmIdentity: "fly:vm-1", enrollmentToken: enrollment.token } }); + assert.equal(consumed.enrollmentId, "enrollment-1"); + assert.deepEqual(requests.map(request => request.url), [ + "https://factorize.example/internal/credential-proxy/enroll", + "https://factorize.example/internal/credential-proxy/consume-enrollment", + ]); + assert.ok(requests.every(request => request.init.headers.Authorization === `Bearer ${controlToken}` && request.init.redirect === "error")); +}); + +test("proxy enrollment authenticator binds issued and consumed state to the mTLS VM and current boot", async () => { + const controlToken = `fzc_${"d".repeat(43)}`; + const requests = []; + const grants = [{ id: "tool-1", origin: "https://api.example.test", credential: "synthetic-secret" }]; + const fetchImpl = async (url, init) => { + const request = { url: String(url), init, body: JSON.parse(init.body) }; + requests.push(request); + if (requests.length === 1) return Response.json({ enrollmentId: "enrollment-1", token: "e".repeat(43), grantRevision: 7 }); + return Response.json({ consumed: true, binding: { + tenantId: "11111111-1111-4111-8111-111111111111", + runId: request.body.runId, + vmIdentity: request.body.vmIdentity, + proxyIdentity: "fly:proxy-1", + bootEpoch, + enrollmentId: "enrollment-1", + grantRevision: 7, + } }); + }; + const authenticateRun = createProxyEnrollmentAuthenticator({ + controlOrigin: "https://factorize.example", + controlCredential: controlToken, + bootEpoch, + resolveRunCredential: (id, vmIdentity) => id === runId && vmIdentity === "fly:vm-1" ? runCredential : undefined, + fetchImpl, + resolveGrants: async binding => { + assert.equal(binding.runId, "33333333-3333-4333-8333-333333333333"); + return grants; + }, + }); + const connection = await authenticateRun({ + runId, + vmIdentity: "fly:vm-1", + bootEpoch, + }); + assert.deepEqual(requests.map(request => request.url), [ + "https://factorize.example/internal/credential-proxy/enroll", + "https://factorize.example/internal/credential-proxy/consume-enrollment", + ]); + assert.deepEqual(requests.map(request => request.body), [ + { runId, vmIdentity: "fly:vm-1", runCredential }, + { runId, vmIdentity: "fly:vm-1", enrollmentToken: "e".repeat(43) }, + ]); + assert.ok(requests.every(request => request.init.headers.Authorization === `Bearer ${controlToken}`)); + assert.equal(connection.authenticated, true); + assert.equal(connection.enrollmentConsumed, true); + assert.equal(connection.vmId, "fly:vm-1"); + assert.equal(connection.bootEpoch, bootEpoch); + assert.equal(connection.connectionId, "enrollment-1"); + assert.deepEqual(connection.grants, grants); + await assert.rejects(authenticateRun({ + runId, + vmIdentity: "fly:vm-1", + bootEpoch: "44444444-4444-4444-8444-444444444444", + }), /invalid proxy enrollment identity/); + assert.equal(requests.length, 2, "a stale boot cannot issue another enrollment"); + await assert.rejects(authenticateRun({ runId, vmIdentity: "fly:vm-2", bootEpoch }), /run credential is unavailable/); + assert.equal(requests.length, 2, "a missing job run credential cannot issue enrollment"); +}); diff --git a/packages/credential-proxy/test/decrypted-grants.test.mjs b/packages/credential-proxy/test/decrypted-grants.test.mjs new file mode 100644 index 00000000..10163ee3 --- /dev/null +++ b/packages/credential-proxy/test/decrypted-grants.test.mjs @@ -0,0 +1,100 @@ +import assert from "node:assert/strict"; +import { webcrypto } from "node:crypto"; +import { test } from "node:test"; +import { createDecryptedGrantResolver, createProxyIntegrationGrantResolver } from "../src/decrypted-grants.mjs"; +import { createProxyRuntime } from "../src/runtime.mjs"; + +const encoder = new TextEncoder(); +const salt = encoder.encode("factorize.unlock-key.v1"); +const info = encoder.encode("factorize.record.aes-256-gcm.v1"); +const ownerId = "owner-fixture"; +const unlockKey = `fzk1.${Buffer.from(webcrypto.getRandomValues(new Uint8Array(32))).toString("base64url")}`; + +async function encryptCredential({ id, origin, revision, credential }) { + const raw = Buffer.from(unlockKey.slice("fzk1.".length), "base64url"); + const material = await webcrypto.subtle.importKey("raw", raw, "HKDF", false, ["deriveKey"]); + const aesKey = await webcrypto.subtle.deriveKey({ name: "HKDF", hash: "SHA-256", salt, info }, material, + { name: "AES-GCM", length: 256 }, false, ["encrypt"]); + const nonce = webcrypto.getRandomValues(new Uint8Array(12)); + const additionalData = encoder.encode(JSON.stringify(["factorize-record-v1", ownerId, id, "bearer", origin, revision])); + const ciphertext = new Uint8Array(await webcrypto.subtle.encrypt({ name: "AES-GCM", iv: nonce, additionalData, tagLength: 128 }, aesKey, encoder.encode(credential))); + return `${Buffer.from(nonce).toString("base64")}.${Buffer.from(ciphertext).toString("base64")}`; +} + +test("resolver decrypts only selected active bearer records using the shared record binding", async () => { + const first = { id: "integration-a", kind: "bearer", origin: "https://api.example.test", revision: 3, deletedAt: null }; + const second = { id: "integration-b", kind: "bearer", origin: "https://other.example.test", revision: 4, deletedAt: null }; + const records = [ + { ...first, encryptedPayload: await encryptCredential({ ...first, credential: "first-token" }) }, + { ...second, encryptedPayload: await encryptCredential({ ...second, credential: "second-token" }) }, + { id: "deleted", kind: "bearer", origin: "https://api.example.test", revision: 5, deletedAt: new Date(), encryptedPayload: "unused" }, + ]; + const resolver = createDecryptedGrantResolver({ + readSnapshot: async () => ({ ownerId, records }), + getUnlockKey: async () => unlockKey, + }); + assert.deepEqual(await resolver({ tenantId: "tenant-fixture", integrationIds: ["integration-b"] }), [ + { id: "integration-b", name: "integration-b", description: null, origin: "https://other.example.test", credential: "second-token" }, + ]); + assert.deepEqual(await resolver({ tenantId: "tenant-fixture", integrationIds: [] }), []); + const runResolver = createDecryptedGrantResolver({ + readSnapshot: async () => ({ ownerId, records }), + resolveIntegrationIds: async binding => binding.runId === "run-fixture" ? ["integration-a"] : [], + getUnlockKey: async () => unlockKey, + }); + assert.deepEqual(await runResolver({ tenantId: "tenant-fixture", runId: "run-fixture" }), [ + { id: "integration-a", name: "integration-a", description: null, origin: "https://api.example.test", credential: "first-token" }, + ]); +}); + +test("resolver rejects missing, ambiguous, malformed, and misbound grants and omits deleted selections", async () => { + const record = { id: "integration-a", kind: "bearer", origin: "https://api.example.test", revision: 3, deletedAt: null }; + const encryptedPayload = await encryptCredential({ ...record, credential: "synthetic-token" }); + const resolver = createDecryptedGrantResolver({ readSnapshot: async () => ({ ownerId, records: [{ ...record, encryptedPayload }] }), getUnlockKey: async () => unlockKey }); + await assert.rejects(resolver({ tenantId: "tenant-fixture", integrationIds: ["missing"] }), /unavailable/); + const deleted = createDecryptedGrantResolver({ readSnapshot: async () => ({ ownerId, records: [{ ...record, deletedAt: new Date(), encryptedPayload: null }] }), getUnlockKey: async () => unlockKey }); + assert.deepEqual(await deleted({ tenantId: "tenant-fixture", integrationIds: ["integration-a"] }), []); + await assert.rejects(resolver({ tenantId: "tenant-fixture", integrationIds: ["integration-a", "integration-a"] }), /selection/); + const tampered = createDecryptedGrantResolver({ readSnapshot: async () => ({ ownerId: "other-owner", records: [{ ...record, encryptedPayload }] }), getUnlockKey: async () => unlockKey }); + await assert.rejects(tampered({ tenantId: "tenant-fixture", integrationIds: ["integration-a"] })); + const locked = createDecryptedGrantResolver({ readSnapshot: async () => ({ ownerId, records: [{ ...record, encryptedPayload }] }), getUnlockKey: async () => undefined }); + await assert.rejects(locked({ tenantId: "tenant-fixture", integrationIds: ["integration-a"] }), /locked/); +}); + +test("resolver rejects non-HTTPS destinations and invalid credential bytes", async () => { + const badOrigin = { id: "integration-a", kind: "bearer", origin: "http://api.example.test", revision: 3, deletedAt: null, + encryptedPayload: await encryptCredential({ id: "integration-a", origin: "http://api.example.test", revision: 3, credential: "token" }) }; + const badOriginResolver = createDecryptedGrantResolver({ readSnapshot: async () => ({ ownerId, records: [badOrigin] }), getUnlockKey: async () => unlockKey }); + await assert.rejects(badOriginResolver({ tenantId: "tenant-fixture", integrationIds: ["integration-a"] }), /integration origin/); + const badCredential = { ...badOrigin, origin: "https://api.example.test", encryptedPayload: await encryptCredential({ id: "integration-a", origin: "https://api.example.test", revision: 3, credential: "bad\ncredential" }) }; + const badCredentialResolver = createDecryptedGrantResolver({ readSnapshot: async () => ({ ownerId, records: [badCredential] }), getUnlockKey: async () => unlockKey }); + await assert.rejects(badCredentialResolver({ tenantId: "tenant-fixture", integrationIds: ["integration-a"] }), /credential/); +}); + +test("production grant resolver reads the run-bound snapshot and only decrypts while the runtime is unlocked", async () => { + const record = { id: "integration-a", kind: "bearer", origin: "https://api.example.test", revision: 3, deletedAt: null }; + const encryptedPayload = await encryptCredential({ ...record, credential: "snapshot-token" }); + const runtime = createProxyRuntime({ bootEpoch: "11111111-1111-4111-8111-111111111111" }); + const requests = []; + const resolve = createProxyIntegrationGrantResolver({ + runtime, + controlOrigin: "https://factorize.example", + controlCredential: `fzc_${"c".repeat(43)}`, + fetchImpl: async (url, init) => { + requests.push({ url: String(url), authorization: new Headers(init.headers).get("authorization"), body: JSON.parse(init.body) }); + return Response.json({ ownerId, records: [{ ...record, encryptedPayload }], modelAccount: null }); + }, + }); + const binding = { tenantId: ownerId, runId: "22222222-2222-4222-8222-222222222222", grantRevision: 7 }; + await assert.rejects(resolve(binding), /locked/); + runtime.unlock(unlockKey); + assert.deepEqual(await resolve(binding), [{ id: record.id, name: record.id, description: null, origin: record.origin, credential: "snapshot-token" }]); + assert.deepEqual(requests, [{ + url: "https://factorize.example/internal/credential-proxy/integration-snapshot", + authorization: `Bearer fzc_${"c".repeat(43)}`, + body: { runId: binding.runId, grantRevision: binding.grantRevision }, + }]); + await runtime.lock(); + await assert.rejects(resolve(binding), /locked/); + await runtime.shutdown(); +}); diff --git a/packages/credential-proxy/test/destination-policy.test.mjs b/packages/credential-proxy/test/destination-policy.test.mjs index d6266632..264a3055 100644 --- a/packages/credential-proxy/test/destination-policy.test.mjs +++ b/packages/credential-proxy/test/destination-policy.test.mjs @@ -18,7 +18,7 @@ test("URL policy permits paths and queries only on the configured HTTPS origin", const result = await connect("https://api.example.test:8443/v1/items?limit=5"); assert.equal(result.url.pathname, "/v1/items"); assert.equal(result.url.search, "?limit=5"); - assert.deepEqual(calls, [{ hostname: "api.example.test", port: 8443, address: "93.184.216.34", servername: "api.example.test" }]); + assert.deepEqual(calls, [{ hostname: "api.example.test", port: 8443, address: "93.184.216.34", servername: "api.example.test", timeoutMs: 10_000 }]); }); test("rejects unsafe and malformed request URLs before resolution", async () => { diff --git a/packages/credential-proxy/test/fixtures/proxy-cert.pem b/packages/credential-proxy/test/fixtures/proxy-cert.pem new file mode 100644 index 00000000..dd16e31e --- /dev/null +++ b/packages/credential-proxy/test/fixtures/proxy-cert.pem @@ -0,0 +1,19 @@ +-----BEGIN CERTIFICATE----- +MIIDDzCCAfegAwIBAgIUMxculG7HF30xOo+RWOG+FmORP70wDQYJKoZIhvcNAQEL +BQAwGzEZMBcGA1UEAwwQYXBpLmV4YW1wbGUudGVzdDAeFw0yNjExMTIxMTQ0Mzla +Fw0yNzExMTIxMTQ0MzlaMBQxEjAQBgNVBAMMCTEyNy4wLjAuMTCCASIwDQYJKoZI +hvcNAQEBBQADggEPADCCAQoCggEBAMKWAZNteehv0YdtXEy3WVKax9bqVJ+44QES +4Tmf3pspx9KxUoxEq/DePw/IZqwX4NNBPCoWVxq8fOaZWQ4GlBgtLPdu8PRGconw +JeFnBxFIDvsEQ1ElAndNGv1YgBWvPd2CQASXlO/hj7QogtGpkzH6+DVHQK4DO3ol +uEZO8W6BECB5Imii1X871S77PhKq0Rs6SmPuKxFQlrW6KsCYqgW5whSxr/jBMGRY +gvvhnaIQx2Y0zQdDTFOyhWYVUlv7LEKgKvmvMh0+4YPJB6pZpa2O7tmRUmNSyTJi +BKvJxSK/EASVj1p52CbJnGgEgeLB0WxK3827kaP2Y5qGoB8A92MCAwEAAaNSMFAw +DwYDVR0RBAgwBocEfwAAATATBgNVHSUEDDAKBggrBgEFBQcDATAJBgNVHRMEAjAA +MB0GA1UdDgQWBBRnufSGyhOp/K9MV0Q1L6bh3O/uyzANBgkqhkiG9w0BAQsFAAOC +AQEAM+OJXc2S22LNexkQSF915kwFvbE/n3P/DKlRlOSR0Rg58fJTHwp2Qrmk2T5z +Lj5nj9nQoxBdittd/i1EzEilPBMNkS5v+yGQ+j2KZ7RGQakjWyZUftJ4iSUX0dJj +35GEONpqkR3T1cAnlilVgcDb5kCBJJmmL78TY4uMRIjKO0c2rMuXZj+mZTTEKBd4 +ejWqHDAou6MvQbKPr7UcXyi7yrvGJYlFrser3MhW565QuGQBMueSF5csHcAqsEw8 +zo4HpIgQm9unIeWITH2pJ06VKDC89YExk6H8CbgXXZr2tobJDvQwnPj/5ufI/pGZ +lj/JTVRcIpCOEWdvqo9Per/dBQ== +-----END CERTIFICATE----- diff --git a/packages/credential-proxy/test/fixtures/upstream-cert.pem b/packages/credential-proxy/test/fixtures/upstream-cert.pem new file mode 100644 index 00000000..afec81f5 --- /dev/null +++ b/packages/credential-proxy/test/fixtures/upstream-cert.pem @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDTjCCAjagAwIBAgIUPy+fWru0gF7QNal9Xg4vXdIuUCgwDQYJKoZIhvcNAQEL +BQAwGzEZMBcGA1UEAwwQYXBpLmV4YW1wbGUudGVzdDAeFw0yNjExMTEyMzA3Mjla +Fw0zNjExMDgyMzA3MjlaMBsxGTAXBgNVBAMMEGFwaS5leGFtcGxlLnRlc3QwggEi +MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDClgGTbXnob9GHbVxMt1lSmsfW +6lSfuOEBEuE5n96bKcfSsVKMRKvw3j8PyGasF+DTQTwqFlcavHzmmVkOBpQYLSz3 +bvD0RnKJ8CXhZwcRSA77BENRJQJ3TRr9WIAVrz3dgkAEl5Tv4Y+0KILRqZMx+vg1 +R0CuAzt6JbhGTvFugRAgeSJootV/O9Uu+z4SqtEbOkpj7isRUJa1uirAmKoFucIU +sa/4wTBkWIL74Z2iEMdmNM0HQ0xTsoVmFVJb+yxCoCr5rzIdPuGDyQeqWaWtju7Z +kVJjUskyYgSrycUivxAElY9aedgmyZxoBIHiwdFsSt/Nu5Gj9mOahqAfAPdjAgMB +AAGjgYkwgYYwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAqQwRAYDVR0R +BD0wO4IQYXBpLmV4YW1wbGUudGVzdIYndXJuOmZhY3Rvcml6ZTp2bTpZWEJwTG1W +NFlXMXdiR1V1ZEdWemRBMB0GA1UdDgQWBBRnufSGyhOp/K9MV0Q1L6bh3O/uyzAN +BgkqhkiG9w0BAQsFAAOCAQEASBvpzgJtGPUxB8ao20lUU2J95DRPFH6BEPoJq38+ +62D7YTNIesVPykdnZQy/8uVWczIAKURAU/MRv10N8K6nfo/PAx3Hi2m8qbnoTcRa +2JLoE++bQltKI7GAg//wviMpb3CjgObjoIYVa8pdHvR2PLEIroJuGitOyZTWDBFh +bu5VcH5FDkCLD8DzatgGyEVDDR9vELzS5m1j9WgVXAHnMhnKegmvhhJhsdT+/LEE +gtNFdHYNRPGbxhJ6Kc8CbmSZN7T8zUjCTHfcAB5iz6lrQRbbbxzIekXeqzv+Ya0C +vnY+CxOlQab0mac1Y72EnoDMSkvAywIbdwe9qcpITlrKmQ== +-----END CERTIFICATE----- diff --git a/packages/credential-proxy/test/fixtures/upstream-key.pem b/packages/credential-proxy/test/fixtures/upstream-key.pem new file mode 100644 index 00000000..69f0bcaa --- /dev/null +++ b/packages/credential-proxy/test/fixtures/upstream-key.pem @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDClgGTbXnob9GH +bVxMt1lSmsfW6lSfuOEBEuE5n96bKcfSsVKMRKvw3j8PyGasF+DTQTwqFlcavHzm +mVkOBpQYLSz3bvD0RnKJ8CXhZwcRSA77BENRJQJ3TRr9WIAVrz3dgkAEl5Tv4Y+0 +KILRqZMx+vg1R0CuAzt6JbhGTvFugRAgeSJootV/O9Uu+z4SqtEbOkpj7isRUJa1 +uirAmKoFucIUsa/4wTBkWIL74Z2iEMdmNM0HQ0xTsoVmFVJb+yxCoCr5rzIdPuGD +yQeqWaWtju7ZkVJjUskyYgSrycUivxAElY9aedgmyZxoBIHiwdFsSt/Nu5Gj9mOa +hqAfAPdjAgMBAAECggEAHCixGbCiD6z+2WZOERwHZRjZCpTmoiaTCSC+TyR/hlzJ +eggHNh1J9JV08sfBc+4QrwGlv0F0YX4dZRGT/Z+eFtGSzEvVusPnF2cVVEUglLOk +11aKl1ePv8v8t+TyxDCT91Y+BWwaNYyiyyOBqu2rS3RSLwOcrJHzLX3/6MzLy33I +1++WK9wGUKEyzfQ8k363yUv1CmhN/4kJs9a00OLc9pvBKO/LLxMxGZ3lMUsVhVUc +3h0yjlRvhvYDvZeLJKGhWVhn1wPfa6BNPwMNatJ6O5ubdA1qegOOxi+VDheSg05H +MVILqoFWt6B7AY+iMHbzbR1YJxSYWNVA9X9KWqtWoQKBgQD7EnZ1137IBFeC0UwK +bvjnh92lPCg3Ek8iBaH+tHga08KZr4xR10tsbqXIKxoGBeE0m3uOQbnZ0cDYSlSP +MsEXk3hthnpSNT1Ln/K1sc5ht7wb0xhDzbVsRjhtH0Ae64LrExhn/wlTirrtwDcx +6Dw1zymbr7a6LflUtHNB7V28VwKBgQDGZ7kWt7w7yDZlLQPkEmJ3BIXTvDKPsfCv +1R4Xwn/ZHaXA8J6CHZ3tEzOSWSPU7TOCSwu3mrMS+CGK5GINOYJ54YfNcgbQJ+7G +Yzw7yUFidbhJ6TBulhmCO8gdR743CRweC0K28bVe5suPnNNdlJDyowwtboua9Vn5 +H0sxcNz11QKBgQDmqxDAtYqaUzrfSlE1aL0vYqklLP65kg238Tri2oVaCGHKdvlR +G58E8SdInyjajb5Zadv7V7FbSJWHYcsMgRHcCjXjUmDBdH5rI3mGquQ07OxbF/c8 +HBtpbZ5byxFLFw8B0Fpk+XBlG5kf47ASHw5qVXkhuy8DIEfpJZya3mx3IwKBgFNN +IU0Ef6ep/AymxcdwYJ8P6xBnJ+NlFmT82dGn+/2wkPNdE3AV90SelzyO6Jhxrvq5 +f4XrEdyZCZxDH1fmGX4KzvXXNj0XRw1MhPGlIiqwultt6MPFiZhAeVix0o9Q59lF +aRFekoPx6u21LplP0zvMcURx8XNyKACMvwlXFa6xAoGAK5qaim0rJzwrtlr8T1z/ +HxzM4ZAZJchb8fN7LExj2ZrzCmZY2pnpS8oH3ae9iopV9LKUR+b6OwgnV8eWv1qx +Ldc3l7R/8AD9MOmuFwU36VozB/esox6r6MKaBl8m009wceouZG7uV4KvKr2E2Kry +dfnJ6oBHEnM8GcAQjZ3tqJA= +-----END PRIVATE KEY----- diff --git a/packages/credential-proxy/test/fly-guest-helper.e2e.test.mjs b/packages/credential-proxy/test/fly-guest-helper.e2e.test.mjs new file mode 100644 index 00000000..914f2838 --- /dev/null +++ b/packages/credential-proxy/test/fly-guest-helper.e2e.test.mjs @@ -0,0 +1,324 @@ +import assert from "node:assert/strict"; +import { webcrypto } from "node:crypto"; +import { spawn } from "node:child_process"; +import { mkdtemp, readFile, rm, writeFile, mkdir } from "node:fs/promises"; +import { createServer as createHttpsServer } from "node:https"; +import { connect as tcpConnect } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { once } from "node:events"; +import { connect as tlsConnect } from "node:tls"; +import { test } from "node:test"; +import { FACTORIZE_FLY_TUNNEL_HELPER } from "../../api/src/fly-job-tunnel-helper.ts"; +import { createBearerForwarder } from "../src/bearer-forwarder.mjs"; +import { createAuthenticatedProxyListener } from "../src/proxy-listener.mjs"; +import { createProxyRuntime } from "../src/runtime.mjs"; +import { createProxyRunCredentialStore } from "../src/run-credential-store.mjs"; +import { createProxyIntegrationGrantResolver } from "../src/decrypted-grants.mjs"; +import { startAuthenticatedProxyTunnel, startVmTunnelHelper } from "../src/tunnel-transport.mjs"; + +const fixtures = new URL("./fixtures/", import.meta.url); +const key = await readFile(new URL("upstream-key.pem", fixtures)); +const cert = await readFile(new URL("upstream-cert.pem", fixtures)); +const proxyCert = await readFile(new URL("proxy-cert.pem", fixtures)); + +async function freePort() { + const { createServer } = await import("node:net"); + const server = createServer(); + server.listen(0, "127.0.0.1"); + await once(server, "listening"); + const { port } = server.address(); + await new Promise(resolve => server.close(resolve)); + return port; +} + +async function waitForServer(child, port) { + const deadline = Date.now() + 5000; + while (Date.now() < deadline) { + if (child.exitCode !== null) throw new Error(`Fly guest helper exited (${child.exitCode})`); + try { + const socket = await new Promise((resolve, reject) => { + const candidate = tcpConnect(port, "127.0.0.1"); + candidate.once("connect", () => resolve(candidate)); + candidate.once("error", reject); + }); + socket.destroy(); + return; + } catch { await new Promise(resolve => setTimeout(resolve, 25)); } + } + throw new Error("Fly guest helper did not listen"); +} + +async function waitForProxyUnlock(port) { + const deadline = Date.now() + 2000; + while (Date.now() < deadline) { + try { + const response = await fetch(`http://127.0.0.1:${port}/integrations`, { signal: AbortSignal.timeout(250) }); + if (response.ok && (await response.json()).state === "unlocked") return; + } catch {} + await new Promise(resolve => setTimeout(resolve, 10)); + } + throw new Error("proxy did not refresh its scoped grants after unlock"); +} + +test("generated Fly guest helper forwards through identity-bound mTLS and rejects another VM binding", { timeout: 10_000 }, async () => { + let observedAuthorization; + const upstream = createHttpsServer({ key, cert }, (request, response) => { + observedAuthorization = request.headers.authorization; + response.end("fly-guest-tunnel-ok"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = upstream.address().port; + const forwarder = createBearerForwarder({ + connectorFactory: origin => async requestUrl => { + assert.equal(origin, `https://api.example.test:${upstreamPort}`); + const url = new URL(requestUrl); + const socket = tlsConnect({ host: "127.0.0.1", port: upstreamPort, servername: url.hostname, ca: cert }); + await once(socket, "secureConnect"); + return { socket, url }; + }, + }); + const runtime = createProxyRuntime({ + listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }), + }); + runtime.unlock("synthetic-unlock-key"); + const credentials = createProxyRunCredentialStore(); + const runId = "33333333-3333-4333-8333-333333333333"; + const crossVmRunId = "44444444-4444-4444-8444-444444444444"; + const runCredential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + const crossVmCredential = `fzt_run_${Buffer.from(crossVmRunId).toString("base64url")}_${"S".repeat(43)}`; + credentials.authorize({ runId, vmIdentity: "api.example.test", credential: runCredential }); + credentials.authorize({ runId: crossVmRunId, vmIdentity: "another-vm.example.test", credential: crossVmCredential }); + let enrollments = 0; + let rejectedRunBinding = false; + const tunnel = await startAuthenticatedProxyTunnel({ + tls: { key, cert: proxyCert, ca: cert }, + runtime, + controlOrigin: "https://factorize.example", + controlCredential: `fzc_${"c".repeat(43)}`, + resolveRunCredential: (id, vmIdentity) => { + if (id === crossVmRunId && vmIdentity === "api.example.test") rejectedRunBinding = true; + return credentials.resolve(id, vmIdentity); + }, + fetchImpl: async (url, init) => { + const path = new URL(url).pathname; + const body = JSON.parse(init.body); + if (path.endsWith("/enroll")) { + enrollments++; + if (body.runId !== runId || body.vmIdentity !== "api.example.test") return Response.json({ error: "run is not bound to this VM" }, { status: 403 }); + return Response.json({ enrollmentId: `11111111-1111-4111-8111-${String(enrollments).padStart(12, "0")}`, token: "e".repeat(43), grantRevision: 7 }); + } + const enrollmentId = `11111111-1111-4111-8111-${String(enrollments).padStart(12, "0")}`; + return Response.json({ consumed: true, binding: { + tenantId: "22222222-2222-4222-8222-222222222222", + runId, + vmIdentity: "api.example.test", + proxyIdentity: "proxy-fixture", + bootEpoch: runtime.bootEpoch, + enrollmentId, + grantRevision: 7, + } }); + }, + resolveGrants: async () => [{ id: "tool-1", origin: `https://api.example.test:${upstreamPort}`, credential: "synthetic-bearer" }], + }); + + const root = await mkdtemp(join(tmpdir(), "factorize-fly-guest-")); + const configRoot = join(root, "proxy"); + const helperPort = await freePort(); + let helper; + try { + await mkdir(configRoot); + await Promise.all([ + writeFile(join(configRoot, "client.key"), key), + writeFile(join(configRoot, "client.crt"), cert), + writeFile(join(configRoot, "ca.crt"), cert), + writeFile(join(configRoot, "config.json"), JSON.stringify({ + proxyHost: "127.0.0.1", proxyPort: tunnel.address().port, localPort: helperPort, runId, + })), + ]); + const source = FACTORIZE_FLY_TUNNEL_HELPER.replace( + 'const root = "/home/exedev/.factorize-job/proxy";', + `const root = ${JSON.stringify(configRoot)};`, + ); + helper = spawn(process.execPath, ["--input-type=module", "-e", source], { stdio: "ignore" }); + await waitForServer(helper, helperPort); + const response = await fetch(`http://127.0.0.1:${helperPort}/integrations/tool-1/echo`, { + headers: { authorization: "Bearer caller-controlled" }, + }); + assert.equal(response.status, 200); + assert.equal(await response.text(), "fly-guest-tunnel-ok"); + assert.equal(observedAuthorization, "Bearer synthetic-bearer"); + + // This VM certificate identifies api.example.test, while the second run is + // durably bound to another VM. The real control authenticator must fail closed. + const secondPort = await freePort(); + const secondRoot = join(root, "other-run"); + await mkdir(secondRoot); + await Promise.all([ + writeFile(join(secondRoot, "client.key"), key), + writeFile(join(secondRoot, "client.crt"), cert), + writeFile(join(secondRoot, "ca.crt"), cert), + writeFile(join(secondRoot, "config.json"), JSON.stringify({ + proxyHost: "127.0.0.1", proxyPort: tunnel.address().port, localPort: secondPort, runId: crossVmRunId, + })), + ]); + const otherSource = FACTORIZE_FLY_TUNNEL_HELPER.replace( + 'const root = "/home/exedev/.factorize-job/proxy";', + `const root = ${JSON.stringify(secondRoot)};`, + ); + const otherHelper = spawn(process.execPath, ["--input-type=module", "-e", otherSource], { stdio: "ignore" }); + try { + await waitForServer(otherHelper, secondPort); + await assert.rejects(fetch(`http://127.0.0.1:${secondPort}/integrations/tool-1/echo`, { + signal: AbortSignal.timeout(1000), + }), /fetch failed|aborted|timeout/i); + assert.equal(rejectedRunBinding, true); + } finally { + otherHelper.kill("SIGTERM"); + } + } finally { + helper?.kill("SIGTERM"); + await rm(root, { recursive: true, force: true }); + await tunnel.close(); + await runtime.shutdown(); + credentials.clear(); + await new Promise((resolve, reject) => upstream.close(error => error ? reject(error) : resolve())); + } +}); + +test("locked scoped run reconnects with its current grant while another VM stays denied", { timeout: 10_000 }, async () => { + const ownerId = "owner-e2e-fixture"; + const integrationId = "scoped-metrics"; + const runId = "55555555-5555-4555-8555-555555555555"; + const otherRunId = "66666666-6666-4666-8666-666666666666"; + const vmIdentity = "api.example.test"; + const otherVmIdentity = "another-vm.example.test"; + const controlCredential = `fzc_${"c".repeat(43)}`; + const unlockKey = `fzk1.${Buffer.from(webcrypto.getRandomValues(new Uint8Array(32))).toString("base64url")}`; + const bootEpoch = "77777777-7777-4777-8777-777777777777"; + const grantRevision = 9; + const runCredential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + const otherRunCredential = `fzt_run_${Buffer.from(otherRunId).toString("base64url")}_${"S".repeat(43)}`; + const activeRun = { vmIdentity, integrationIds: [integrationId], grantRevision }; + let selectedCredential = "initial-scoped-bearer"; + let observedAuthorizations = []; + let snapshotReads = 0; + const upstream = createHttpsServer({ key, cert }, (request, response) => { + observedAuthorizations.push(request.headers.authorization); + response.end("scoped-e2e-ok"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = upstream.address().port; + const origin = `https://api.example.test:${upstreamPort}`; + const forwarder = createBearerForwarder({ connectorFactory: configuredOrigin => async requestUrl => { + assert.equal(configuredOrigin, origin); + const url = new URL(requestUrl); + const socket = tlsConnect({ host: "127.0.0.1", port: upstreamPort, servername: url.hostname, ca: cert }); + await once(socket, "secureConnect"); + return { socket, url }; + } }); + const runtime = createProxyRuntime({ bootEpoch, listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }) }); + const credentials = createProxyRunCredentialStore(); + credentials.authorize({ runId, vmIdentity, credential: runCredential }); + credentials.authorize({ runId: otherRunId, vmIdentity: otherVmIdentity, credential: otherRunCredential }); + const recordFor = async credential => { + const raw = Buffer.from(unlockKey.slice("fzk1.".length), "base64url"); + const material = await webcrypto.subtle.importKey("raw", raw, "HKDF", false, ["deriveKey"]); + const aesKey = await webcrypto.subtle.deriveKey({ name: "HKDF", hash: "SHA-256", + salt: new TextEncoder().encode("factorize.unlock-key.v1"), + info: new TextEncoder().encode("factorize.record.aes-256-gcm.v1") }, material, + { name: "AES-GCM", length: 256 }, false, ["encrypt"]); + const nonce = webcrypto.getRandomValues(new Uint8Array(12)); + const aad = new TextEncoder().encode(JSON.stringify(["factorize-record-v1", ownerId, integrationId, "bearer", origin, 1])); + const ciphertext = await webcrypto.subtle.encrypt({ name: "AES-GCM", iv: nonce, additionalData: aad, tagLength: 128 }, aesKey, new TextEncoder().encode(credential)); + return { id: integrationId, kind: "bearer", name: "Metrics", description: "current scoped grant", origin, revision: 1, + deletedAt: null, encryptedPayload: `${Buffer.from(nonce).toString("base64")}.${Buffer.from(ciphertext).toString("base64")}` }; + }; + const record = await recordFor(selectedCredential); + const fetchControl = async (url, init) => { + const path = new URL(url).pathname; + assert.equal(new Headers(init.headers).get("authorization"), `Bearer ${controlCredential}`); + const body = JSON.parse(init.body); + if (path.endsWith("/enroll")) { + assert.deepEqual(activeRun.integrationIds, [integrationId], "Factorize run snapshot contains only the selected integration"); + assert.equal(body.runId, runId); + assert.equal(body.vmIdentity, vmIdentity); + assert.equal(body.runCredential, runCredential); + return Response.json({ enrollmentId: "88888888-8888-4888-8888-888888888888", token: "e".repeat(43), grantRevision }); + } + if (path.endsWith("/consume-enrollment")) { + assert.equal(body.runId, runId); + assert.equal(body.vmIdentity, vmIdentity); + return Response.json({ consumed: true, binding: { tenantId: ownerId, runId, vmIdentity, + proxyIdentity: "proxy-fixture", bootEpoch, enrollmentId: "88888888-8888-4888-8888-888888888888", grantRevision } }); + } + if (path.endsWith("/integration-snapshot")) { + assert.deepEqual(body, { runId, grantRevision }); + assert.equal(activeRun.vmIdentity, vmIdentity); + assert.deepEqual(activeRun.integrationIds, [integrationId]); + snapshotReads++; + return Response.json({ ownerId, records: activeRun.integrationIds.includes(record.id) ? [record] : [], modelAccount: null }); + } + return Response.json({ error: "not found" }, { status: 404 }); + }; + const resolveGrants = createProxyIntegrationGrantResolver({ runtime, controlOrigin: "https://factorize.example", + controlCredential, fetchImpl: fetchControl }); + const tunnel = await startAuthenticatedProxyTunnel({ + tls: { key, cert: proxyCert, ca: cert }, runtime, + controlOrigin: "https://factorize.example", controlCredential, + resolveGrants, + resolveRunCredential: (id, identity) => credentials.resolve(id, identity), + fetchImpl: fetchControl, + }); + const helperPort = await freePort(); + const helper = await startVmTunnelHelper({ + tls: { key, cert, ca: cert }, runId, + proxyHost: "127.0.0.1", proxyPort: tunnel.address().port, port: helperPort, + wait: milliseconds => new Promise(resolve => setTimeout(resolve, Math.min(milliseconds, 15))), + }); + const otherPort = await freePort(); + const otherHelper = await startVmTunnelHelper({ + tls: { key, cert, ca: cert }, runId: otherRunId, + proxyHost: "127.0.0.1", proxyPort: tunnel.address().port, port: otherPort, + wait: milliseconds => new Promise(resolve => setTimeout(resolve, Math.min(milliseconds, 15))), + }); + try { + const denied = fetch(`http://127.0.0.1:${otherPort}/integrations/${integrationId}/echo`, { signal: AbortSignal.timeout(250) }); + await assert.rejects(denied, /aborted|timeout|fetch failed/i); + assert.equal(snapshotReads, 0, "a credential bound to another VM cannot reach the scoped snapshot"); + + const waitingWhileLocked = fetch(`http://127.0.0.1:${helperPort}/integrations/${integrationId}/echo`); + await new Promise(resolve => setTimeout(resolve, 80)); + assert.equal(runtime.state, "locked"); + assert.equal(snapshotReads, 0, "locked proxy denies selected Factorize grants before decrypting a snapshot"); + assert.deepEqual(observedAuthorizations, [], "locked proxy does not forward to the upstream"); + + runtime.unlock(unlockKey); + const first = await waitingWhileLocked; + assert.equal(first.status, 200); + assert.equal(await first.text(), "scoped-e2e-ok"); + assert.deepEqual(observedAuthorizations, ["Bearer initial-scoped-bearer"]); + const readsBeforeReconnect = snapshotReads; + assert.ok(readsBeforeReconnect >= 1, "accepted scoped tunnel reads the current Factorize snapshot"); + + await runtime.lock(); + selectedCredential = "updated-current-scoped-bearer"; + Object.assign(record, await recordFor(selectedCredential)); + runtime.unlock(unlockKey); + await waitForProxyUnlock(helperPort); + const reconnected = await fetch(`http://127.0.0.1:${helperPort}/integrations/${integrationId}/after-reconnect`); + assert.equal(reconnected.status, 200); + assert.equal(await reconnected.text(), "scoped-e2e-ok"); + assert.deepEqual(observedAuthorizations, ["Bearer initial-scoped-bearer", "Bearer updated-current-scoped-bearer"]); + assert.ok(snapshotReads > readsBeforeReconnect, "reconnect performs a fresh scoped snapshot read"); + } finally { + await helper.close(); + await otherHelper.close(); + await tunnel.close(); + credentials.clear(); + await runtime.shutdown(); + await new Promise((resolve, reject) => upstream.close(error => error ? reject(error) : resolve())); + } +}); diff --git a/packages/credential-proxy/test/model-account-crypto.test.mjs b/packages/credential-proxy/test/model-account-crypto.test.mjs new file mode 100644 index 00000000..2308a6df --- /dev/null +++ b/packages/credential-proxy/test/model-account-crypto.test.mjs @@ -0,0 +1,37 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import { decryptModelAccountCache, encryptModelAccountCache } from "../src/model-account-crypto.mjs"; + +const ownerId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; +const accountId = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"; +const otherAccountId = "cccccccc-cccc-4ccc-8ccc-cccccccccccc"; +const keyGeneration = "dddddddd-dddd-4ddd-8ddd-dddddddddddd"; +const unlockKey = `fzk1.${randomBytes(32).toString("base64url")}`; +const binding = { ownerId, accountId, keyGeneration, revision: 7 }; + +test("model account cache round-trips only for its complete owner/account/generation/revision binding", async () => { + const plaintext = new TextEncoder().encode('{"access_token":"synthetic-model-token"}'); + const encrypted = await encryptModelAccountCache(plaintext, unlockKey, binding); + assert.equal(encrypted.includes("synthetic-model-token"), false); + const restored = await decryptModelAccountCache(encrypted, unlockKey, binding); + assert.deepEqual(restored, plaintext); + restored.fill(0); + plaintext.fill(0); + + for (const changed of [ + { ...binding, ownerId: "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee" }, + { ...binding, accountId: otherAccountId }, + { ...binding, keyGeneration: "ffffffff-ffff-4fff-8fff-ffffffffffff" }, + { ...binding, revision: 8 }, + ]) await assert.rejects(decryptModelAccountCache(encrypted, unlockKey, changed)); + await assert.rejects(decryptModelAccountCache(encrypted, `fzk1.${randomBytes(32).toString("base64url")}`, binding)); +}); + +test("model account cache rejects malformed keys, ciphertext, and metadata", async () => { + const plaintext = new TextEncoder().encode("synthetic-cache"); + await assert.rejects(encryptModelAccountCache(plaintext, "not-an-unlock-key", binding), /malformed unlock key/); + await assert.rejects(encryptModelAccountCache(plaintext, unlockKey, { ...binding, revision: 0 }), /invalid model account encryption binding/); + await assert.rejects(decryptModelAccountCache("bad.payload.extra", unlockKey, binding), /malformed model account ciphertext/); + await assert.rejects(decryptModelAccountCache("AA==.AA==", unlockKey, binding), /malformed model account ciphertext/); +}); diff --git a/packages/credential-proxy/test/model-account-lifecycle.test.mjs b/packages/credential-proxy/test/model-account-lifecycle.test.mjs new file mode 100644 index 00000000..310b4dfb --- /dev/null +++ b/packages/credential-proxy/test/model-account-lifecycle.test.mjs @@ -0,0 +1,332 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createModelAccountLifecycle } from "../src/model-account-lifecycle.mjs"; + +const accountId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; +const resetAccountId = "cccccccc-cccc-4ccc-8ccc-cccccccccccc"; +const account = { id: accountId, kind: "model", ownerId: "owner", keyGeneration: "generation", revision: 4 }; + +async function fixture(t, overrides = {}) { + const root = await mkdtemp(join(tmpdir(), "factorize-model-account-")); + t.after(() => rm(root, { recursive: true, force: true })); + const home = join(root, accountId); + await mkdir(home, { recursive: true }); + await writeFile(join(home, "auth.json"), JSON.stringify({ tokens: { access_token: "synthetic-access", refresh_token: "synthetic-refresh" } })); + const calls = [], events = [], methods = []; + const helper = { invoke: async request => { + methods.push(request.method); + if (request.method === "device-login") return { state: "pending", completion: overrides.completion ?? Promise.resolve({ state: "authorized" }) }; + if (request.method === "refresh") { calls.push("refresh"); return overrides.refreshResult ?? { state: "refreshed" }; } + if (request.method === "account-status") return { state: "authenticated" }; + return { state: request.method === "cancel" ? "canceled" : "logged-out" }; + } }; + let saves = 0; + const lifecycle = createModelAccountLifecycle({ + authHelper: helper, + accountHomeRoot: root, + getUnlockKey: async () => "synthetic-unlock-key", + getAccount: async () => overrides.getAccount ? overrides.getAccount() : account, + encryptCache: async (plain, key, binding) => { + assert.equal(key, "synthetic-unlock-key"); + events.push({ type: "encrypt", binding, plaintext: new TextDecoder().decode(plain) }); + return `encrypted.${Buffer.from(plain).toString("base64")}`; + }, + persistEncrypted: async request => { + saves++; + events.push({ type: "persist", request }); + if (overrides.persistEncrypted) return overrides.persistEncrypted(request, saves); + }, + onUnsaved: value => events.push({ type: "unsaved", ...value }), + retryDelaysMs: overrides.retryDelaysMs ?? [], + sleep: async () => {}, + ...overrides.options, + }); + return { root, home, lifecycle, events, methods, calls, get saves() { return saves; } }; +} + +test("device authorization persists only encrypted, account-bound cache after success", async t => { + const f = await fixture(t); + const login = await f.lifecycle.deviceLogin(accountId); + assert.equal(login.state, "pending"); + assert.deepEqual(await login.completion, { state: "saved" }); + const [encrypted, saved] = f.events.filter(event => event.type === "encrypt" || event.type === "persist"); + assert.equal(encrypted.binding.ownerId, "owner"); + assert.equal(encrypted.binding.recordType, "model"); + assert.equal(encrypted.binding.revision, 5); + assert.equal(saved.request.expectedRevision, 4); + assert.match(saved.request.encryptedPayload, /^encrypted\./); + assert.equal("unlockKey" in saved.request, false); +}); + +test("denied or expired device authorization does not persist token state", async t => { + const f = await fixture(t, { completion: Promise.resolve({ state: "reauthentication-required" }) }); + const login = await f.lifecycle.deviceLogin(accountId); + assert.deepEqual(await login.completion, { state: "reauthentication-required" }); + assert.equal(f.saves, 0); +}); + +test("expired refresh explicitly requires reauthentication without saving stale tokens", async t => { + const f = await fixture(t, { refreshResult: { state: "reauthentication-required" } }); + assert.deepEqual(await f.lifecycle.refresh(accountId), { state: "reauthentication-required" }); + assert.equal(f.saves, 0); +}); + +test("refresh keeps the working local cache and reports bounded encrypted-save failures", async t => { + const f = await fixture(t, { retryDelaysMs: [0, 0], persistEncrypted: async () => { throw new Error("fixture outage"); } }); + assert.deepEqual(await f.lifecycle.refresh(accountId), { state: "unsaved", attempts: 3 }); + assert.equal(f.lifecycle.isUnsaved(accountId), true); + assert.equal(f.calls.length, 1); + assert.equal(f.saves, 3); + assert.deepEqual(f.events.filter(event => event.type === "unsaved").map(event => event.unsaved), [true]); +}); + +test("refresh requests for one account serialize and use the latest revision", async t => { + let revision = 0, inFlight = 0, maximumInFlight = 0; + const f = await fixture(t, { + getAccount: () => ({ ...account, revision }), + persistEncrypted: async () => { + inFlight++; + maximumInFlight = Math.max(maximumInFlight, inFlight); + await new Promise(resolve => setImmediate(resolve)); + revision++; + inFlight--; + }, + }); + await Promise.all([f.lifecycle.refresh(accountId), f.lifecycle.refresh(accountId)]); + assert.equal(maximumInFlight, 1); + assert.equal(f.saves, 2); +}); + +test("refresh re-encrypts against a newer account revision after a concurrent edit", async t => { + let revision = 4; + const f = await fixture(t, { + getAccount: () => ({ ...account, revision }), + persistEncrypted: async request => { + if (request.expectedRevision !== revision || f.saves === 1) { + revision++; + const error = new Error("stale model account revision"); + error.code = "stale_revision"; + throw error; + } + revision++; + }, + }); + assert.deepEqual(await f.lifecycle.refresh(accountId), { state: "saved" }); + const writes = f.events.filter(event => event.type === "persist").map(event => event.request); + assert.deepEqual(writes.map(request => request.expectedRevision), [4, 5]); + assert.notEqual(writes[0].idempotencyKey, writes[1].idempotencyKey); + const encryptions = f.events.filter(event => event.type === "encrypt"); + assert.deepEqual(encryptions.map(event => event.binding.revision), [5, 6]); +}); + +test("refresh does not retry a stale cache write after the unlock-key generation rotates", async t => { + let generation = "generation"; + const f = await fixture(t, { + getAccount: () => ({ ...account, keyGeneration: generation }), + persistEncrypted: async () => { + generation = "rotated-generation"; + const error = new Error("stale model account revision"); + error.code = "stale_revision"; + throw error; + }, + }); + + await assert.rejects(f.lifecycle.refresh(accountId), /changed before token persistence/); + assert.equal(f.saves, 1); + assert.deepEqual(f.events.filter(event => event.type === "encrypt").map(event => event.binding.keyGeneration), ["generation"]); +}); + +test("refresh does not retry a transient cache write after the key generation rotates during backoff", async t => { + let generation = "generation"; + const f = await fixture(t, { + retryDelaysMs: [1], + getAccount: () => ({ ...account, keyGeneration: generation }), + options: { sleep: async () => { generation = "rotated-generation"; } }, + persistEncrypted: async () => { throw new Error("temporary persistence failure"); }, + }); + + await assert.rejects(f.lifecycle.refresh(accountId), /changed before token persistence/); + assert.equal(f.saves, 1); + assert.deepEqual(f.events.filter(event => event.type === "encrypt").map(event => event.binding.keyGeneration), ["generation"]); +}); + +test("deletion invalidates a retry and removes the account cache", async t => { + let release; + const waiting = new Promise(resolve => { release = resolve; }); + const f = await fixture(t, { + retryDelaysMs: [0, 0, 0], + persistEncrypted: async (_request, attempt) => { + if (attempt === 1) { await waiting; throw new Error("first save failed"); } + }, + }); + const refreshing = f.lifecycle.refresh(accountId); + while (!f.saves) await new Promise(resolve => setImmediate(resolve)); + const deleting = f.lifecycle.deleteAccount(accountId); + release(); + await assert.rejects(refreshing, /changed before token persistence/); + assert.deepEqual(await deleting, { state: "deleted" }); + await assert.rejects(readFile(join(f.home, "auth.json")), { code: "ENOENT" }); + assert.ok(f.methods.includes("logout")); +}); + +test("reset snapshot invalidates a refresh cache write already in flight", async t => { + let release; + let entered; + const waiting = new Promise(resolve => { release = resolve; }); + const started = new Promise(resolve => { entered = resolve; }); + const f = await fixture(t, { + persistEncrypted: async () => { + entered(); + await waiting; + const error = new Error("stale model account revision"); + error.code = "stale_revision"; + throw error; + }, + }); + + const refreshing = f.lifecycle.refresh(accountId); + await started; + const resetting = f.lifecycle.reconcileAll([]); + await new Promise(resolve => setImmediate(resolve)); + release(); + + await assert.rejects(refreshing, /changed before token persistence/); + assert.deepEqual(await resetting, { removed: [accountId], accounts: [] }); + assert.equal(f.saves, 1); + await assert.rejects(readFile(join(f.home, "auth.json")), { code: "ENOENT" }); + assert.ok(f.methods.includes("logout")); +}); + +test("deletion during refresh persistence backoff prevents a delayed retry", async t => { + let releaseBackoff; + let enteredBackoff; + const waiting = new Promise(resolve => { releaseBackoff = resolve; }); + const started = new Promise(resolve => { enteredBackoff = resolve; }); + const f = await fixture(t, { + retryDelaysMs: [1], + options: { sleep: async () => { enteredBackoff(); await waiting; } }, + persistEncrypted: async () => { throw new Error("temporary persistence failure"); }, + }); + + const refreshing = f.lifecycle.refresh(accountId); + await started; + const deleting = f.lifecycle.deleteAccount(accountId); + releaseBackoff(); + + await assert.rejects(refreshing, /changed before token persistence/); + assert.deepEqual(await deleting, { state: "deleted" }); + assert.equal(f.saves, 1); + await assert.rejects(readFile(join(f.home, "auth.json")), { code: "ENOENT" }); + assert.ok(f.methods.includes("logout")); +}); + +test("reset during refresh persistence backoff prevents a delayed retry and clears the cache", async t => { + let releaseBackoff; + let enteredBackoff; + let releaseSnapshot; + let enteredSnapshot; + const waiting = new Promise(resolve => { releaseBackoff = resolve; }); + const started = new Promise(resolve => { enteredBackoff = resolve; }); + const snapshotWaiting = new Promise(resolve => { releaseSnapshot = resolve; }); + const snapshotStarted = new Promise(resolve => { enteredSnapshot = resolve; }); + const f = await fixture(t, { + retryDelaysMs: [1], + options: { + sleep: async () => { enteredBackoff(); await waiting; }, + readDirectory: async () => { enteredSnapshot(); await snapshotWaiting; return [{ name: accountId }]; }, + }, + persistEncrypted: async () => { throw new Error("temporary persistence failure"); }, + }); + + const refreshing = f.lifecycle.refresh(accountId); + await started; + const resetting = f.lifecycle.reconcileAll([]); + await snapshotStarted; + releaseSnapshot(); + await new Promise(resolve => setImmediate(resolve)); + releaseBackoff(); + + await assert.rejects(refreshing, /changed before token persistence/); + assert.deepEqual(await resetting, { removed: [accountId], accounts: [] }); + assert.equal(f.saves, 1); + await assert.rejects(readFile(join(f.home, "auth.json")), { code: "ENOENT" }); + assert.ok(f.methods.includes("logout")); +}); + +test("deletion racing pending device login prevents a late authorization from restoring its cache", async t => { + let complete; + const f = await fixture(t, { completion: new Promise(resolve => { complete = resolve; }) }); + const login = await f.lifecycle.deviceLogin(accountId); + const deleting = f.lifecycle.deleteAccount(accountId); + complete({ state: "authorized" }); + assert.deepEqual(await login.completion, { state: "deleted" }); + assert.deepEqual(await deleting, { state: "deleted" }); + await assert.rejects(readFile(join(f.home, "auth.json")), { code: "ENOENT" }); + assert.equal(f.saves, 0); +}); + +test("unlock reconciliation restores the encrypted Codex cache before checking account status", async t => { + const persisted = new TextEncoder().encode(JSON.stringify({ tokens: { access_token: "restored-access", refresh_token: "restored-refresh" } })); + const encryptedPayload = `cache.${Buffer.from(persisted).toString("base64")}`; + const f = await fixture(t, { options: { + getEncryptedCache: async id => ({ + accountId: id, ownerId: "owner", keyGeneration: "generation", revision: 8, encryptedPayload, + }), + decryptCache: async (payload, key, binding) => { + assert.equal(payload, encryptedPayload); + assert.equal(key, "synthetic-unlock-key"); + assert.deepEqual(binding, { + ownerId: "owner", recordId: accountId, recordType: "model", destination: "factorize", + revision: 8, keyGeneration: "generation", + }); + return new Uint8Array(persisted); + }, + } }); + + assert.deepEqual(await f.lifecycle.reconcile(accountId), { state: "saved" }); + assert.deepEqual(JSON.parse(await readFile(join(f.home, "auth.json"), "utf8")), { + tokens: { access_token: "restored-access", refresh_token: "restored-refresh" }, + }); + assert.ok(f.methods.indexOf("account-status") > -1); + assert.equal(f.events.find(event => event.type === "persist").request.expectedRevision, 4); +}); + +test("unlock reconciliation removes stale cache when the account key generation changed", async t => { + const f = await fixture(t, { options: { + getEncryptedCache: async id => ({ + accountId: id, ownerId: "owner", keyGeneration: "old-generation", revision: 8, encryptedPayload: "cache.payload", + }), + decryptCache: async () => { throw new Error("must not decrypt stale cache"); }, + } }); + + assert.deepEqual(await f.lifecycle.reconcile(accountId), { state: "reauthentication-required" }); + await assert.rejects(readFile(join(f.home, "auth.json")), { code: "ENOENT" }); + assert.equal(f.methods.includes("account-status"), false); +}); + +test("unlock snapshot reconciliation logs out and removes cached accounts deleted by reset", async t => { + const f = await fixture(t); + const staleHome = join(f.root, resetAccountId); + await mkdir(staleHome, { recursive: true }); + await writeFile(join(staleHome, "auth.json"), JSON.stringify({ tokens: { access_token: "stale-reset-token" } })); + + assert.deepEqual(await f.lifecycle.reconcileAll([account]), { + removed: [resetAccountId], + accounts: [{ accountId, state: "saved" }], + }); + await assert.rejects(readFile(join(staleHome, "auth.json")), { code: "ENOENT" }); + assert.ok(f.methods.includes("logout")); + assert.equal(f.saves, 1); +}); + +test("unlock snapshot rejects duplicate account identities before touching local state", async t => { + const f = await fixture(t); + await assert.rejects(f.lifecycle.reconcileAll([account, account]), /snapshot is invalid/); + assert.deepEqual(JSON.parse(await readFile(join(f.home, "auth.json"), "utf8")), { + tokens: { access_token: "synthetic-access", refresh_token: "synthetic-refresh" }, + }); + assert.equal(f.methods.length, 0); +}); diff --git a/packages/credential-proxy/test/proxy-listener.e2e.test.mjs b/packages/credential-proxy/test/proxy-listener.e2e.test.mjs new file mode 100644 index 00000000..f5a18a6d --- /dev/null +++ b/packages/credential-proxy/test/proxy-listener.e2e.test.mjs @@ -0,0 +1,609 @@ +import assert from "node:assert/strict"; +import { readFile } from "node:fs/promises"; +import { createServer as createHttpsServer } from "node:https"; +import { once } from "node:events"; +import { connect as tlsConnect } from "node:tls"; +import { test } from "node:test"; +import { createBearerForwarder } from "../src/bearer-forwarder.mjs"; +import { createAuthenticatedProxyListener } from "../src/proxy-listener.mjs"; +import { createProxyRuntime } from "../src/runtime.mjs"; +import { createProxyRunCredentialStore } from "../src/run-credential-store.mjs"; +import { createOriginConnector } from "../src/destination-policy.mjs"; +import { createAuthenticatedTunnelServer, peerIdentity, startAuthenticatedProxyTunnel, startVmTunnelHelper } from "../src/tunnel-transport.mjs"; + +const fixtures = new URL("./fixtures/", import.meta.url); +const key = await readFile(new URL("upstream-key.pem", fixtures)); +const cert = await readFile(new URL("upstream-cert.pem", fixtures)); + +async function waitForEvent(emitter, event) { + const pending = once(emitter, event); + return pending; +} + +test("mTLS VM identity comes from one canonical URI SAN bound to the certificate CN", () => { + const identity = "api.example.test"; + const encoded = Buffer.from(identity).toString("base64url"); + const socket = (subjectaltname, commonName = identity, authorized = true) => ({ + authorized, + getPeerCertificate: () => ({ subject: { CN: commonName }, subjectaltname }), + }); + assert.equal(peerIdentity(socket(`DNS:api.example.test, URI:urn:factorize:vm:${encoded}`)), identity); + assert.throws(() => peerIdentity(socket("DNS:api.example.test")), /one VM identity URI/); + assert.throws(() => peerIdentity(socket(`URI:urn:factorize:vm:${encoded}, URI:urn:factorize:vm:${encoded}`)), /one VM identity URI/); + assert.throws(() => peerIdentity(socket("URI:urn:factorize:vm:not_canonical=")), /invalid VM identity URI/); + assert.throws(() => peerIdentity(socket(`URI:urn:factorize:vm:${encoded}`, "other-vm")), /inconsistent/); + assert.throws(() => peerIdentity(socket(`URI:urn:factorize:vm:${encoded}`, identity, false)), /not authorized/); +}); + +test("authenticated tunnel listener streams over TLS, confines grants and cancels upstream work", { timeout: 5_000 }, async () => { + let observedAuthorization; + let upstreamEnded = false; + let cancelUpstream; + let finishUpstream; + let cancelStartedResolve; + const cancelStarted = new Promise(resolve => { cancelStartedResolve = resolve; }); + const upstream = createHttpsServer({ key, cert }, async (request, response) => { + observedAuthorization = request.headers.authorization; + if (request.url === "/cancel") { + const closed = new Promise(resolve => request.socket.once("close", resolve)); + response.writeHead(200, { "content-type": "text/plain" }); + response.write("first chunk"); + cancelUpstream = closed; + cancelStartedResolve(); + return; + } + const received = []; + for await (const chunk of request) received.push(chunk); + assert.equal(Buffer.concat(received).toString(), "request-body"); + response.writeHead(200, { "content-type": "text/plain" }); + response.write("first chunk"); + response.once("close", () => { upstreamEnded = true; }); + await new Promise(resolve => { finishUpstream = resolve; }); + response.end(" second chunk"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = upstream.address().port; + const forwarder = createBearerForwarder({ + connectorFactory: origin => async requestUrl => { + assert.equal(origin, `https://api.example.test:${upstreamPort}`); + const url = new URL(requestUrl); + const socket = tlsConnect({ host: "127.0.0.1", port: upstreamPort, servername: url.hostname, ca: cert }); + await once(socket, "secureConnect"); + return { socket, url }; + }, + }); + const runtime = createProxyRuntime({ + listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }), + }); + let listener; + try { + runtime.unlock("synthetic-unlock-key"); + listener = await runtime.acceptAuthenticatedConnection({ + authenticated: true, + enrollmentConsumed: true, + bootEpoch: runtime.bootEpoch, + connectionId: "consumed-enrollment-1", + runId: "run-1", + vmId: "vm-1", + grants: [{ id: "tool-1", origin: `https://api.example.test:${upstreamPort}`, credential: "synthetic-bearer" }], + }); + const address = listener.address; + assert.equal(address.address, "127.0.0.1"); + const base = `http://127.0.0.1:${address.port}`; + const response = await fetch(`${base}/integrations/tool-1/echo?source=vm`, { + method: "POST", + headers: { authorization: "Bearer caller-token", "content-type": "text/plain" }, + body: "request-body", + }); + assert.equal(response.status, 200); + assert.equal(observedAuthorization, "Bearer synthetic-bearer"); + const reader = response.body.getReader(); + const first = await reader.read(); + assert.equal(Buffer.from(first.value).toString(), "first chunk"); + assert.equal(upstreamEnded, false, "response was delivered before the upstream finished"); + assert.equal(forwarder.activeRequests, 1, "streaming response retains its concurrency slot"); + finishUpstream(); + const second = await reader.read(); + assert.equal(Buffer.from(second.value).toString(), " second chunk"); + assert.equal((await reader.read()).done, true); + assert.equal(forwarder.activeRequests, 0); + + const denied = await fetch(`${base}/integrations/other-tool/echo`); + assert.equal(denied.status, 403); + assert.equal(observedAuthorization, "Bearer synthetic-bearer", "ungranted request never reached the upstream"); + + const abort = new AbortController(); + const cancelled = fetch(`${base}/integrations/tool-1/cancel`, { signal: abort.signal }); + await cancelStarted; + abort.abort(); + await assert.rejects(cancelled, error => error.name === "AbortError"); + assert.ok(cancelUpstream, "upstream cancellation hook was installed"); + await Promise.race([cancelUpstream, new Promise((_, reject) => setTimeout(() => reject(new Error("upstream was not cancelled")), 1_000))]); + assert.equal(forwarder.activeRequests, 0); + } finally { + finishUpstream?.(); + if (listener) await runtime.disconnect("consumed-enrollment-1"); + await runtime.shutdown(); + await new Promise((resolve, reject) => upstream.close(error => error ? reject(error) : resolve())); + } +}); + +test("reflection exposes safe metadata for only the authenticated VM connection and tracks live grant changes", { timeout: 5_000 }, async () => { + const runtime = createProxyRuntime(); + const connections = []; + let first; + let second; + let firstLiveGrants; + try { + runtime.unlock("synthetic-unlock-key"); + firstLiveGrants = [ + { id: "selected-1", name: "Billing API", description: "Billing help", origin: "https://billing.example", credential: "secret-first" }, + { id: "selected-2", name: "Metrics API", description: null, origin: "https://metrics.example", credential: "secret-second" }, + ]; + first = await runtime.acceptAuthenticatedConnection({ + authenticated: true, enrollmentConsumed: true, bootEpoch: runtime.bootEpoch, + connectionId: "reflection-connection-1", runId: "run-1", vmId: "vm-1", + grants: firstLiveGrants, + refreshGrants: async () => firstLiveGrants, + }); + second = await runtime.acceptAuthenticatedConnection({ + authenticated: true, enrollmentConsumed: true, bootEpoch: runtime.bootEpoch, + connectionId: "reflection-connection-2", runId: "run-2", vmId: "vm-2", + grants: [{ id: "other-vm-only", name: "Private API", description: "other VM", origin: "https://private.example", credential: "secret-other-vm" }], + }); + connections.push(first, second); + + const reflect = async listener => { + const address = listener.address; + assert.equal(address.address, "127.0.0.1"); + const response = await fetch(`http://127.0.0.1:${address.port}/integrations`); + assert.equal(response.status, 200); + assert.equal(response.headers.get("cache-control"), "no-store"); + return response.json(); + }; + const firstReflection = await reflect(first); + assert.equal(firstReflection.state, "unlocked"); + assert.deepEqual(firstReflection.integrations, [ + { id: "selected-1", source: "portable", name: "Billing API", description: "Billing help", endpoint: "/integrations/selected-1/", example: 'curl "$FACTORIZE_PROXY_URL/integrations/selected-1/"', availability: "available" }, + { id: "selected-2", source: "portable", name: "Metrics API", description: null, endpoint: "/integrations/selected-2/", example: 'curl "$FACTORIZE_PROXY_URL/integrations/selected-2/"', availability: "available" }, + ]); + assert.equal(JSON.stringify(firstReflection).includes("secret-"), false); + assert.equal(JSON.stringify(firstReflection).includes("https://billing.example"), false); + + const secondReflection = await reflect(second); + assert.deepEqual(secondReflection.integrations.map(item => item.id), ["other-vm-only"]); + assert.equal(JSON.stringify(secondReflection).includes("selected-1"), false); + + first.replaceGrants([{ id: "selected-2", name: "Metrics API", description: "current help", origin: "https://metrics.example", credential: "replacement-secret" }]); + const refreshed = await reflect(first); + assert.deepEqual(refreshed.integrations.map(item => [item.id, item.description]), [["selected-2", "current help"]]); + assert.equal(JSON.stringify(refreshed).includes("replacement-secret"), false); + + await runtime.lock(); + const locked = await reflect(first); + assert.deepEqual(locked, { state: "locked", integrations: [ + { id: "selected-2", source: "portable", name: "Metrics API", description: "current help", endpoint: "/integrations/selected-2/", example: 'curl "$FACTORIZE_PROXY_URL/integrations/selected-2/"', availability: "locked" }, + ] }); + assert.equal((await fetch(`http://127.0.0.1:${first.address.port}/integrations/selected-2/`)).status, 423); + firstLiveGrants = [{ id: "selected-2", name: "Metrics API", description: "after unlock", origin: "https://metrics.example", credential: "fresh-secret" }]; + await runtime.unlockAndPrepare("synthetic-unlock-key", async () => {}); + const unlocked = await reflect(first); + assert.equal(unlocked.state, "unlocked"); + assert.deepEqual(unlocked.integrations.map(item => [item.id, item.description]), [["selected-2", "after unlock"]]); + assert.equal(JSON.stringify(unlocked).includes("fresh-secret"), false); + + const methodDenied = await fetch(`http://127.0.0.1:${first.address.port}/integrations`, { method: "POST" }); + assert.equal(methodDenied.status, 405); + } finally { + await runtime.shutdown(); + } +}); + +test("live integration edits replace credentials and deletion removes access from an established grant", { timeout: 5_000 }, async () => { + const observed = []; + const upstream = createHttpsServer({ key, cert }, (request, response) => { + observed.push(request.headers.authorization); + response.writeHead(200, { "content-type": "text/plain" }); + response.end("ok"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const port = upstream.address().port; + const forwarder = createBearerForwarder({ connectorFactory: origin => async requestUrl => { + assert.equal(origin, `https://api.example.test:${port}`); + const url = new URL(requestUrl); + const socket = tlsConnect({ host: "127.0.0.1", port, servername: url.hostname, ca: cert }); + await once(socket, "secureConnect"); + return { socket, url }; + } }); + let grants = [{ id: "tool-1", origin: `https://api.example.test:${port}`, credential: "first-secret" }]; + const runtime = createProxyRuntime({ + grantRefreshIntervalMs: 100, + listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }), + }); + let listener; + try { + runtime.unlock("synthetic-unlock-key"); + listener = await runtime.acceptAuthenticatedConnection({ + authenticated: true, enrollmentConsumed: true, bootEpoch: runtime.bootEpoch, + connectionId: "live-integration-connection", runId: "live-integration-run", vmId: "live-integration-vm", + grants, refreshGrants: async () => grants, + }); + const base = `http://127.0.0.1:${listener.address.port}`; + assert.equal((await fetch(`${base}/integrations/tool-1/echo`)).status, 200); + grants = [{ id: "tool-1", origin: `https://api.example.test:${port}`, credential: "replacement-secret" }]; + await new Promise(resolve => setTimeout(resolve, 150)); + assert.equal((await fetch(`${base}/integrations/tool-1/echo`)).status, 200); + grants = []; + await new Promise(resolve => setTimeout(resolve, 150)); + assert.equal((await fetch(`${base}/integrations/tool-1/echo`)).status, 403); + assert.deepEqual(observed, ["Bearer first-secret", "Bearer replacement-secret"]); + } finally { + if (listener) await runtime.disconnect("live-integration-connection"); + await runtime.shutdown(); + await new Promise(resolve => upstream.close(resolve)); + } +}); + +test("listener refuses an unconsumed or unauthenticated connection", async () => { + await assert.rejects(createAuthenticatedProxyListener({ connection: { + authenticated: true, connectionId: "connection", runId: "run", vmId: "vm", grants: [], + } }), /authenticated consumed connection/); +}); + +test("runtime rejects a consumed enrollment from an earlier proxy boot", async () => { + const runtime = createProxyRuntime({ bootEpoch: "11111111-1111-4111-8111-111111111111" }); + runtime.unlock("synthetic-unlock-key"); + try { + await assert.rejects(runtime.acceptAuthenticatedConnection({ + authenticated: true, + enrollmentConsumed: true, + bootEpoch: "22222222-2222-4222-8222-222222222222", + connectionId: "stale-enrollment", + runId: "run-1", + vmId: "vm-1", + grants: [], + }), /different proxy boot/); + } finally { + await runtime.shutdown(); + } +}); + +test("VM helper uses mutual TLS and consumed enrollment to reach only its granted integration listener", { timeout: 5_000 }, async () => { + let observedAuthorization; + let enrollmentCount = 0; + let authenticationCount = 0; + let mismatchReachedResolve; + const mismatchReached = new Promise(resolve => { mismatchReachedResolve = resolve; }); + const upstream = createHttpsServer({ key, cert }, (request, response) => { + observedAuthorization = request.headers.authorization; + response.end("transport-ok"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = upstream.address().port; + const forwarder = createBearerForwarder({ + connectorFactory: origin => async requestUrl => { + assert.equal(origin, `https://api.example.test:${upstreamPort}`); + const url = new URL(requestUrl); + const socket = tlsConnect({ host: "127.0.0.1", port: upstreamPort, servername: url.hostname, ca: cert }); + await once(socket, "secureConnect"); + return { socket, url }; + }, + }); + const runtime = createProxyRuntime({ listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }) }); + const tunnel = createAuthenticatedTunnelServer({ + tls: { key, cert, ca: cert }, + runtime, + authenticateEnrollment: async ({ token, vmIdentity, bootEpoch }) => { + authenticationCount++; + assert.ok(["A", "B", "C", "X"].some(value => token === value.repeat(43))); + assert.equal(vmIdentity, "api.example.test"); + assert.equal(bootEpoch, runtime.bootEpoch); + if (token === "A".repeat(43)) throw new Error("temporary control-plane outage"); + if (token === "X".repeat(43)) mismatchReachedResolve(); + return { + authenticated: true, + enrollmentConsumed: true, + vmIdentity: token === "X".repeat(43) ? "other-vm" : vmIdentity, + bootEpoch, + connectionId: "transport-enrollment-1", + runId: "run-transport-1", + vmId: vmIdentity, + grants: [{ id: "tool-1", origin: `https://api.example.test:${upstreamPort}`, credential: "synthetic-bearer" }], + }; + }, + }); + await tunnel.listening; + runtime.unlock("synthetic-unlock-key"); + const local = await startVmTunnelHelper({ + tls: { key, cert, ca: cert, servername: "api.example.test" }, + enrollment: async () => ({ token: ["A", "B", "C"][enrollmentCount++ % 3].repeat(43) }), + proxyHost: "127.0.0.1", + proxyPort: tunnel.address().port, + wait: async () => {}, + }); + let mismatchLocal; + try { + const response = await fetch(`http://127.0.0.1:${local.address.port}/integrations/tool-1/echo`, { + headers: { authorization: "Bearer caller-token" }, + }); + assert.equal(response.status, 200); + assert.equal(await response.text(), "transport-ok"); + assert.equal(observedAuthorization, "Bearer synthetic-bearer"); + const denied = await fetch(`http://127.0.0.1:${local.address.port}/integrations/other-tool/echo`); + assert.equal(denied.status, 403); + assert.ok(enrollmentCount >= 3, "failed setup and local connections get fresh one-use enrollment"); + assert.equal(enrollmentCount, authenticationCount, "each enrollment is independently consumed by the proxy transport"); + mismatchLocal = await startVmTunnelHelper({ + tls: { key, cert, ca: cert, servername: "api.example.test" }, + enrollment: async () => ({ token: (enrollmentCount++, "X".repeat(43)) }), + proxyHost: "127.0.0.1", + proxyPort: tunnel.address().port, + }); + const rejectedRequest = fetch(`http://127.0.0.1:${mismatchLocal.address.port}/integrations/tool-1/echo`); + await mismatchReached; + await mismatchLocal.close(); + mismatchLocal = undefined; + await assert.rejects(rejectedRequest, /fetch failed/, "mismatched VM binding is rejected"); + assert.equal(authenticationCount, enrollmentCount); + } finally { + await mismatchLocal?.close(); + await local.close(); + await tunnel.close(); + await runtime.shutdown(); + await new Promise((resolve, reject) => upstream.close(error => error ? reject(error) : resolve())); + } +}); + +test("VM helper authenticates the active run while the proxy performs authenticated enrollment", { timeout: 5_000 }, async () => { + const runtime = createProxyRuntime(); + const runCredentials = createProxyRunCredentialStore(); + const requests = []; + let enrollmentSequence = 0; + const runId = "33333333-3333-4333-8333-333333333333"; + const runCredential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + runCredentials.authorize({ runId, vmIdentity: "api.example.test", credential: runCredential }); + const tunnel = await startAuthenticatedProxyTunnel({ + tls: { key, cert, ca: cert }, + runtime, + controlOrigin: "https://factorize.example", + controlCredential: `fzc_${"c".repeat(43)}`, + resolveRunCredential: (id, vmIdentity) => runCredentials.resolve(id, vmIdentity), + fetchImpl: async (url, init) => { + const body = JSON.parse(init.body); + requests.push({ url: String(url), authorization: init.headers.Authorization, body }); + if (String(url).endsWith("/enroll")) { + enrollmentSequence++; + return Response.json({ enrollmentId: `11111111-1111-4111-8111-${String(enrollmentSequence).padStart(12, "0")}`, token: "e".repeat(43), grantRevision: 7 }); + } + const enrollmentId = `11111111-1111-4111-8111-${String(enrollmentSequence).padStart(12, "0")}`; + return Response.json({ consumed: true, binding: { + tenantId: "22222222-2222-4222-8222-222222222222", + runId, + vmIdentity: "api.example.test", + proxyIdentity: "proxy-fixture", + bootEpoch: runtime.bootEpoch, + enrollmentId, + grantRevision: 7, + } }); + }, + resolveGrants: async binding => { + assert.equal(binding.runId, runId); + assert.equal(binding.bootEpoch, runtime.bootEpoch); + return []; + }, + }); + runtime.unlock("synthetic-unlock-key"); + const local = await startVmTunnelHelper({ + tls: { key, cert, ca: cert, servername: "api.example.test" }, + runId, + proxyHost: "127.0.0.1", + proxyPort: tunnel.address().port, + }); + try { + for (let index = 0; index < 2; index++) { + const response = await fetch(`http://127.0.0.1:${local.address.port}/unrecognized`, { headers: { connection: "close" } }); + assert.equal(response.status, 404); + } + assert.deepEqual(requests.map(({ url, authorization, body }) => ({ + path: new URL(url).pathname, authorization, body, + })), [ + { path: "/internal/credential-proxy/enroll", authorization: `Bearer fzc_${"c".repeat(43)}`, body: { runId, vmIdentity: "api.example.test", runCredential } }, + { path: "/internal/credential-proxy/consume-enrollment", authorization: `Bearer fzc_${"c".repeat(43)}`, body: { runId, vmIdentity: "api.example.test", enrollmentToken: "e".repeat(43) } }, + { path: "/internal/credential-proxy/enroll", authorization: `Bearer fzc_${"c".repeat(43)}`, body: { runId, vmIdentity: "api.example.test", runCredential } }, + { path: "/internal/credential-proxy/consume-enrollment", authorization: `Bearer fzc_${"c".repeat(43)}`, body: { runId, vmIdentity: "api.example.test", enrollmentToken: "e".repeat(43) } }, + ]); + } finally { + await local.close(); + await tunnel.close(); + await runtime.shutdown(); + runCredentials.clear(); + } +}); + +test("authenticated reconnect receives current grants after live changes and lock/unlock", { timeout: 10_000 }, async () => { + let currentGrants = [{ id: "tool-1", origin: "https://api.example.test", credential: "initial-grant" }]; + const observedAuthorization = []; + let authentications = 0; + const upstream = createHttpsServer({ key, cert }, (_request, response) => { + observedAuthorization.push(_request.headers.authorization); + response.end("current-grant"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = upstream.address().port; + const forwarder = createBearerForwarder({ + connectorFactory: configuredOrigin => async requestUrl => { + assert.equal(configuredOrigin, `https://api.example.test:${upstreamPort}`); + const url = new URL(requestUrl); + const socket = tlsConnect({ host: "127.0.0.1", port: upstreamPort, servername: url.hostname, ca: cert }); + await once(socket, "secureConnect"); + return { socket, url }; + }, + }); + const runtime = createProxyRuntime({ + listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }), + }); + const tunnel = createAuthenticatedTunnelServer({ + tls: { key, cert, ca: cert }, + runtime, + authenticateRun: async ({ runId, vmIdentity, bootEpoch }) => { + authentications++; + if (runtime.state !== "unlocked") throw new Error("proxy is locked"); + return { + authenticated: true, + enrollmentConsumed: true, + vmIdentity, + bootEpoch, + connectionId: `reconnect-${authentications}`, + runId, + vmId: vmIdentity, + grants: currentGrants.map(grant => ({ ...grant, origin: `https://api.example.test:${upstreamPort}` })), + }; + }, + }); + await tunnel.listening; + runtime.unlock("synthetic-unlock-key"); + const local = await startVmTunnelHelper({ + tls: { key, cert, ca: cert, servername: "api.example.test" }, + runId: "33333333-3333-4333-8333-333333333333", + proxyHost: "127.0.0.1", + proxyPort: tunnel.address().port, + wait: milliseconds => new Promise(resolve => setTimeout(resolve, Math.min(milliseconds, 10))), + }); + const request = async id => { + const response = await fetch(`http://127.0.0.1:${local.address.port}/integrations/${id}/echo`, { + headers: { connection: "close" }, + }); + return { status: response.status, body: await response.text() }; + }; + try { + assert.deepEqual(await request("tool-1"), { status: 200, body: "current-grant" }); + assert.deepEqual(observedAuthorization, ["Bearer initial-grant"]); + + currentGrants = [{ id: "tool-2", origin: "https://api.example.test", credential: "live-update" }]; + assert.deepEqual(await request("tool-1"), { status: 403, body: "" }, "a new authenticated tunnel cannot retain an old grant"); + assert.deepEqual(await request("tool-2"), { status: 200, body: "current-grant" }); + assert.deepEqual(observedAuthorization, ["Bearer initial-grant", "Bearer live-update"]); + + await runtime.lock(); + currentGrants = [{ id: "tool-3", origin: "https://api.example.test", credential: "after-unlock" }]; + const pending = request("tool-3"); + await new Promise(resolve => setTimeout(resolve, 40)); + assert.ok(authentications >= 5, "the helper retries with fresh authenticated connections while locked"); + runtime.unlock("synthetic-unlock-key"); + assert.deepEqual(await pending, { status: 200, body: "current-grant" }); + assert.deepEqual(observedAuthorization, ["Bearer initial-grant", "Bearer live-update", "Bearer after-unlock"]); + } finally { + await local.close(); + await tunnel.close(); + await runtime.shutdown(); + await new Promise((resolve, reject) => upstream.close(error => error ? reject(error) : resolve())); + } +}); + +test("TLS forwarding keeps the configured origin through redirects and never replays an interrupted write", { timeout: 10_000 }, async () => { + let requests = 0; + let writeStartedResolve; + const writeStarted = new Promise(resolve => { writeStartedResolve = resolve; }); + const upstream = createHttpsServer({ key, cert }, async (request, response) => { + requests++; + if (request.url === "/redirect") { + response.writeHead(302, { location: "https://other.example.test/should-not-be-requested" }).end(); + return; + } + if (request.url === "/write") { + writeStartedResolve(); + request.resume(); + await new Promise(resolve => request.socket.once("close", resolve)); + return; + } + response.end("bound-origin"); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = upstream.address().port; + const origin = `https://api.example.test:${upstreamPort}`; + const forwarder = createBearerForwarder({ + connectorFactory: configuredOrigin => createOriginConnector(configuredOrigin, { + resolve: async hostname => { + assert.equal(hostname, "api.example.test"); + return [{ address: "93.184.216.34", family: 4 }]; + }, + connect: async ({ hostname, port, address }) => { + const socket = tlsConnect({ host: "127.0.0.1", port, servername: hostname, ca: cert }); + await once(socket, "secureConnect"); + // The fixture maps a validated public test address to this local TLS upstream. + Object.defineProperty(socket, "remoteAddress", { value: address }); + return socket; + }, + }), + }); + const runtime = createProxyRuntime({ + listenerFactory: ({ connection }) => createAuthenticatedProxyListener({ connection, forwarder }), + }); + runtime.unlock("synthetic-unlock-key"); + const listener = await runtime.acceptAuthenticatedConnection({ + authenticated: true, + enrollmentConsumed: true, + bootEpoch: runtime.bootEpoch, + connectionId: "destination-boundary-connection", + runId: "destination-boundary-run", + vmId: "destination-boundary-vm", + grants: [{ id: "tool-1", origin, credential: "synthetic-bearer" }], + }); + const tunnel = createAuthenticatedTunnelServer({ + tls: { key, cert, ca: cert }, + runtime, + authenticateEnrollment: async ({ token, vmIdentity, bootEpoch }) => ({ + authenticated: true, + enrollmentConsumed: true, + vmIdentity, + bootEpoch, + connectionId: `destination-${token.slice(0, 4)}`, + runId: "destination-boundary-run", + vmId: vmIdentity, + grants: [{ id: "tool-1", origin, credential: "synthetic-bearer" }], + }), + }); + await tunnel.listening; + const local = await startVmTunnelHelper({ + tls: { key, cert, ca: cert, servername: "api.example.test" }, + enrollment: async () => ({ token: "E".repeat(43) }), + proxyHost: "127.0.0.1", + proxyPort: tunnel.address().port, + wait: async () => {}, + }); + const base = `http://127.0.0.1:${local.address.port}`; + try { + const ok = await fetch(`${base}/integrations/tool-1/safe/path?next=https%3A%2F%2Fother.example.test`, { headers: { connection: "close" } }); + assert.equal(ok.status, 200); + assert.equal(await ok.text(), "bound-origin"); + + const redirected = await fetch(`${base}/integrations/tool-1/redirect`, { headers: { connection: "close" } }); + assert.equal(redirected.status, 502, "a redirect response is surfaced as an upstream failure"); + assert.equal(requests, 2, "the redirect target is never contacted"); + + const interrupted = fetch(`${base}/integrations/tool-1/write`, { + method: "POST", + headers: { connection: "close", "content-type": "text/plain" }, + body: "single-write", + }); + await writeStarted; + await runtime.lock(); + const interruptedResponse = await interrupted; + assert.equal(interruptedResponse.status, 499, "the interrupted operation is reported as cancelled"); + assert.equal(await interruptedResponse.text(), "request cancelled"); + await new Promise(resolve => setTimeout(resolve, 50)); + assert.equal(requests, 3, "the interrupted POST was sent once and never replayed"); + assert.equal(forwarder.activeRequests, 0); + } finally { + await local.close(); + await tunnel.close(); + await runtime.disconnect(listener.connectionId); + await runtime.shutdown(); + await new Promise((resolve, reject) => upstream.close(error => error ? reject(error) : resolve())); + } +}); diff --git a/packages/credential-proxy/test/run-credential-store.test.mjs b/packages/credential-proxy/test/run-credential-store.test.mjs new file mode 100644 index 00000000..8f89539c --- /dev/null +++ b/packages/credential-proxy/test/run-credential-store.test.mjs @@ -0,0 +1,34 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { createProxyRunCredentialStore } from "../src/run-credential-store.mjs"; + +const runId = "33333333-3333-4333-8333-333333333333"; +const vmIdentity = "fly:factorize-app:machine-1"; +const credential = `fzt_run_${Buffer.from(runId).toString("base64url")}_${"R".repeat(43)}`; + +test("provider delivery stores a credential only for its bound run and VM in process memory", () => { + const store = createProxyRunCredentialStore(); + assert.equal(store.authorize({ runId, vmIdentity, credential }), true); + assert.equal(store.resolve(runId, vmIdentity), credential); + assert.equal(store.resolve(runId, "fly:factorize-app:other-machine"), undefined); + assert.equal(store.resolve("44444444-4444-4444-8444-444444444444", vmIdentity), undefined); + store.clear(); + assert.equal(store.resolve(runId, vmIdentity), undefined); +}); + +test("revocation removes delivered credentials and permanently closes the run for this boot", () => { + const store = createProxyRunCredentialStore(); + store.authorize({ runId, vmIdentity, credential }); + store.revoke(runId); + assert.equal(store.resolve(runId, vmIdentity), undefined); + assert.throws(() => store.authorize({ runId, vmIdentity, credential }), /revoked/); +}); + +test("rejects malformed run, VM, and scoped credential inputs", () => { + const store = createProxyRunCredentialStore(); + for (const value of [ + { runId: "../other", vmIdentity, credential }, + { runId, vmIdentity: "bad\nidentity", credential }, + { runId, vmIdentity, credential: "fzt_run_invalid" }, + ]) assert.throws(() => store.authorize(value), /invalid run credential authorization/); +}); diff --git a/packages/credential-proxy/test/runtime-process.test.mjs b/packages/credential-proxy/test/runtime-process.test.mjs index 64fb7fe7..1e4d5d90 100644 --- a/packages/credential-proxy/test/runtime-process.test.mjs +++ b/packages/credential-proxy/test/runtime-process.test.mjs @@ -19,8 +19,8 @@ async function waitFor(predicate, label) { } } -async function startProxy() { - const child = spawn(process.execPath, [entry], { stdio: ["pipe", "pipe", "pipe"] }); +async function startProxy(directory) { + const child = spawn(process.execPath, [entry], { stdio: ["pipe", "pipe", "pipe"], env: { ...process.env, FACTORIZE_PROXY_BOOTSTRAP_SOCKET: join(directory, "bootstrap.sock") } }); let pending = ""; let log = ""; let stderr = ""; @@ -38,9 +38,13 @@ async function startProxy() { }); child.stderr.setEncoding("utf8").on("data", (chunk) => { stderr += chunk; }); await waitFor(() => messages.length > 0, "proxy startup"); - assert.deepEqual(messages.shift(), { type: "ready", state: "locked" }); + const ready = messages.shift(); + assert.equal(ready.type, "ready"); + assert.equal(ready.state, "locked"); + assert.match(ready.bootEpoch, /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i); return { child, + bootEpoch: ready.bootEpoch, get logs() { return `${log}${stderr}`; }, command(command) { child.stdin.write(`${JSON.stringify(command)}\n`); @@ -55,32 +59,33 @@ test("process starts locked, unlocks in memory, shuts down bounded, and restarts try { await writeFile(join(directory, "identity.json"), JSON.stringify({ identity: "public-fixture", generation: 1 })); await writeFile(join(directory, "public-metadata.json"), JSON.stringify({ displayName: "fixture" })); - const first = await startProxy(); + const first = await startProxy(directory); processes.push(first); - assert.deepEqual(await first.command({ type: "status" }), { type: "status", state: "locked" }); - assert.deepEqual(await first.command({ type: "unlock", key: keyMarker }), { type: "status", state: "unlocked" }); + assert.deepEqual(await first.command({ type: "status" }), { type: "status", state: "locked", proxyControlRegistered: false }); + assert.deepEqual(await first.command({ type: "unlock", key: keyMarker }), { type: "status", state: "unlocked", proxyControlRegistered: false }); first.child.kill("SIGTERM"); await waitFor(() => first.child.exitCode !== null || first.child.signalCode !== null, "first process stop"); assert.equal(first.child.exitCode, 0); - const restarted = await startProxy(); + const restarted = await startProxy(directory); processes.push(restarted); - assert.deepEqual(await restarted.command({ type: "status" }), { type: "status", state: "locked" }); - assert.deepEqual(await restarted.command({ type: "unlock", key: tokenMarker }), { type: "status", state: "unlocked" }); + assert.notEqual(restarted.bootEpoch, first.bootEpoch, "every process restart gets a fresh enrollment epoch"); + assert.deepEqual(await restarted.command({ type: "status" }), { type: "status", state: "locked", proxyControlRegistered: false }); + assert.deepEqual(await restarted.command({ type: "unlock", key: tokenMarker }), { type: "status", state: "unlocked", proxyControlRegistered: false }); restarted.child.stdin.write('{"type":"stop"}\n'); await waitFor(() => restarted.child.exitCode !== null || restarted.child.signalCode !== null, "command shutdown"); assert.equal(restarted.child.exitCode, 0); - const crashed = await startProxy(); + const crashed = await startProxy(directory); processes.push(crashed); - assert.deepEqual(await crashed.command({ type: "unlock", key: keyMarker }), { type: "status", state: "unlocked" }); + assert.deepEqual(await crashed.command({ type: "unlock", key: keyMarker }), { type: "status", state: "unlocked", proxyControlRegistered: false }); crashed.child.kill("SIGKILL"); await waitFor(() => crashed.child.exitCode !== null || crashed.child.signalCode !== null, "crash stop"); assert.equal(crashed.child.signalCode, "SIGKILL"); - const afterCrash = await startProxy(); + const afterCrash = await startProxy(directory); processes.push(afterCrash); - assert.deepEqual(await afterCrash.command({ type: "status" }), { type: "status", state: "locked" }); + assert.deepEqual(await afterCrash.command({ type: "status" }), { type: "status", state: "locked", proxyControlRegistered: false }); afterCrash.child.kill("SIGTERM"); await waitFor(() => afterCrash.child.exitCode !== null || afterCrash.child.signalCode !== null, "post-crash shutdown"); assert.equal(afterCrash.child.exitCode, 0); diff --git a/packages/credential-proxy/test/runtime-revocation.test.mjs b/packages/credential-proxy/test/runtime-revocation.test.mjs new file mode 100644 index 00000000..1eb7850c --- /dev/null +++ b/packages/credential-proxy/test/runtime-revocation.test.mjs @@ -0,0 +1,236 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { createProxyRuntime } from "../src/runtime.mjs"; +import { startProxyRunRevocationMonitor } from "../src/control-client.mjs"; + +const bootEpoch = "11111111-1111-4111-8111-111111111111"; + +function connection(runId, connectionId) { + return { authenticated: true, enrollmentConsumed: true, runId, connectionId, bootEpoch }; +} + +test("unlock keeps proxy access closed until encrypted account reconciliation finishes", async () => { + const runtime = createProxyRuntime({ bootEpoch, listenerFactory: async ({ connection: value }) => ({ connectionId: value.connectionId, close: async () => {} }) }); + let finishPreparation; + let preparing; + const started = new Promise(resolve => { preparing = resolve; }); + const prepared = new Promise(resolve => { finishPreparation = resolve; }); + try { + const unlocking = runtime.unlockAndPrepare("synthetic-unlock-key", async () => { + assert.equal(runtime.state, "preparing"); + assert.equal(runtime.getUnlockKey(), "synthetic-unlock-key"); + preparing(); + await prepared; + }); + await started; + await assert.rejects(runtime.acceptAuthenticatedConnection(connection("run-pending", "connection-pending")), /locked/); + assert.equal(runtime.state, "preparing"); + finishPreparation(); + await unlocking; + assert.equal(runtime.state, "unlocked"); + const listener = await runtime.acceptAuthenticatedConnection(connection("run-pending", "connection-ready")); + assert.equal(listener.connectionId, "connection-ready"); + } finally { + finishPreparation(); + await runtime.shutdown(); + } +}); + +test("unlock preparation failure clears the key and leaves proxy locked", async () => { + const runtime = createProxyRuntime({ bootEpoch }); + try { + await assert.rejects(runtime.unlockAndPrepare("synthetic-unlock-key", async () => { + assert.equal(runtime.getUnlockKey(), "synthetic-unlock-key"); + throw new Error("snapshot reconciliation failed"); + }), /snapshot reconciliation failed/); + assert.equal(runtime.state, "locked"); + assert.throws(() => runtime.getUnlockKey(), /locked/); + await assert.rejects(runtime.acceptAuthenticatedConnection(connection("run-failed", "connection-failed")), /locked/); + } finally { + await runtime.shutdown(); + } +}); + +test("run revocation closes every established connection and rejects later enrollment", async () => { + const closed = []; + const runtime = createProxyRuntime({ + bootEpoch, + listenerFactory: async ({ connection: value }) => ({ + connectionId: value.connectionId, + close: async () => { closed.push(value.connectionId); }, + }), + }); + runtime.unlock("synthetic-unlock-key"); + try { + await runtime.acceptAuthenticatedConnection(connection("run-a", "connection-a1")); + await runtime.acceptAuthenticatedConnection(connection("run-a", "connection-a2")); + await runtime.acceptAuthenticatedConnection(connection("run-b", "connection-b1")); + + assert.equal(await runtime.revokeRun("run-a"), 2); + assert.deepEqual(closed.sort(), ["connection-a1", "connection-a2"]); + await assert.rejects(runtime.acceptAuthenticatedConnection(connection("run-a", "connection-a3")), /revoked/); + + const stillActive = await runtime.acceptAuthenticatedConnection(connection("run-b", "connection-b2")); + assert.equal(stillActive.connectionId, "connection-b2"); + assert.equal(await runtime.revokeRun("run-a"), 0); + assert.deepEqual(closed.sort(), ["connection-a1", "connection-a2"]); + } finally { + await runtime.shutdown(); + } +}); + +test("key-generation replacement locks proxy memory and closes every connection", async () => { + const closed = []; + const runtime = createProxyRuntime({ bootEpoch, listenerFactory: async ({ connection: value }) => ({ connectionId: value.connectionId, close: async () => { closed.push(value.connectionId); } }) }); + runtime.unlock("synthetic-old-unlock-key"); + await runtime.acceptAuthenticatedConnection(connection("run-old-key", "connection-old-key")); + const monitor = startProxyRunRevocationMonitor({ + controlOrigin: "https://factorize.example", controlCredential: `fzc_${"A".repeat(43)}`, bootEpoch, runtime, intervalMs: 100, + fetchImpl: async () => Response.json({ revokedRunIds: [], lockProxy: true }), + }); + try { + const deadline = Date.now() + 1_000; + while (runtime.state !== "locked" && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 10)); + assert.equal(runtime.state, "locked"); + assert.deepEqual(closed, ["connection-old-key"]); + assert.throws(() => runtime.getUnlockKey(), /locked/); + } finally { await monitor.close(); await runtime.shutdown(); } +}); + +test("revocation racing listener creation closes the new listener before accepting it", async () => { + let resolveListener; + let closed = false; + const runtime = createProxyRuntime({ + bootEpoch, + listenerFactory: async () => new Promise(resolve => { resolveListener = resolve; }), + }); + runtime.unlock("synthetic-unlock-key"); + try { + const accepting = runtime.acceptAuthenticatedConnection(connection("run-race", "connection-race")); + await new Promise(resolve => setImmediate(resolve)); + await runtime.revokeRun("run-race"); + resolveListener({ connectionId: "connection-race", close: async () => { closed = true; } }); + await assert.rejects(accepting, /revoked/); + assert.equal(closed, true); + } finally { + await runtime.shutdown(); + } +}); + +test("an established run stays available during a control outage and closes when revocation arrives", async () => { + let releaseRevocation; + let observedSecondPoll; + const secondPoll = new Promise(resolve => { observedSecondPoll = resolve; }); + const revocationResponse = new Promise(resolve => { releaseRevocation = resolve; }); + const closePromises = new Map(); + const runtime = createProxyRuntime({ + bootEpoch, + listenerFactory: async ({ connection: value }) => ({ + connectionId: value.connectionId, + close: async () => closePromises.get(value.connectionId)?.(), + }), + }); + runtime.unlock("synthetic-unlock-key"); + let closed = false; + let resolveClosed; + const didClose = new Promise(resolve => { resolveClosed = resolve; }); + closePromises.set("connection-outage", () => { closed = true; resolveClosed(); }); + const revokedRunId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; + await runtime.acceptAuthenticatedConnection(connection(revokedRunId, "connection-outage")); + + let polls = 0; + const monitor = startProxyRunRevocationMonitor({ + controlOrigin: "https://factorize.example", + controlCredential: `fzc_${"A".repeat(43)}`, + bootEpoch, + runtime, + intervalMs: 100, + fetchImpl: async () => { + polls += 1; + if (polls === 1) throw new Error("control plane unavailable"); + observedSecondPoll(); + return revocationResponse; + }, + }); + try { + await secondPoll; + assert.equal(closed, false, "a failed control-plane poll does not impose a connection lease"); + releaseRevocation(Response.json({ revokedRunIds: [revokedRunId] })); + await didClose; + assert.equal(closed, true, "durable revocation closes the established connection when delivered"); + } finally { + releaseRevocation(Response.json({ revokedRunIds: [] })); + await monitor.close(); + await runtime.shutdown(); + } +}); + +test("connection grant refresh swaps current integrations and tolerates failed polls", async () => { + let grants = ["initial"]; + let attempts = 0; + const replaced = []; + let firstAttempt; + let secondReplacement; + const attempted = new Promise(resolve => { firstAttempt = resolve; }); + const replacedUpdated = new Promise(resolve => { secondReplacement = resolve; }); + const runtime = createProxyRuntime({ + bootEpoch, + grantRefreshIntervalMs: 100, + listenerFactory: async ({ connection: value }) => ({ + connectionId: value.connectionId, + replaceGrants: next => { replaced.push(next); if (next[0] === "updated") secondReplacement(); }, + close: async () => {}, + }), + }); + runtime.unlock("synthetic-unlock-key"); + try { + await runtime.acceptAuthenticatedConnection({ ...connection("run-refresh", "connection-refresh"), refreshGrants: async () => { + attempts += 1; + if (attempts === 1) { firstAttempt(); throw new Error("temporary control-plane outage"); } + return grants; + } }); + await attempted; + assert.deepEqual(replaced, [], "a failed refresh preserves the established grants"); + grants = ["updated"]; + await replacedUpdated; + assert.deepEqual(replaced, [["updated"]]); + } finally { + await runtime.shutdown(); + } +}); + +test("a delayed grant refresh cannot restore access after run revocation", async () => { + let resolveRefresh; + let refreshStarted; + const started = new Promise(resolve => { refreshStarted = resolve; }); + const refreshed = new Promise(resolve => { resolveRefresh = resolve; }); + const replaced = []; + let closed = false; + const runId = "run-refresh-revoked"; + const runtime = createProxyRuntime({ + bootEpoch, + grantRefreshIntervalMs: 100, + listenerFactory: async ({ connection: value }) => ({ + connectionId: value.connectionId, + replaceGrants: grants => replaced.push(grants), + close: async () => { closed = true; }, + }), + }); + runtime.unlock("synthetic-unlock-key"); + try { + await runtime.acceptAuthenticatedConnection({ + ...connection(runId, "connection-refresh-revoked"), + refreshGrants: async () => { refreshStarted(); return refreshed; }, + }); + await started; + assert.equal(await runtime.revokeRun(runId), 1); + assert.equal(closed, true); + + resolveRefresh(["stale-restored-grant"]); + await new Promise(resolve => setImmediate(resolve)); + assert.deepEqual(replaced, [], "a delayed snapshot must not replace grants after revocation"); + } finally { + resolveRefresh([]); + await runtime.shutdown(); + } +}); diff --git a/packages/credential-proxy/test/tls-identity.test.mjs b/packages/credential-proxy/test/tls-identity.test.mjs new file mode 100644 index 00000000..dd09554d --- /dev/null +++ b/packages/credential-proxy/test/tls-identity.test.mjs @@ -0,0 +1,50 @@ +import assert from "node:assert/strict"; +import { execFile as execFileCallback } from "node:child_process"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { test } from "node:test"; +import { createProxyTlsAuthority } from "../src/tls-identity.mjs"; + +const execFile = promisify(execFileCallback); +const bootEpoch = "11111111-1111-4111-8111-111111111111"; + +test("proxy TLS authority provisions a boot-bound server cert and signs only canonical VM identities", async () => { + const directory = await mkdtemp(join(tmpdir(), "factorize-tls-test-")); + const authority = await createProxyTlsAuthority({ serverName: "proxy.internal.test", bootEpoch }); + try { + const keyPath = join(directory, "vm-key.pem"); + const csrPath = join(directory, "vm.csr"); + await execFile("openssl", ["genpkey", "-algorithm", "EC", "-pkeyopt", "ec_paramgen_curve:P-256", "-out", keyPath]); + await execFile("openssl", ["req", "-new", "-key", keyPath, "-subj", "/CN=ignored-by-issuer", "-out", csrPath]); + const csr = await readFile(csrPath, "utf8"); + const issued = await authority.signVmCsr({ vmIdentity: "fly:machine-7", csr }); + assert.equal(issued.trustBundle, authority.trustBundle); + assert.match(issued.certificate, /BEGIN CERTIFICATE/); + await writeFile(join(directory, "root.pem"), issued.trustBundle, { mode: 0o600 }); + await writeFile(join(directory, "client.pem"), issued.certificate, { mode: 0o600 }); + await execFile("openssl", ["verify", "-CAfile", join(directory, "root.pem"), join(directory, "client.pem")]); + const { stdout: clientText } = await execFile("openssl", ["x509", "-in", join(directory, "client.pem"), "-noout", "-subject", "-ext", "subjectAltName"]); + assert.match(clientText, /CN\s*=\s*fly:machine-7/); + assert.match(clientText, /URI:urn:factorize:vm:Zmx5Om1hY2hpbmUtNw/); + const { stdout: usageText } = await execFile("openssl", ["x509", "-in", join(directory, "client.pem"), "-noout", "-ext", "extendedKeyUsage"]); + assert.match(usageText, /TLS Web Client Authentication/); + + await writeFile(join(directory, "server.pem"), authority.tls.cert, { mode: 0o600 }); + await writeFile(join(directory, "server-root.pem"), authority.tls.ca, { mode: 0o600 }); + await execFile("openssl", ["verify", "-CAfile", join(directory, "server-root.pem"), "-verify_hostname", "proxy.internal.test", join(directory, "server.pem")]); + await assert.rejects(authority.signVmCsr({ vmIdentity: "../machine", csr }), /invalid VM certificate request/); + await assert.rejects(authority.signVmCsr({ vmIdentity: "fly:machine-7", csr: "not a CSR" })); + await authority.close(); + await assert.rejects(authority.signVmCsr({ vmIdentity: "fly:machine-7", csr }), /authority is closed/); + } finally { + await authority.close(); + await rm(directory, { recursive: true, force: true }); + } +}); + +test("proxy TLS authority rejects malformed server names and boot identities", async () => { + await assert.rejects(createProxyTlsAuthority({ serverName: "https://proxy.test", bootEpoch })); + await assert.rejects(createProxyTlsAuthority({ serverName: "proxy.test", bootEpoch: "old" })); +}); diff --git a/packages/docs/llms-full.txt b/packages/docs/llms-full.txt index 4591b998..266e6c3d 100644 --- a/packages/docs/llms-full.txt +++ b/packages/docs/llms-full.txt @@ -889,6 +889,7 @@ This guide records the agreed design in [GEN-2824](https://linear.app/generation - Factorize is trusted to provision and pair the proxy, administer cloud machines, and authorize run enrollment. Initial pairing is trusted provisioning, not independent user verification. The paired identity includes proxy, Fly app/Machine, HTTPS origin, SSH host fingerprint, and generation. Unexpected identity changes must fail closed and require an authenticated owner's explicit approval. - A named bearer integration is bound to one public HTTPS origin. A granted job can use HTTP methods, paths, queries, and bodies on that origin. Redirects are rejected and destinations must be public. Agent compromise can exercise all authority in that job's grant. Already-sent requests cannot be recalled; upstream effects can persist independently, including tokens returned by an upstream service. - Factorize run API credentials are separate, per-run scoped/revocable credentials. They are not encrypted with the user's key and are delivered to proxy memory over a separate control path; they must never be placed in a Fly agent VM. Keep grants explicit and minimal. Native exe.dev integrations remain available alongside portable integrations in a separate namespace; adding a portable integration must not change native behavior or implicitly grant it to a job. +- Portable integration reflection is served at `GET $FACTORIZE_PROXY_URL/integrations` only through that VM's authenticated tunnel. It lists the connection's selected integration names, descriptions, local paths, examples, and availability; it omits upstream origins and all credentials. Locking clears decrypted grants and aborts in-flight forwarding while preserving safe metadata as `locked` for that established connection. A later unlock refreshes grants from the current run snapshot before restoring access. Each VM connection has an independent listener and reflection view. - Database or server-encryption-key theft alone does not reveal the inner user-encrypted records. A compromised Factorize control plane can authorize malicious workloads and administer provisioned cloud machines. A compromised proxy host can access secrets while unlocked. This design does not promise physical erasure of host memory, prevent a key holder from decrypting their ciphertext, or prevent upstream services from retaining access they have granted. ## Lifecycle and recovery runbook diff --git a/packages/docs/openapi.yaml b/packages/docs/openapi.yaml index 652141b8..7551ab5c 100644 --- a/packages/docs/openapi.yaml +++ b/packages/docs/openapi.yaml @@ -1806,18 +1806,479 @@ components: schema: $ref: "#/components/schemas/Error" paths: + /api/v1/model-accounts: + get: + summary: List model accounts available to the owner + description: |- + Requires an interactive owner browser session. Returns account metadata and the collection revision only; encrypted account caches and credentials are never returned. + + Requires the integrations:read scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "200": + description: Model account metadata and collection revision + content: + application/json: + schema: + type: object + properties: + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + accounts: + type: array + items: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + name: + type: string + description: + type: + - string + - "null" + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + required: + - id + - name + - description + - revision + additionalProperties: false + required: + - revision + - accounts + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: get_api_v1_model_accounts + tags: + - model-accounts + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: integrations:read + x-owner-session-required: true + post: + summary: Create a model account + description: |- + Requires an interactive owner browser session and the current collection revision. Accepts only user-encrypted initial account state; credentials are established directly with the credential proxy. + + Requires the integrations:write scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "201": + description: Created model account + content: + application/json: + schema: + type: object + properties: + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + duplicate: + type: boolean + required: + - revision + - duplicate + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + "415": + $ref: "#/components/responses/Error415" + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: post_api_v1_model_accounts + tags: + - model-accounts + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: integrations:write + x-owner-session-required: true + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + expectedRevision: + type: integer + minimum: 0 + maximum: 9007199254740991 + idempotencyKey: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + name: + type: string + minLength: 1 + maxLength: 120 + description: + anyOf: + - type: string + maxLength: 500 + - type: "null" + encryptedPayload: + type: string + minLength: 1 + maxLength: 65536 + required: + - id + - expectedRevision + - idempotencyKey + - name + - description + - encryptedPayload + additionalProperties: false + /api/v1/model-accounts/{accountId}: + delete: + summary: Delete a model account + description: |- + Requires an interactive owner browser session and the current collection revision. Deletion is recorded as a tombstone so future snapshots and unlock reconciliation cannot restore the account. + + Requires the integrations:write scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "200": + description: Deleted model account + content: + application/json: + schema: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + deleted: + type: boolean + const: true + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + duplicate: + type: boolean + required: + - id + - deleted + - revision + - duplicate + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + "415": + $ref: "#/components/responses/Error415" + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: delete_api_v1_model_accounts_accountId + tags: + - model-accounts + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: integrations:write + x-owner-session-required: true + parameters: + - name: accountId + in: path + required: true + schema: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + expectedRevision: + type: integer + minimum: 0 + maximum: 9007199254740991 + idempotencyKey: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + required: + - expectedRevision + - idempotencyKey + additionalProperties: false + /api/v1/model-accounts/{accountId}/browser-capability: + post: + summary: Issue a short-lived browser capability for a paired model account + description: |- + Requires an interactive owner browser session. Returns a single-use, 60-second capability scoped to the account, action, proxy boot and exact application origin. The browser sends it directly to the paired proxy; Factorize never receives model-account tokens. + + Requires the integrations:write scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "200": + description: Paired proxy endpoint and one-use capability + content: + application/json: + schema: + type: object + properties: + endpoint: + type: string + format: uri + capability: + type: string + required: + - endpoint + - capability + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + "415": + $ref: "#/components/responses/Error415" + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: post_api_v1_model_accounts_accountId_browser_capability + tags: + - model-accounts + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: integrations:write + x-owner-session-required: true + parameters: + - name: accountId + in: path + required: true + schema: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + action: + type: string + enum: + - authorize + - status + - cancel + required: + - action + additionalProperties: false /api/v1/unlock-key/verification: get: - summary: Get the current encrypted unlock-key verification record + summary: Get the current encrypted unlock-key verification record + description: |- + Requires an interactive owner browser session. Returns user-encrypted verification ciphertext and its key generation; the unlock key is never accepted or returned. + + Requires the integrations:read scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "200": + description: Encrypted verification record or empty setup state + content: + application/json: + schema: + type: object + properties: + ownerId: + type: string + verification: + anyOf: + - type: object + properties: + generationId: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + encryptedVerification: + type: string + required: + - generationId + - encryptedVerification + additionalProperties: false + - type: "null" + required: + - ownerId + - verification + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: get_api_v1_unlock_key_verification + tags: + - unlock-key + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: integrations:read + x-owner-session-required: true + post: + summary: Create the encrypted unlock-key verification record + description: |- + Requires an interactive owner browser session. Stores only browser-encrypted verification ciphertext and permits one setup per account. + + Requires the integrations:write scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "200": + description: Created verification record + content: + application/json: + schema: + type: object + properties: + generationId: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + required: + - generationId + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + "415": + $ref: "#/components/responses/Error415" + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: post_api_v1_unlock_key_verification + tags: + - unlock-key + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: integrations:write + x-owner-session-required: true + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + generationId: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + encryptedVerification: + type: string + minLength: 1 + maxLength: 4096 + required: + - generationId + - encryptedVerification + additionalProperties: false + /api/v1/unlock-key/rotation-snapshot: + get: + summary: Get encrypted state for owner key rotation description: |- - Requires an interactive owner browser session. Returns user-encrypted verification ciphertext and its key generation; the unlock key is never accepted or returned. + Owner browser session only. Returns encrypted records, tombstones, and pending change history for local re-encryption; plaintext credentials and unlock keys are never returned. Requires the integrations:read scope. An interactive owner session is required; bearer tokens cannot call this operation. responses: "200": - description: Encrypted verification record or empty setup state + description: Encrypted rekey snapshot content: application/json: schema: @@ -1825,24 +2286,136 @@ paths: properties: ownerId: type: string - verification: - anyOf: - - type: object - properties: - generationId: - type: string - format: uuid - pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ - encryptedVerification: - type: string - required: - - generationId - - encryptedVerification - additionalProperties: false - - type: "null" + generationId: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + records: + maxItems: 5000 + type: array + items: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + kind: + type: string + enum: + - bearer + - model + name: + type: string + minLength: 1 + maxLength: 120 + origin: + type: + - string + - "null" + description: + anyOf: + - type: string + maxLength: 500 + - type: "null" + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + encryptedPayload: + anyOf: + - type: string + maxLength: 65536 + - type: "null" + deletedAt: + anyOf: + - type: string + format: date-time + pattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?(?:Z|([+-](?:[01]\d|2[0-3]):[0-5]\d)))$ + - type: "null" + required: + - id + - kind + - name + - origin + - description + - revision + - encryptedPayload + - deletedAt + additionalProperties: false + changes: + maxItems: 10000 + type: array + items: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + kind: + type: string + enum: + - bearer + - model + name: + type: string + minLength: 1 + maxLength: 120 + origin: + type: + - string + - "null" + description: + anyOf: + - type: string + maxLength: 500 + - type: "null" + revision: + type: integer + exclusiveMinimum: 0 + maximum: 9007199254740991 + idempotencyKey: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + operation: + type: string + enum: + - upsert + - delete + encryptedPayload: + anyOf: + - type: string + maxLength: 65536 + - type: "null" + recordRevision: + anyOf: + - type: integer + exclusiveMinimum: 0 + maximum: 9007199254740991 + - type: "null" + required: + - id + - kind + - name + - origin + - description + - revision + - idempotencyKey + - operation + - encryptedPayload + additionalProperties: false required: - ownerId - - verification + - generationId + - revision + - records + - changes additionalProperties: false "400": $ref: "#/components/responses/Error400" @@ -1860,7 +2433,7 @@ paths: $ref: "#/components/responses/Error500" "503": $ref: "#/components/responses/Error503" - operationId: get_api_v1_unlock_key_verification + operationId: get_api_v1_unlock_key_rotation_snapshot tags: - unlock-key security: @@ -1868,17 +2441,18 @@ paths: - cookieAuth: [] x-required-scope: integrations:read x-owner-session-required: true + /api/v1/unlock-key/rotation: post: - summary: Create the encrypted unlock-key verification record + summary: Atomically replace the unlock-key generation description: |- - Requires an interactive owner browser session. Stores only browser-encrypted verification ciphertext and permits one setup per account. + Owner browser session only. Commits a revision-checked encrypted generation and verification record, revokes grants, consumes enrollments, and locks the proxy. Fly connection and execution configuration are preserved. Requires the integrations:write scope. An interactive owner session is required; bearer tokens cannot call this operation. responses: "200": - description: Created verification record + description: Key generation replaced content: application/json: schema: @@ -1888,8 +2462,16 @@ paths: type: string format: uuid pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + duplicate: + type: boolean required: - generationId + - revision + - duplicate additionalProperties: false "400": $ref: "#/components/responses/Error400" @@ -1909,7 +2491,7 @@ paths: $ref: "#/components/responses/Error500" "503": $ref: "#/components/responses/Error503" - operationId: post_api_v1_unlock_key_verification + operationId: post_api_v1_unlock_key_rotation tags: - unlock-key security: @@ -1924,17 +2506,145 @@ paths: schema: type: object properties: + expectedGenerationId: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ generationId: type: string format: uuid pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + expectedRevision: + type: integer + minimum: 0 + maximum: 9007199254740991 encryptedVerification: type: string minLength: 1 maxLength: 4096 + records: + maxItems: 5000 + type: array + items: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + kind: + type: string + enum: + - bearer + - model + name: + type: string + minLength: 1 + maxLength: 120 + origin: + type: + - string + - "null" + description: + anyOf: + - type: string + maxLength: 500 + - type: "null" + revision: + type: integer + minimum: 0 + maximum: 9007199254740991 + encryptedPayload: + anyOf: + - type: string + maxLength: 65536 + - type: "null" + deletedAt: + anyOf: + - type: string + format: date-time + pattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?(?:Z|([+-](?:[01]\d|2[0-3]):[0-5]\d)))$ + - type: "null" + required: + - id + - kind + - name + - origin + - description + - revision + - encryptedPayload + - deletedAt + additionalProperties: false + changes: + maxItems: 10000 + type: array + items: + type: object + properties: + id: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + kind: + type: string + enum: + - bearer + - model + name: + type: string + minLength: 1 + maxLength: 120 + origin: + type: + - string + - "null" + description: + anyOf: + - type: string + maxLength: 500 + - type: "null" + revision: + type: integer + exclusiveMinimum: 0 + maximum: 9007199254740991 + idempotencyKey: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ + operation: + type: string + enum: + - upsert + - delete + encryptedPayload: + anyOf: + - type: string + maxLength: 65536 + - type: "null" + recordRevision: + anyOf: + - type: integer + exclusiveMinimum: 0 + maximum: 9007199254740991 + - type: "null" + required: + - id + - kind + - name + - origin + - description + - revision + - idempotencyKey + - operation + - encryptedPayload + additionalProperties: false required: + - expectedGenerationId - generationId + - expectedRevision - encryptedVerification + - records + - changes additionalProperties: false /api/v1/integrations/bearer: get: @@ -13086,6 +13796,67 @@ paths: schema: type: string minLength: 1 + /api/v1/runs/{runId}/retry-launch: + post: + summary: Manually retry a blocked launch + description: |- + Retries only a pre-launch proxy-gated run after its prerequisites have been resolved. The run remains blocked until an interactive owner explicitly retries it; unlock or status changes do not automatically launch it. + + Requires the runs:cancel scope. + + An interactive owner session is required; bearer tokens cannot call this operation. + responses: + "200": + description: Manual retry queued + content: + application/json: + schema: + type: object + required: + - runId + - retryRequested + properties: + runId: + type: string + format: uuid + retryRequested: + type: boolean + const: true + additionalProperties: false + "400": + $ref: "#/components/responses/Error400" + "401": + $ref: "#/components/responses/Error401" + "403": + $ref: "#/components/responses/Error403" + description: Interactive owner session required + "404": + $ref: "#/components/responses/Error404" + "409": + $ref: "#/components/responses/Error409" + description: Run is not waiting for a manual launch retry + "429": + $ref: "#/components/responses/Error429" + "500": + $ref: "#/components/responses/Error500" + "503": + $ref: "#/components/responses/Error503" + operationId: post_api_v1_runs_runId_retry_launch + tags: + - runs + security: + - bearerAuth: [] + - cookieAuth: [] + x-required-scope: runs:cancel + x-owner-session-required: true + parameters: + - name: runId + in: path + required: true + schema: + type: string + format: uuid + pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ /api/v1/runs/{runId}/kill: post: summary: Kill run and immediately release its concurrency slot diff --git a/packages/docs/self-hosting/credential-proxy-security.mdx b/packages/docs/self-hosting/credential-proxy-security.mdx index 1ccd9bdc..5eadf520 100644 --- a/packages/docs/self-hosting/credential-proxy-security.mdx +++ b/packages/docs/self-hosting/credential-proxy-security.mdx @@ -13,6 +13,7 @@ This guide records the agreed design in [GEN-2824](https://linear.app/generation - Factorize is trusted to provision and pair the proxy, administer cloud machines, and authorize run enrollment. Initial pairing is trusted provisioning, not independent user verification. The paired identity includes proxy, Fly app/Machine, HTTPS origin, SSH host fingerprint, and generation. Unexpected identity changes must fail closed and require an authenticated owner's explicit approval. - A named bearer integration is bound to one public HTTPS origin. A granted job can use HTTP methods, paths, queries, and bodies on that origin. Redirects are rejected and destinations must be public. Agent compromise can exercise all authority in that job's grant. Already-sent requests cannot be recalled; upstream effects can persist independently, including tokens returned by an upstream service. - Factorize run API credentials are separate, per-run scoped/revocable credentials. They are not encrypted with the user's key and are delivered to proxy memory over a separate control path; they must never be placed in a Fly agent VM. Keep grants explicit and minimal. Native exe.dev integrations remain available alongside portable integrations in a separate namespace; adding a portable integration must not change native behavior or implicitly grant it to a job. +- Portable integration reflection is served at `GET $FACTORIZE_PROXY_URL/integrations` only through that VM's authenticated tunnel. It lists the connection's selected integration names, descriptions, local paths, examples, and availability; it omits upstream origins and all credentials. Locking clears decrypted grants and aborts in-flight forwarding while preserving safe metadata as `locked` for that established connection. A later unlock refreshes grants from the current run snapshot before restoring access. Each VM connection has an independent listener and reflection view. - Database or server-encryption-key theft alone does not reveal the inner user-encrypted records. A compromised Factorize control plane can authorize malicious workloads and administer provisioned cloud machines. A compromised proxy host can access secrets while unlocked. This design does not promise physical erasure of host memory, prevent a key holder from decrypting their ciphertext, or prevent upstream services from retaining access they have granted. ## Lifecycle and recovery runbook diff --git a/scripts/build-credential-proxy-image.sh b/scripts/build-credential-proxy-image.sh new file mode 100755 index 00000000..27b4f24d --- /dev/null +++ b/scripts/build-credential-proxy-image.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +set -euo pipefail + +repository_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$repository_root" + +source_revision="$(git rev-parse HEAD)" +source_short="$(git rev-parse --short=12 HEAD)" +artifact_dir="${FACTORIZE_PROXY_IMAGE_ARTIFACT_DIR:-${HOME}/.factorize-job/credential-proxy-image}" +mkdir -p "$artifact_dir" +metadata_file="$(mktemp)" +trap 'rm -f "$metadata_file"' EXIT + +docker buildx build \ + --platform linux/amd64 \ + --file packages/credential-proxy/Dockerfile \ + --tag "factorize-credential-proxy:source-${source_short}" \ + --build-arg "SOURCE_REVISION=${source_revision}" \ + --provenance=mode=max \ + --sbom=generator=docker/buildkit-syft-scanner@sha256:873a7458c3c8ac8e32c7b796c10e1600b07b963d2aa000a72a2d97aaaa2d6315 \ + --metadata-file "$metadata_file" \ + --output "type=oci,dest=${artifact_dir}/credential-proxy-${source_short}.oci.tar" \ + . + +image_digest="$(jq -er '."containerimage.digest" | select(test("^sha256:[0-9a-f]{64}$"))' "$metadata_file")" +jq -n \ + --arg sourceRevision "$source_revision" \ + --arg imageDigest "$image_digest" \ + --arg baseImage "node:24.12.0-bookworm-slim@sha256:7326fb2dbdce998edd72140946851be64ef4a643e8715e138ca467e8e9d92c99" \ + --arg codexVersion "codex-cli 0.167.1" \ + --arg artifact "credential-proxy-${source_short}.oci.tar" \ + '{schemaVersion:1,sourceRevision:$sourceRevision,baseImage:$baseImage,codexVersion:$codexVersion,imageDigest:$imageDigest,ociArchive:$artifact,provenance:"BuildKit max provenance and SBOM are attached to the OCI output"}' \ + > "${artifact_dir}/build-provenance.json" +cat "${artifact_dir}/build-provenance.json"