Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
158 commits
Select commit Hold shift + click to select a range
2c56c36
feat(credential-proxy): add bounded bearer forwarding
Nov 11, 2026
41d09a3
feat(credential-proxy): stream authorized tunnel requests
Nov 11, 2026
f00b2f0
feat(credential-proxy): connect authenticated VM tunnel
Nov 11, 2026
08fddf2
Bind enrollment acceptance to proxy boot epoch
Nov 11, 2026
adfefc5
Bind tunnel enrollment to authenticated VM peer
Nov 11, 2026
5b44328
Add authenticated proxy boot registration
Nov 11, 2026
1ed566e
Add secure proxy bootstrap provisioning boundary
Nov 11, 2026
edf98ea
Provision credential proxy with isolated bootstrap
Nov 11, 2026
e631553
Retry proxy tunnel enrollment before forwarding
Nov 11, 2026
545d9e7
Add authenticated proxy enrollment control API
Nov 11, 2026
f1ff6ad
Wire proxy-side run enrollment into tunnel auth
Nov 11, 2026
b4462a2
Wire proxy control enrollment into tunnel startup
Nov 11, 2026
a96a584
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 11, 2026
ef4b952
Retain proxy control credential after boot registration
Nov 11, 2026
59e6bea
Authenticate tunnel enrollment with run credential
Nov 11, 2026
2188c7d
Keep run credential out of consume request
Nov 11, 2026
78b86ee
Cover invalid run enrollment bindings
Nov 11, 2026
c46c3a2
Narrow active run identity before enrollment
Nov 11, 2026
fd74010
Close proxy tunnels when a run is revoked
Nov 11, 2026
68d348e
Add decrypted run integration grant resolver
Nov 11, 2026
eb90329
Expose selected integrations to authenticated proxy
Nov 11, 2026
3e79721
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 11, 2026
6f13d7b
Fix encrypted snapshot proxy test fixture
Nov 11, 2026
fea379c
Wire proxy snapshots into local grant resolution
Nov 11, 2026
b765691
Preserve grant resolver selection boundaries
Nov 11, 2026
15335fa
Resolve empty and locked grants safely
Nov 11, 2026
32414a2
Match snapshot control request contract
Nov 11, 2026
c22a412
Bind tunnel VM identity to certificate SAN
Nov 11, 2026
3ed3f07
GEN-2824 provision mTLS identities and revoke tunnels
Nov 11, 2026
a3aad85
GEN-2824 include selected model account snapshot
Nov 11, 2026
463ce53
GEN-2824 update snapshot fixture for model selection
Nov 12, 2026
44c5bde
GEN-2824 add isolated Codex auth helper
Nov 12, 2026
69f05bc
GEN-2824 persist encrypted Codex account refreshes
Nov 12, 2026
454232c
GEN-2824 encrypt Codex account caches in proxy
Nov 12, 2026
28f8f1c
GEN-2824 restore encrypted Codex caches on unlock
Nov 12, 2026
d27d95a
GEN-2824 persist encrypted model caches by revision
Nov 12, 2026
df221e5
GEN-2824 compose model cache lifecycle into proxy startup
Nov 12, 2026
a20d434
GEN-2824 add private browser model authorization control
Nov 12, 2026
c876176
GEN-2824 issue paired model account browser capabilities
Nov 12, 2026
dd266cc
GEN-2824 connect model account browser authorization
Nov 12, 2026
2ccb27b
GEN-2824 cover capability endpoint in browser test
Nov 12, 2026
87341ad
GEN-2824 add model account create and delete
Nov 12, 2026
4e4a5c0
GEN-2824 recover stale model account cache writes
Nov 12, 2026
e5a6a97
GEN-2824 clear deleted model caches on unlock
Nov 12, 2026
29cd548
GEN-2824 cover model cache key rotation race
Nov 12, 2026
7a4fdfa
GEN-2824 cover model reset refresh race
Nov 12, 2026
5a5f01a
GEN-2824 cover reset during refresh retry
Nov 12, 2026
8960123
GEN-2824 invalidate refresh retries after model rotation
Nov 12, 2026
e627df1
GEN-2824 define portable proxy launch gates
Nov 12, 2026
e85bf4a
GEN-2824 disable Fly job machine restarts
Nov 12, 2026
1f25532
GEN-2824 persist proxy launch blocks and manual retry
Nov 12, 2026
eda72e9
Merge origin/main into GEN-2824 proxy launch work
Nov 12, 2026
eca2eca
feat(credential-proxy): add bounded bearer forwarding
Nov 11, 2026
a6a282f
feat(credential-proxy): stream authorized tunnel requests
Nov 11, 2026
f1d2c84
feat(credential-proxy): connect authenticated VM tunnel
Nov 11, 2026
8c6be50
Bind enrollment acceptance to proxy boot epoch
Nov 11, 2026
1ec0345
Bind tunnel enrollment to authenticated VM peer
Nov 11, 2026
4ae0b74
Add authenticated proxy boot registration
Nov 11, 2026
56fd6c2
Add secure proxy bootstrap provisioning boundary
Nov 11, 2026
6df5360
Provision credential proxy with isolated bootstrap
Nov 11, 2026
207d68f
Retry proxy tunnel enrollment before forwarding
Nov 11, 2026
497f4bd
Add authenticated proxy enrollment control API
Nov 11, 2026
516938c
Wire proxy-side run enrollment into tunnel auth
Nov 11, 2026
8c35922
Wire proxy control enrollment into tunnel startup
Nov 11, 2026
712f6fe
Retain proxy control credential after boot registration
Nov 11, 2026
f5fcd51
Authenticate tunnel enrollment with run credential
Nov 11, 2026
3fcb724
Keep run credential out of consume request
Nov 11, 2026
62b0729
Cover invalid run enrollment bindings
Nov 11, 2026
9e85768
Narrow active run identity before enrollment
Nov 11, 2026
ced6454
Close proxy tunnels when a run is revoked
Nov 11, 2026
184cd43
Add decrypted run integration grant resolver
Nov 11, 2026
7457e58
Expose selected integrations to authenticated proxy
Nov 11, 2026
af71a90
Fix encrypted snapshot proxy test fixture
Nov 11, 2026
69fa35e
Wire proxy snapshots into local grant resolution
Nov 11, 2026
3467c03
Preserve grant resolver selection boundaries
Nov 11, 2026
a3a74a2
Resolve empty and locked grants safely
Nov 11, 2026
2872e4e
Match snapshot control request contract
Nov 11, 2026
41f35d1
Bind tunnel VM identity to certificate SAN
Nov 11, 2026
434c008
GEN-2824 provision mTLS identities and revoke tunnels
Nov 11, 2026
41b91d2
GEN-2824 include selected model account snapshot
Nov 11, 2026
1411909
GEN-2824 update snapshot fixture for model selection
Nov 12, 2026
850e36a
GEN-2824 add isolated Codex auth helper
Nov 12, 2026
75f9df0
GEN-2824 persist encrypted Codex account refreshes
Nov 12, 2026
32cc5b7
GEN-2824 encrypt Codex account caches in proxy
Nov 12, 2026
d5c433b
GEN-2824 restore encrypted Codex caches on unlock
Nov 12, 2026
4653583
GEN-2824 persist encrypted model caches by revision
Nov 12, 2026
182b8bf
GEN-2824 compose model cache lifecycle into proxy startup
Nov 12, 2026
555656a
GEN-2824 add private browser model authorization control
Nov 12, 2026
96d8272
GEN-2824 issue paired model account browser capabilities
Nov 12, 2026
8a68534
GEN-2824 connect model account browser authorization
Nov 12, 2026
6e81b33
GEN-2824 cover capability endpoint in browser test
Nov 12, 2026
834e2b3
GEN-2824 add model account create and delete
Nov 12, 2026
b99aa42
GEN-2824 recover stale model account cache writes
Nov 12, 2026
4883c88
GEN-2824 clear deleted model caches on unlock
Nov 12, 2026
1d56983
GEN-2824 cover model cache key rotation race
Nov 12, 2026
6ed5e76
GEN-2824 cover model reset refresh race
Nov 12, 2026
4b67c2a
GEN-2824 cover reset during refresh retry
Nov 12, 2026
3981a80
GEN-2824 invalidate refresh retries after model rotation
Nov 12, 2026
3414c7b
GEN-2824 define portable proxy launch gates
Nov 12, 2026
6d29321
GEN-2824 disable Fly job machine restarts
Nov 12, 2026
a5c6d8d
GEN-2824 persist proxy launch blocks and manual retry
Nov 12, 2026
cdf1d02
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 12, 2026
28dd751
GEN-2824 report proxy unlock state to scheduler
Nov 12, 2026
02492a0
Merge remote-tracking branch 'origin/GEN-2824-proxy-forwarding' into …
Nov 12, 2026
f2ac3f9
GEN-2824 persist blocked proxy launch state
Nov 12, 2026
d529ab1
GEN-2824 cover proxy launch lifecycle states
Nov 12, 2026
542a82c
GEN-2824 recheck proxy gate after manual retry
Nov 12, 2026
628b728
GEN-2824 prepare restart-safe Fly job machines
Nov 12, 2026
2aa9963
GEN-2824 add initial Fly job backend
Nov 12, 2026
3eead27
GEN-2824 select Fly backend in run scheduler
Nov 12, 2026
471e944
GEN-2824 cover Fly scheduler lifecycle retries
Nov 12, 2026
321f338
GEN-2824 share the execution job launcher
Nov 12, 2026
12c1d0b
GEN-2824 cover Fly scheduler recovery
Nov 12, 2026
77e039d
GEN-2824 cover revocation during control outage
Nov 12, 2026
2064250
Refresh integration grants on active proxy connections
Nov 12, 2026
620a284
GEN-2824 reject delayed grants after revocation
Nov 12, 2026
9df3a14
GEN-2824 authorize provider-bound run grants
Nov 12, 2026
7599378
GEN-2824 keep revoked grants closed on reconnect
Nov 12, 2026
d0dce67
GEN-2824 deliver run credentials to proxy memory
Nov 12, 2026
269e441
GEN-2824 type Fly credential isolation test
Nov 12, 2026
5cc9623
GEN-2824 revoke credentials for terminal unconnected runs
Nov 12, 2026
80b00a0
GEN-2824 connect Fly guest proxy tunnel
Nov 12, 2026
0c745da
test: exercise generated Fly guest tunnel helper
Nov 12, 2026
cee1750
GEN-2824 cover custom Fly Codex launch
Nov 12, 2026
8042d6d
GEN-2824 cover Fly startup and cleanup lifecycle
Nov 12, 2026
cd416c0
test: type Fly machine mock arguments
Nov 12, 2026
acf87e8
fix: skip redundant Fly stop for stopped Machines
Nov 12, 2026
62c195d
test: cover Fly start acknowledgement recovery
Nov 12, 2026
8ff1ce9
GEN-2824 verify Fly artifact collection before cleanup
Nov 12, 2026
3c3af12
GEN-2824 cover Fly guest launch failures
Nov 12, 2026
6685ae5
GEN-2824 exercise shared agent job launcher
Nov 12, 2026
72c2869
GEN-2824 test custom image launcher setup
Nov 12, 2026
e31e4fe
GEN-2824 validate Fly Codex image version
Nov 12, 2026
b4564e4
GEN-2824 verify Fly proxy image configuration
Nov 12, 2026
c34c2bc
GEN-2824 strengthen Fly image acceptance checks
Nov 12, 2026
82fda0b
GEN-2824 restrict Fly proxy to reviewed image
Nov 12, 2026
768fd73
GEN-2824 clean up Fly proxy on invalid boot identity
Nov 12, 2026
fc28b64
GEN-2824 verify Fly proxy provisioning identity
Nov 12, 2026
834ac1a
GEN-2824 require complete returned Fly identity
Nov 12, 2026
0ea4cb2
GEN-2824 honor Fly proxy machine size
Nov 12, 2026
9e61833
GEN-2824 cover Fly proxy size boundaries
Nov 12, 2026
7d035ea
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 12, 2026
7db53e5
GEN-2824 assert custom Fly code image configuration
Nov 12, 2026
7ccbd56
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 12, 2026
3be9774
GEN-2824 preserve failed guest process results
Nov 12, 2026
c2e06c7
GEN-2824 assert Fly process failure state
Nov 12, 2026
9840956
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 12, 2026
0d99bef
GEN-2824 gate proxy access on unlock reconciliation
Nov 12, 2026
b134980
Merge remote-tracking branch 'origin/main' into GEN-2824-proxy-forwar…
Nov 12, 2026
0903772
GEN-2824 add unlock key rotation and reset
Nov 12, 2026
28ae6e2
GEN-2824 verify rotation preserves provider setup
Nov 12, 2026
31c6136
Bind model cache saves to key generation
Nov 12, 2026
344d8d6
GEN-2824 add owner and VM reflection boundaries
Nov 12, 2026
f772519
GEN-2824 refresh proxy security docs export
Nov 12, 2026
4f406ea
test credential proxy reconnect and destination boundaries
Nov 12, 2026
386381e
GEN-2824 package proxy sources without test fixtures
Nov 12, 2026
590dbb7
Add credential proxy image and scoped e2e fixture
Nov 12, 2026
1b20d10
Stabilize proxy grant refresh E2E
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions docs/fly-guest-transport.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,57 @@

The execution host must have a supported `fly` CLI installed and access to the selected Fly app and Machine. The token must authorize Machine execution and SFTP access. Machines must be running. The guest image must support Fly's Machine exec and SFTP paths; images do not need exe.dev's SSH tooling or systemd. `run()` accepts a nonempty executable and argv plus optional stdin bytes, returns captured stdout/stderr and the process exit code, and enforces a deadline of at most ten minutes and a combined output cap of at most 16 MiB. File operations enforce a configurable transfer cap of at most 64 MiB and use normalized absolute guest paths. Timed out operations terminate the CLI process group.

Both execution backends install and invoke the same small `factorize-job` guest launcher as `exedev`. It requires Python 3, the `exedev` account, and a writable `/home/exedev/.factorize-job` directory. It accepts a bounded JSON launch payload over stdin, starts the code script or agent argv without a shell command assembled from the payload, writes structured process state, preserves stdout/stderr and the run result, and exposes `factorize-job status` for Fly scheduler polling. Fly images also need `runuser`, `/bin/sh`, `/usr/local/bin`, Node.js and OpenSSL for portable proxy access, and the provider's documented Machine exec/SFTP support. Codex images must provide the tested `codex-cli 0.167.1`; Fly checks this as the `exedev` user before launch and returns a bounded failure for a missing or unsupported version. Code jobs do not require Codex. Image verification checks Python 3 for exe.dev execution; Fly reports a bounded launch failure when guest prerequisites are absent. The helper and job Machine are not configured to restart and replay a failed process. Custom entrypoints can finish their own setup before the backend invokes the launcher.

The local protocol fixtures verify process result handling, timeout, output overflow, disconnect, safe argv construction, and file byte transfer. They do not validate live Fly authentication, Fly CLI compatibility, routing to a real Machine, or the guest's exec/SFTP capabilities. A separate opt-in validation requires a Fly account, an authorized app/Machine and token, an installed Fly CLI, and a custom image with Fly exec/SFTP support; no Fly network call is part of the required test suite.

For portable integrations, the scheduler sends the scoped run credential over Fly's authenticated Machine exec channel directly to the dedicated credential proxy's private Unix socket. The proxy keeps it in process memory, bound to that run and provider VM identity, and removes it when durable revocation arrives. The credential is not part of Machine configuration, launch files, or the job VM payload. During guest launch, the VM generates its own private key and CSR; the proxy signs the CSR for the provider VM identity, and only the certificate and trust bundle return to the scheduler for installation. An internal-only Fly service exposes the proxy tunnel listener on port 9090. The guest starts a loopback CONNECT forwarder and sets HTTP(S)_PROXY for the job process. Each tunnel connection presents only its run ID over mTLS; the proxy retrieves its in-memory credential to perform fresh control-plane enrollment.

## Credential-proxy package and local evidence

`packages/credential-proxy` is a private npm workspace of native ES modules and
uses Node.js built-ins at runtime. Its package entry points are declared in
`package.json`; only `src/` is included in the package artifact, excluding the
test suite's synthetic TLS keys and fixtures. The Fly VM tunnel helper is generated from
`packages/api/src/fly-job-tunnel-helper.ts`. After `npm ci`, inspect the local
package contents without publishing with:

```sh
npm pack --dry-run --workspace=@factorize/credential-proxy --json
npm run test:proxy
```

The provisioner accepts only an image pinned by digest and equal to the reviewed
image supplied by deployment configuration. Proxy control credentials enter
through the private bootstrap socket; run credentials enter through Fly's
authenticated Machine exec channel and remain in proxy memory. Neither user
unlock keys nor integration credentials belong in image build arguments, image
configuration, guest environment variables, or launch payloads.

The minimal credential-proxy OCI recipe is `packages/credential-proxy/Dockerfile`.
It pins the Node 24.12.0 multi-platform manifest digest, installs the
authentication-only Codex CLI 0.167.1 from a checked-in lockfile, copies only
proxy runtime source, and runs as the unprivileged `node` user. Build locally
without publishing using `scripts/build-credential-proxy-image.sh`; Docker
Buildx emits an OCI archive with max-mode provenance and an SBOM under
`~/.factorize-job/credential-proxy-image/`, alongside a compact JSON summary
binding the archive digest to the source commit, base digest, and Codex version.
The OCI archive and generated provenance are inspectable/rebuildable artifacts;
this repository deliberately has no publishing workflow, and this local recipe
does not establish that any production digest has been published or approved.
The provisioner's reviewed-digest configuration must still be updated through
the separate release process after image review. Live Fly networking also
remains unverified.

Local coverage is split across fixtures to avoid repeating full application
startup: `packages/credential-proxy/test/fly-guest-helper.e2e.test.mjs` executes
the generated guest helper against an mTLS proxy and local HTTPS upstream and
rejects a mismatched VM binding; `proxy-listener.e2e.test.mjs` checks per-VM
reflection, locked access, configured-origin forwarding, redirect rejection,
interrupted writes, and reconnect after lock/unlock. These synthetic fixtures do
not establish live provider or production behavior. The combined
`locked scoped run reconnects with its current grant while another VM stays denied`
fixture exercises the production grant resolver with a synthetic Factorize run
snapshot and local TLS upstream. The repository runbook
`packages/docs/self-hosting/credential-proxy-security.mdx` documents key handling, lifecycle recovery,
accepted provider/control-plane risks, and the independent Fly shutdown steps.
10 changes: 10 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,15 @@
"engines": {
"node": ">=24.12.0",
"npm": ">=11.19.0"
},
"allowScripts": {
"esbuild@0.25.4": true,
"esbuild@0.18.20": true,
"esbuild@0.25.12": true,
"esbuild@0.28.2": true,
"esbuild@0.28.1": true,
"sharp@0.33.5": true,
"workerd@1.20251118.0": true,
"workerd@1.20260910.1": true
}
}
Loading
Loading