Repository navigation
Add native AWS store providers - #414
ChristianPavilonis wants to merge 22 commits into
Conversation
5dc18fe to
63892b0
Compare
# Conflicts: # crates/edgezero-cli/Cargo.toml
prk-Jr
left a comment
There was a problem hiding this comment.
PR Review
Summary
Reviewed at 3733482f. The commits since the previous changes-requested review (at c66f93f) only merge the stack and repair fixtures, so the blocking findings from that review still apply at this head. This pass adds one documentation blocker and several smaller items. Requesting changes because the earlier blockers are still open.
😃 Praise
- Every level of the bindings TOML is
deny_unknown_fields, including the internally tagged provider enums and the nested limit tables. The file-type check uses the opened handle, so there is no type race between checking and reading. - Every Axum feature combination compiles: none,
cli,native-bindings,axum, eachaws-*alone, andaws-stores. With no AWS feature, a bindings file fails closed withUnsupportedProviderbefore any local store opens. - The listener binds only after source validation, store preparation and the initializer complete.
Findings
Earlier blockers still open at 3733482f
- 🔧
run_appstill ignoresEDGEZERO__STORE_BINDINGS_FILE. Onlyrun_generated_appreads it (dev_server.rs:990vs:1017). The demo and theaxum.mdentrypoint still callrun_app. - 🔧 The AWS fixtures still fail on macOS.
cargo test -p edgezero-store-aws --all-features --libfailed in three runs at this head (2, 3 and 1 of 79 failed) withread request headers: Os { code: 35, kind: WouldBlock }. Commite9181716keeps the listener alive longer but does not make accepted streams blocking. - 🔧 Unchanged at this head:
- The
UPDATINGtable status is still rejected (dynamodb_kv.rs:687). - SDK connectors still accumulate under tight limits (
dynamodb_kv.rs:634). - The docs contradictions are still present.
- CI still runs no axum lib tests with an
aws-*feature. - The two open questions (local secret IDs sharing env variables, container pulls in the required CLI job) have no answer yet.
- The
New, blocking
- 🔧 The guide's
config pushcommand cannot reach the guard it describes (docs/guide/native-aws-stores.md:281, inline). This belongs with the existing docs blocker.
New, non-blocking
- 🤔 Secret name length is validated on a trimmed value, but runtime uses the untrimmed one (
native_bindings.rs:600, inline). - 🤔 The env-selected bindings file cannot use non-default trusted limits (
run_options.rs:293, inline). - 🤔 A compiled-out AppConfig provider is reported as a missing token (
run_options.rs:183, inline). - ⛏ Setting one timeout limit alone can be rejected because of another field's default (
settings.rs:187, inline). - ⛏ The CLI reference does not list the new flags.
docs/guide/cli-reference.mdhas no--adapter,--store-bindings,--target,--target-features,--target-all-featuresor--target-no-default-featuresforconfig validate, and no--store-bindingsforconfig push. It also does not mention that--adapter axumonvalidateswitches to bindings-only mode. This file is not in the diff, so the finding goes here. - ⛏ "Existing" wording in
dynamodb-kv.md(docs/guide/dynamodb-kv.md:18, inline). This is a sibling of the earliernative-aws-stores.mdwording comment. - 📝 Addition to the earlier read-cap comment (
dynamodb_kv.rs:357).get_byteschecksmax_value_bytesbefore the expiry filter (:354-367). A row that is over a lowered cap and has also expired therefore returnsUnavailableinstead of reading as absent, until the DynamoDB TTL sweep removes it. Capping reads at the fixed 393,216-byte maximum, as already suggested, fixes both cases. - 📝 The CodeQL clear-text logging alert on
scripts/native_aws_test_admin.mjslooks like a false positive. The script only accepts a literalhttp://127.0.0.1:<port>endpoint with a dummy SigV4 header. The logged result holds hard-coded fixture values, and stdout is the IPC channel to the Rust test. Dismissing it as test fixture data seems reasonable.
CI Status
- GitHub Actions at
3733482f: all checks pass. - Local, macOS, at
3733482f:edgezero-store-aws --all-features --libfails intermittently, as described above. cargo checkacross the Axum feature matrix: passes.- Axum lib
native_/run_optionstests withaws-stores,aws-dynamodb,test-utils: 70 passed. - CLI lib bindings/push tests: 10/10.
- Node
native_aws_*tests: 16/16. - Not run: the Docker emulator suite, the ignored generated-build proofs, and anything against real AWS.
|
|
||
| ## Push remains local | ||
|
|
||
| `edgezero config push --adapter axum` writes local config snapshot files. It does not publish |
There was a problem hiding this comment.
🔧 This command cannot reach the push guard described here
The bundled edgezero binary parses config push as ConfigCmd::Push(ConfigCmdStubArgs) (args.rs:100). That stub accepts any flags, --store-bindings included, and run_config_push always returns the "requires the typed downstream CLI" error (config.rs:314-322). cli-reference.md says the same.
Scenario: an operator copies edgezero config push --adapter axum --store-bindings /abs/bindings.toml. The command exits with the stub error. They never see the AWS-binding refusal, and nothing is written locally either.
Fix: use the typed CLI form here, as blob-app-config-migration.md does:
<your-app>-cli config push --adapter axum --store-bindings /absolute/path/bindings.toml| } | ||
| if kind == "secrets" && source != NativeBindingSource::Supplied { | ||
| let namespace = if source == NativeBindingSource::Local { | ||
| env.store_setting(kind, id, "name").map_or(id, str::trim) |
There was a problem hiding this comment.
🤔 The length check measures a trimmed name, but runtime uses the untrimmed name
For local secret IDs this checks store_setting(...).map_or(id, str::trim) against MAX_NAME_LEN. The runtime store is built with options.env_config().store_name("secrets", id) (dev_server.rs:1467). EnvConfig::store_name (env_config.rs:170-174) filters out only blank and control values; it does not trim.
Scenario: EDGEZERO__STORES__SECRETS__SIGNING__NAME holds 512 bytes plus a trailing space. Startup accepts it and reports Ready. Then every SecretHandle::get_bytes call fails validate_name with 513 bytes. The fail-fast this check exists for (secret_namespace_limits_are_checked_before_preparation) becomes a failure on every request.
Fix: measure what runtime uses, env.store_name(kind, id).len(), or trim in both places.
| } | ||
| } | ||
| } | ||
| if let Some(path) = options.env.get(&["store_bindings_file"]).map(str::to_owned) { |
There was a problem hiding this comment.
🤔 With an env-selected bindings file, the trusted file limits cannot be changed
from_vars selects the bindings file as soon as the locator is present, using the default NativeBindingsLimits (256 KiB, 128 IDs). Any later with_native_bindings_limits call is rejected with "limits must precede selection" (:470-487). The NativeBindingsLimits doc presents these as host-controlled trusted bounds. On the env-located path, which from_env, run_production_app and run_generated_app all use, the host has no way to set them. A larger deployment has to drop from_env and rebuild every hosting setting by hand.
Fix: record the locator in from_vars and read the file lazily, either at plan validation or the first bindings() call. Alternatively, add a from_env_with_limits(limits) constructor.
| } | ||
| let mut options = Self::from_vars(pairs)?; | ||
| options.non_unicode_overrides = non_unicode; | ||
| // Only this env-loading boundary may resolve named token variables. |
There was a problem hiding this comment.
🤔 A compiled-out AppConfig provider is reported as a missing token
capture_agent_tokens runs on the env-loading path whatever features are compiled. Suppose a binary built without aws-appconfig-agent loads a bindings file with [config.x] provider = "aws-appconfig-agent" and access_token_env = "T", and T is unset. Startup fails with the Agent-token-absent error before the capability check (native_stores.rs:258-271) can report "provider not compiled". That sends the operator after a token, when the actual fix is a Cargo feature.
Fix: skip token capture when !cfg!(feature = "aws-appconfig-agent"), or run the provider-capability check before capturing tokens.
| if self.max_value_bytes == 0 | ||
| || self.max_value_bytes > 393_216 | ||
| || self.connect_timeout_ms == 0 | ||
| || self.connect_timeout_ms > self.attempt_timeout_ms |
There was a problem hiding this comment.
⛏ Setting one limit alone can fail because of another field's default
Unset fields are filled in by #[serde(default)], then connect <= attempt <= operation is enforced. Setting only operation_timeout_ms = 1500 keeps the default attempt_timeout_ms = 2000, so validation returns InvalidLimits without naming any field. AwsPreparationLimits::validate has the same behaviour (:475-477).
Fix: document the ordering rule next to the limits, or clamp unset attempt and connect defaults to the configured operation timeout.
| features = ["aws-dynamodb"] | ||
| ``` | ||
|
|
||
| `aws-stores` retains its existing AppConfig Agent and Secrets Manager membership. |
There was a problem hiding this comment.
⛏ "aws-stores retains its existing AppConfig Agent and Secrets Manager membership". aws-stores is introduced in this PR, so it has no earlier membership for readers to rely on. This is a sibling of the wording comment on native-aws-stores.md. Suggest: "aws-stores enables AppConfig Agent and Secrets Manager only."
|
CREATOR: From the server-side identity plan (IABTechLab/trusted-server#1559, Part of #488 |
Brings #275 at e9ffaa8 and the refreshed heads of #499, #460, #406, #407, #409, #414, #422 and #439. Each PR merged the one below it, so this merge has no conflicts. Two edits fit #469 to #275, as #469's open review asks: - #469's ingress JA4 test calls into_core_request with one argument. #275 now passes the Axum outbound executor through that function, so the test gets an executor handle the way the other ingress tests do. - The Axum guide names EdgeZeroAxumService, which #275 removes and check_outbound_legacy_api.sh rejects. The guide now names only AxumDevServer::with_trusted_ja4_header.
Current review readiness
Ready for review. Required current-head CI passed at
c66f93f118141a2ed7469a8ab96b12902f3854e6: test run 42924131967.Minimal fixes originate in #414 and are merged upward:
Deadline; non-timeouts remainUnavailable. The previously failing hosted timeout test and all 78 provider cases now pass without assertion, retry or duration changes.aws-types =1.6.0andaws-smithy-runtime-api =1.19.0. This prevents the schema mismatch and preserves the explicit older behavior version instead of switching retry policy or suppressing deprecations. A fresh consumer passes with compiler and Clippy warnings denied throughout.Local final-top Axum checks pass 488 unit, 28 contract, six AWS startup, four celld startup and 36 hosting cases. Final-top workspace strict Clippy and formatting pass. The prior stable checkpoint passed all six mandatory commands; unchanged inputs retain their receipts, while fresh current-head CI checks the combined state. The current ready state supersedes historical draft-only prerequisites below; unexecuted deployment acceptance is not claimed complete. An additional Fastly WASM runtime job was still running at transition and is not counted as passed. No TS work or deployment qualification is a review prerequisite. Integration #462 remains draft; reviewers
aram356,prk-Jranddhruv8shremain requested. Stack465 and PR bases are unchanged; #453 is excluded.Current review-stack checkpoint
Publication head:
c66f93f118141a2ed7469a8ab96b12902f3854e6. This cleanup preserves the existing feature unit and PR identity. The additional change is the completed generated-native build correction: image-only code now follows the scaffold's existing Fastly-example selection, with no request-policy change. Its focused generator/native proof is retained; fresh per-branch hosted checks are not assumed green.Review order: #275 → #406 → #407 → #409 → #414 → #422 → #439 → #454. Draft integration checkpoint: #462. New SDK/group/codec/phase development and unrelated TS work are not prerequisites or additions to this PR. Earlier notes below retain their original tested revisions and limits; they are not a whole-tree pass for this new head.
Earlier implementation and proof notes
RC refresh checkpoint, 2026-11-10
Current head:
48761e3e528eae0481e9f6e2eb30c0002f00d239.Only the AWS provider delta is replayed onto the updated hosting/diagnostic stack. Duplicate historical foundation/hosting commits are omitted. Explicit production options and initializer startup have a TCP no-ambient-merge regression. Ordinary test PR coverage remains unfiltered; an old donor filter was removed again.
Feature proof passes 417 Axum unit, 5 AWS startup, 33 hosting, 74 AWS provider, 256 CLI unit and 15 Node runner cases, scoped strict Clippy and SDK-isolation checks. Parent namespace/CI composition reruns pass all 5 AWS startup, 33 hosting and 35 Node CI cases. All five tracked locked fetches pass.
Generated native consumer/DynamoDB and pinned disposable-service runtime proofs are still pending integration execution. Bounded retained SDK values are not an RSS cap; timeout cancellation does not prove blocking credential work stopped. AWS clients intentionally retain their normal credential chain, separate from explicit hosting-source selection. No cloud acceptance is claimed.
Draft pending fresh hosted checks and the combined RC audit. Two unchanged native default-parallel deadline misses remain recorded and unclassified; serial passes and untouched-foundation parallel passes do not clear them. No timing bound was relaxed. No deployment, main merge, security-alert dismissal or review-thread resolution occurred. Historical design/review context follows.
Summary
Sendsetup futures.Design and boundaries
This adds native store providers, not a new HTTP runtime.
edgezero-store-awsimplements the existing core store traits and does not depend on Axum. Axum owns provider selection and startup composition; core owns portable handles, per-ID registries and typed configuration resolution.The portable
edgezero.tomldeclares logical store IDs and defaults. AWS resource references belong in a separate versioned native bindings document, selected through an absolute file path inEDGEZERO__STORE_BINDINGS_FILEor through explicitAxumRunOptions. Applications can also supply ready handles or non-Senddeferred setup, without a plugin registry.Each declared ID selects one supplied, explicitly bound or implicit local source. Supplied/file collisions and undeclared IDs are rejected. The runner validates all sources before opening runner-owned local stores or polling deferred setup.
CONFIG_DIR/DATA_DIRare required only for remaining local bindings. An unavailable selected provider fails startup; it never falls back to local data.ConfigStorereadsSecretStorereadsKvStoreoperationsflowchart TD Bindings[Bindings and supplied inputs] --> Validate[Validate sources and compiled providers] Validate --> Prepare[Assemble app and prepare stores in the native runtime] Prepare --> Registries[Complete per-ID registries] Registries --> Initialize[Run app-owned initializer when supplied] Initialize --> Serve[Bind listener and serve requests]One shared AWS preparation session owns startup budgets, client reuse and snapshot deduplication. The initializer receives the same registries used by requests. The application chooses its required config keys, schemas, secret checks and retained startup state; the framework does not infer those requirements or promise an atomic revision across remote resources.
AWS features are default-off and native-only. Existing entrypoints retain their local defaults; newly generated entrypoints select strict startup only when the bindings-file locator is present. Trusted Server adoption is separate work, and this does not claim a source-compatible upgrade from older EdgeZero versions across the #275/#406 prerequisites.
Stack and scope
Existing draft stack member based on
docs/outbound-http-spec/ #275 atea66eff045512cccc60646d6ec339e83ff44fed7, with the compatible #406 production lifecycle bridge. Neither prerequisite PR was rewritten or merged. The user approved adding DynamoDB on this same branch/PR and committing/pushing it after checks.Real AWS/IAM/capacity/deployment/image acceptance remains explicitly excluded. No real AWS service calls, infrastructure changes, image pulls or deployments were performed. SDK fixtures use dummy credentials, isolated or bypassed profile discovery, and literal-loopback endpoints.
DynamoDB KV
dynamodb-kvprovider / generatedaws-dynamodbfeature; existingaws-storesremains AppConfig Agent + Secrets Manager only.[kv.<id>]binding with explicit region/table/physical namespace, no credential or endpoint fields. Normal operator-owned SDK credential/endpoint policy remains supported.pk/sktable; bounded DescribeTable preflight, exact table-name/schema checking, no provisioning or local fallback. All-remote KV needs noDATA_DIR.DynamoDB is live storage, not a retained startup snapshot. The existing default 8 MiB retained-payload cap is per shared config/secret preparation session, not process/RSS or live database contents. Config/secrets still adopt remote changes only on restart; no live config/secret refresh or mutation is added.
Bindings validation remains structural-only / provider availability unknown. No offline Cargo resolver or guessed availability verdict is shipped. Custom shell actions remain untouched.
Verification
Original snapshot phase: all 24 post-rebase gates passed at its recorded source checkpoint; historical evidence remains in the original spec/plan and
/tmp/native-store-integration/final-checks/. This does not claim those old results prove the later DynamoDB edits.DynamoDB extension: 20 code gates plus docs lint/full-format/build passed, against an archived exact staged source tree independent of eight pre-existing dirty files. SHA-256 input manifests matched; later publication-record edits are Markdown-only.
aws-storesmembership, Cloudflare/Fastly/Spin WASM checks.Unavailable, one attempt and the retained/removed row. SDK model Debug disclosure is directly asserted.Senddeferred source validation without polling/opening another local store; existing generic native staging atomicity/drop tests preserved.edgezero serveinvocation; shared argument/action selection has unit/source proof, generated build and startup are executed.An inherited Secrets timeout test transiently returned
Unavailableinstead ofDeadlineunder the combined parallel run. It passed alone, then all 74 provider tests passed serially. The initial failure log is retained; no production/unrelated test was changed to hide it.Commands/fingerprints:
/tmp/dynamodb-baseline/final-checks.sh,/tmp/dynamodb-baseline/final/,snapshot-{before,after}.sha256; checked source tree2fa76fb26377cf41203c255f924d988d6b657002. Final read-only provider review found no confirmed defects and is separate from executed checks.Closes
Closes #410
Checklist
Sendsetup preservedFixture lifetime follow-up
Commit
e9181716975c068cd6894e046eec1ee3e194ad69keeps the loopback endpoint alive until its SDK child completes. A local reproduction of the hosted timeout failure identified a genuine ConnectionRefused after the one-response worker dropped its listener while the SDK still owned retries. The deterministic endpoint-lifetime regression fails before this fix. All 79 AWS tests, strict all-feature/all-target AWS Clippy, and 30 repetitions of the original timeout test pass afterward. This changes only the test fixture. Production timeout limits, retry policy and typed error classification are unchanged. Current-head hosted qualification is running; no release tag is claimed.Owned follow-up review checkpoint
4adeb228ebcc0839f5306e16be6ea9b963e97f36. Source tree224f1f81634a5871a05b569939b13a4f02606b0dis byte-identical to previous head3733482f6bedcaecb162b1e927209f3800f8b388.