Repository navigation
Sync upstream v1 APIs and sandbox lifecycle fixes - #97
Merged
Merged
Conversation
…a APIs (agent-substrate#1852) Fixes agent-substrate#1819 When bootstrap finds an existing cluster without the PodCertificate beta APIs, its reconcile path turns them on in place. That works: the APIs are served once the update finishes. On 1.36 node pools it is only half the repair, and nothing said so. Pod certificate projection is a kubelet feature, gated through 1.36, so pods scheduled onto nodes that were already running fail with `MountVolume.SetUp failed: unimplemented` while the control plane looks healthy. **What changes** - After a successful beta-API update, setup-gcp warns once per node pool below 1.37 and logs the two commands that fix it: - `gcloud container node-pools create <pool>-2 …`, with the machine type, disk, node count and nested virtualization copied from the old pool; - `gcloud container node-pools delete <pool> …` for once workloads have moved. The warning also says to carry over anything else customized on the old pool (taints, service account, autoscaling). The pool's version decides this, not the control plane's, since a 1.37 control plane can still run 1.36 nodes. - The README warning is rewritten to match, building on the `v1` discovery wording from agent-substrate#1924. It drops the claim that enabling the APIs on an existing cluster doesn't make them served, which didn't hold up in testing. - `hack/ate-dev-env.sh.example` had the same claim in a comment; fixed there too. **Why a new pool, not an in-place node recreation** The issue proposed recreating nodes, and the obvious command is upgrading a pool to the version it already runs. GKE skips that: in testing, the `UPGRADE_NODES` operation finished in 4 seconds and no node was replaced. A pool can only be upgraded forward when GKE happens to offer a newer patch. A new pool is the one GKE operation certain to produce new kubelets. Replacing the pool is left to the user rather than done automatically, because it moves running workloads. A flag to do it could be a follow-up if that would be useful. **Verified end to end** on a throwaway 1.36.4 cluster created without the beta APIs: 1. This branch's `create cluster` reconcile path enabled the APIs and logged the warning for `default-pool`. 2. A pod with a pod certificate volume on the existing node failed with `MountVolume.SetUp failed … unimplemented`. 3. The logged `node-pools create` command, pasted as printed, created a pool whose kubelet issued a PodCertificateRequest for the same pod (the mount then waited on `credential bundle is not issued yet`, as expected with no signer running). 4. The logged `node-pools delete` command removed the old pool. - [x] Tests pass (`go test ./tools/setup-gcp/...`) - [x] Appropriate changes to documentation are included in the PR
…ze for reviews (agent-substrate#3109) <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) This will help with PR reviews. cc @bowei <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I had AI critique it. I wrote this change. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
…nt-substrate#3082) Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) ateapi builds every durable snapshot URI with resources.NewActorSnapshotURI or NewTagSnapshotURI, but nothing stops a malformed one from being stored. Readers guarded against it instead: resume failed with DataLoss when the URI did not parse. atepg now rejects a CreateActor, UpdateActor, CreateTag or UpdateTag that would store a durable snapshot URI resources.ParseSnapshotURI rejects. The check lives in the store because workflows write through the store directly and never run declarative validation, and suspend and tagging are where URIs are built. With malformed URIs unreachable, checkSnapshotURI in workflow_resume.go becomes checkSnapshotUpload. It still fails a resume whose local snapshot records no upload. The tests that seeded malformed URIs are removed, and two revert test fixtures move to the current URI layout, which the new check caught. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude Code to write the change and the tests, and reviewed the diff myself. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
…uidelines (agent-substrate#3122) Removes long-outdated `docs/roadmap.md` and related references across the repository; short-term work is tracked directly in GitHub issues, and long-term planning will be refreshed with the community post-1.0. Clarifies security vulnerability reporting guidelines in `.github/SECURITY.md` to request specific sandbox reachability, exploitability paths, and security impact details while directing reporters to check open GitHub issues and avoid theoretical static analysis reports. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass: no code changes - [x] Docs updated
…ubstrate#3090) TestRunScriptLoopsUntilTheBudget counted 3 passes on a slow CI machine: the disk writes between sleeps pushed the loop past its budget before the fourth pass. On the bubble's clock a pass costs only its sleep, so the pass counts and elapsed times are exact and the tests assert them. This fixes a test flake. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes # <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [ X ] Tests pass - [ X ] Docs updated Code written with help of AI. I have reviewed it.
…nt-substrate#3036) The controller reads one Secret, egress-mitm-ca-pool in ate-system, but its RBAC marker granted get, list and watch on every Secret in the cluster. Add namespace and resourceNames to the marker. The rule moves from the ClusterRole to the ate-system Role, which is already bound. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes part of agent-substrate#2936 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I-assisted, I reviewed and tested it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [ ] Docs updated
…nt-substrate#3059) The Deployment and both of its RBAC bindings used the namespace's default ServiceAccount, so any pod in podcertificate-controller-system without a serviceAccountName inherited sign and attest on every Substrate signer. Add ServiceAccount/podcertificate-controller and point the Deployment, the ClusterRoleBinding and the coordinator RoleBinding at it. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes part of agent-substrate#2936 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI-assisted, I reviewed and tested it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [ ] Docs updated
TestActorArbitraryPortAccess has been flaking in e2e since the router moved to static-mtls in agent-substrate#2390, e.g. https://github.com/agent-substrate/substrate/actions/runs/39549120410/job/119054007125 client-go closes the whole port-forward when any one connection on it errors: https://github.com/kubernetes/client-go/blob/v0.37.0/tools/portforward/portforward.go#L549-L550 RouterClient shared one CONNECT port-forward for the life of the client. So one reset from the router failed the tunnel in flight and every Connect after it, which then got "connection refused" from the dead local port. agent-substrate#2961 fixed the same thing in ingressauth by forwarding per attempt. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes a trivial test flake bug, test only change. Spotted while reviewing PRs like agent-substrate#2502 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI-assisted, I reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
…agent-substrate#3063) A resetActorDirs failure is an error related to atelet or the node's disk, not the actor. Return Canceled so ate-api-server does not crash the actor. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [ ] Docs updated
…gent-substrate#3128) follow-up to agent-substrate#2963 (review) The imagecache comment and README say the minimum kernel is 6.5, but overlayfs's `lowerdir+` landed in 6.7 (24e16e385f22, "ovl: add support for appending lowerdirs one by one"), and was backported to 6.6.y in v6.6.23: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/overlayfs/params.c?h=v6.7#n148 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/overlayfs/params.c?h=v6.6.23#n148 6.5 had a different append syntax, `lowerdir=:dir`, but it was disabled before 6.6 shipped (beae836e9c61, "ovl: temporarily disable appending lowedirs") and we don't use it: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/overlayfs/params.c?h=v6.6#n375 AI use: AI-assisted, I reviewed it. - [x] Tests pass - [x] Docs updated
…strate#3094) Part of agent-substrate#2470, follow-up from [3010](agent-substrate#3010 (comment)). atelet can upload a suspended actor's latest local snapshot and die before it reports the upload. If the node is then deleted, the upload is complete but still recorded `PENDING`. Resume crashed the actor, and a revert then deleted the upload. Resume now releases the deleted node and restores the snapshot on another node. atelet there has no local copy, so it reads the upload. The manifest is written last, so a missing or partial upload returns `NOT_FOUND`. - `recordGoneNode` clears `assigned_node`. The snapshot keeps its old locality, so `snapshotToResume` still picks it. - A `NOT_FOUND` from that restore crashes the actor with the existing gone-node message. - `finalizeRunning` records the restored snapshot `COMPLETED` and clears its locality. - `canRelaxRuntime` lets the snapshot fall back to another runtime, as a durable snapshot can. No proto changes. Resume claims a worker before it knows the upload exists, so a missing upload costs one placement. A revert still deletes an unreported upload, so the snapshot is lost if its restore fails for another reason or the actor is reverted before it resumes. AI use: Claude Code helped write the change and tests. I reviewed them. - [ ] Tests pass - [x] Docs updated
…nt-substrate#2602) Once a checkpoint is underway on `atelet`, canceling it mid-flight (for example, when a caller with a shorter deadline times out and retries) kills `runsc checkpoint` while the Sentry continues stopping the workload, leaving the checkpoint incomplete and causing the retry to fail. Detach in-flight checkpoints from caller cancellation once past `lockActor` and `BeginCheckpoint`, keeping the later of the caller's deadline and `defaultCheckpointTimeout` (5m) so a hung worker or upload cannot hold the per-actor lock indefinitely without shortening a longer caller deadline. When a caller times out and retries, the original attempt finishes and commits under `lockActor`, and the queued retry answers from the committed snapshot (agent-substrate#2849). Part of agent-substrate#2836 AI usage: Used Gemini to help trace the checkpoint cancellation flow, draft the fix, and write unit tests. - [x] Tests pass
…be passes (agent-substrate#2117) An actor's egress was turned on only after every container answered its wakeup probe, so a workload that downloads what it needs to become ready (a model, a Git repository) could not start. atunnel closed the connection inside the sandbox before it reached the gateway. Egress is now turned on before the first container starts, on both sandbox classes and for both run and restore; ingress still waits for the probe. The atenet gateway admits `RESUMING` actors as well as `RUNNING` ones, and atunnel and the gateway now log the connections they refuse. The actor's EgressPolicy still applies. The new e2e test gives the golden actor its policy through the template's `goldenEgressPolicy` (agent-substrate#2159), so it runs only with the `GoldenEgressPolicy` preview gate. A restored actor's wakeup probe also waits for a fresh fetch as itself, which covers the restore paths. agentgateway moves to `v0.0.0-alpha.9d36620d`, which admits `RESUMING` actors too (agentgateway/agentgateway#3882), so the test runs on both dataplanes. Tested on kind with gVisor on both dataplanes, before the move to `goldenEgressPolicy`: the test passed with this change and failed without it (on agentgateway, against the previous pin). The rest of the gVisor e2e suite also passes on the agentgateway lane, apart from the external-volume tests, which could not mount NFS on my rootless setup. Part of agent-substrate#1543 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: QuentinBisson <quentin@giantswarm.io> Co-authored-by: Timo Derstappen <timo@giantswarm.io> Co-authored-by: Lior Lieberman <liorlieberman@google.com>
All public APIs now live in one tree, with a version in every package, so a future version can sit next to v1. * Proto packages are `substrate.<api-name>.v1`, so gRPC method names change, for example to `/substrate.ateapi.v1.Control/GetActor`. * Go packages for the protos are named after the API, for example `package ateapi`. * CRDs move from `ate.dev/v1alpha1` to `ate.dev/v1`. They live in `api/infra/k8s/ate/v1`. AI use: AI-assisted, I reviewed and tested it (test automation). Fixes agent-substrate#2753
Fixes agent-substrate#3033 k8s-credential-provider served its unauthenticated /statusz page. `--status-address` now defaults to 127.0.0.1:4040. The manifest passes the same value and no longer declares the statusz container port. To reach the page: kubectl -n ate-system port-forward deploy/k8s-credential-provider 4040:4040 **Checked on kind**: the node and the provider's own pod network get connection refused on <pod IP>:4040, and port-forward still serves the page. AI use: I used Claude Code to write the change and run the kind check, and reviewed the diff myself. - [x] Tests pass - [ ] Docs updated
…gent-substrate#3140) ### Summary Locust's `User Count` in `stats_history.csv` tracks spawned VU goroutines even when `CreateActor` fails or wedged actors are deleted during the run (for example, in resident density benchmarks where the target user count exceeds single-node capacity). * In `append_trial_summary` (`benchmarking/locust/cluster_facts.py`), when `CreateActor` is recorded in `stats.csv`, cap `peak_actors` by `CreateActor` successes minus unretired `DeleteActor` requests (excluding completed `TaskWallClock` trajectories in `sweperf` that retire their actor before starting a fresh one; `stats.csv` is written before boomer's shutdown `DeleteActor` sweep). * Update `test_live_actors_from_create_and_delete` in `benchmarking/locust/unit_tests/test_cluster_facts.py` to cover warmup retry deletes, 100% `CreateActor` failures, partial actor survival, and `sweperf` trajectory retirements. ### Testing ```bash python3 -m unittest discover -s benchmarking/locust/unit_tests -p 'test_*.py'
…nt-substrate#2534) ## Problem `classifyOutcome` in `cmd/atenet/internal/router/extproc/metrics.go` maps a `ReqError` by its HTTP status: 404 → `not_found`, 503 → `unavailable`, 504 → `timeout`, 429 → `rate_limited`. A 403 has no case and falls through to `resume_error`, which the registry describes as "the resume failed for a different reason. The router has a defect." Every egress policy refusal (destination outside the policy, no policy, actor not running, invalid actor certificate) and every ingress `PermissionDenied` therefore records `ate.router.outcome=resume_error` on `atenet.router.route.duration`, so an alert on router defects fires on ordinary policy decisions and a real resume failure hides among them. ## Change - New `ate.router.outcome` value `denied` (`ateattr.RouterOutcomeDenied`): a 403 `ReqError` and a `PermissionDenied` status classify as `denied`; everything else keeps its outcome, and `resume_error` is left for a resume that failed for any other reason. - Registry (`docs/metrics/registry/metrics.yaml`), the `ateattr` comment and the outcome list in `docs/observability.md` list `denied`. The `unavailable` outcome no longer names a denied egress request among its examples, since a denial answers 403. ## Testing - `metrics_test.go`: `PermissionDenied`, a bare 403 and a 403 wrapping an uncoded cause classify as `denied`; an uncoded error, an `Internal` status and a 500 `ReqError` still classify as `resume_error`. - `go test ./cmd/atenet/internal/router/extproc/ ./internal/ateattr/`, `hack/verify/golangci-lint.sh`, `hack/verify/metrics.sh`. This PR was written by an AI agent on behalf of the author.
…64 (agent-substrate#3149) On arm64 kernels 6.19 and newer, KVM clears the active bit on GICv3 LPI list registers during VM exits (Linux commit 73c9726975af, reverting that also works but isn't clearly the right fix). Under nested virtualization, hardware EOI in the L2 microVM guest does not clear ICH_AP1R0_EL2 and stalls subsequent virtio interrupts, hanging microVM boot. This was found by bisecting the kernel previously, which we "solved" by pinning to <= 6.18. This is commonly encountered when setting up limactl defaults, enabling nested virtualization, and then testing substrate on an apple silicon mac. Setting `kvm-arm.vgic_v3_group1_trap=1` on the Lima guest kernel command line traps Group 1 EOI writes to EL2 so KVM clears the active priority in software. ---- <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) This allows our developer docs to use the default Ubuntu 26.04 Lima image instead of pinning to Ubuntu 24.04 and installing a newer git from a PPA. FYI @sfunkenhauser I finally got back to this and found a suitable workaround. Still have not file the kernel bug thread, I haven't had time to figure out how to do that the right way without sending them unwanted ~slop ... we should do that at some point. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Yes, absolutely, I do not have time to bisect the kernel :-), I however vetted this change. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
EItanya
force-pushed
the
sync/upstream-main-20261023
branch
from
October 23, 2026 14:03
b989270 to
a51df70
Compare
EItanya
marked this pull request as ready for review
October 23, 2026 14:12
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261023
branch
from
October 23, 2026 15:41
a51df70 to
cffc7c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring the fork up to upstream
c0451cea0(18 commits), preserving both histories and the fork's agentgateway configuration.Includes versioned public APIs, network access while actors boot, restore from uploaded snapshots when a node is gone, checkpoint cancellation handling, and narrower controller Secret access. Migrate fork-only callers and synchronize Helm CRDs, admission policy, controller RBAC and its RoleBinding, and the credential provider's loopback status listener.
Pin the updated gateway, whose control and credential-provider bindings now use the
substrate.ateapi.v1andsubstrate.credprovider.v1service names. The old names caused the E2E failures withunknown service ateapi.Control. RPC trace names follow the same change. The image build passed for amd64 and arm64.Upgrade clients, control plane, workers, and plugins together. Kubernetes resources now use
ate.dev/v1; existing clusters need a storage-version migration before installing the v1-only CRDs. See the chart upgrade notes. Please apply thebreaking-changelabel.Validation: all 93 gateway Substrate integration tests, five protobuf contract tests, and 38 Helm tests pass locally. The two new service-name tests fail against the old bindings. Chart lint, manifest rendering, CRD parity, and controller RBAC/binding comparisons also pass. All five CI checks pass on
cffc7c7bc, including the full Go suite, repository verification, Helm validation, and gVisor/microVM E2E in standard, preview, and Helm installations. Full local Go validation hit the temporary filesystem limit, and local cluster E2E was not run.All integration fixes are folded into one signed-off merge commit. The sync skill now requires this for future sync PRs. Use a merge commit to retain upstream ancestry.
AI use: Codex prepared the merge, updated the gateway and Helm integration, and reviewed and tested the changes, including an independent agent review.