Skip to content

[AI-2346] Keep sandbox blind tunnels past client idle limits - #2161

Merged
kanat merged 10 commits into
mainfrom
ai-2346-tunnel-idle
Nov 21, 2026
Merged

kanat merged 10 commits into
mainfrom
ai-2346-tunnel-idle

Conversation

@kanat

@kanat kanat commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Why

A blind tunnel (a CONNECT to an allowed domain) closed after 90 s with no byte (idleTimeout, acceleration/internal/sandbox/egress/proxy.go:47 on base). That is the same 90 s as Go's http.DefaultTransport IdleConnTimeout. The proxy's clock starts when it reads the host's last byte, and the client's clock starts later, once that byte has crossed the network and the client has read it. So the proxy closed first while the client still pooled the connection. A POST the client wrote while the close was on its way failed with EOF, and net/http does not send a POST again (net/http/transport.go, shouldRetryRequest). #1513 (AI-2318) fixed the same race for kept connections with keepAliveTimeout (4 min). It left the blind tunnel at 90 s (AI-2346).

How it works

                 last byte from host
proxy reads it ──┬────────────────────────────────────────── time
                 │<-- latency -->│ client has it, pools the conn
                 │
before:          │ proxy closes at +90 s   │ client still pools until +90 s + latency
                 └──────────────── X ──────┼── a POST here goes into a closed tunnel
after:           │ client's limit (Go 90 s, libcurl 118 s) ends first
                 └──────────────────────────────── proxy closes at +4 min
limit before after
blind tunnel, no byte either way 90 s (idleTimeout) 4 min (keepAliveTimeout)
connection to the proxy / grant's tunnel with no next request 4 min 4 min (unchanged)
tunnel.py IDLE_AFTER_EOF (client stopped sending, proxy silent) 90 s, "the proxy's own idle timeout for a tunnel" 240 s, still the proxy's

Changes

  • proxy.go: the blind tunnel uses keepAliveTimeout for its idle limit. The idleTimeout constant and Options.IdleTimeout are removed. Now one limit, with one source, covers every connection the sandbox can pool. A tunnel that carries no byte still closes at that limit.
  • tunnel.py: IDLE_AFTER_EOF stays equal to the proxy's tunnel limit, as its comment says (240 s), so the forwarder does not cut an answer that the proxy would still send.
  • AGENTS.md, comments, and cmd/router/sandboxproxy_test.go (no longer sets the removed option).

Sibling sweep, every other limit in the package:

limit file verdict
handshakeTimeout 10 s proxy.go:44 the time to send a handshake or a request's headers once bytes have started. It is not an idle limit, so no pooling race
headerTimeout 60 s proxy.go time for the host's response header, so no pooling race
dryRunTimeout 30 s bigquery.go BigQuery dry run, so no pooling race
grant's tunnel IdleTimeout proxy.go tunnel already keepAliveTimeout
router's proxy server IdleTimeout cmd/router/sandboxproxy.go:100 already proxy.KeepAliveTimeout()
CONNECT_TIMEOUT 10 s tunnel.py opening the TLS connection, so no pooling race
IDLE_AFTER_EOF tunnel.py applies only after the client stops sending, so a pooling client never reaches it. Its stated rule follows the proxy, so it is raised to 240

Validation

  • New TestAGoClientReusesItsPooledTunnelToAnAllowedDomainUpToItsOwnIdleLimit (proxy_test.go): a Go client 600 ms of latency from the proxy (distantConn) pools its tunnel to an allowed domain whose host keeps the connection. It POSTs again 300 ms before its own idle limit. Durations are scaled so that client idle : proxy limit = 90 s : 4 min. Base with its own default ratio (proxy idle = client idle) fails 10/10 with Post "https://outside.test/echo": EOF. Base with the proxy at the 4 min ratio passes 5/5. Head passes.
  • The existing blind-tunnel tests (closed when nothing moves, kept open while bytes move, closed when an end reads nothing) now set the limit through KeepAliveTimeout, so a tunnel nobody uses still closes at the new limit.
  • TestTheDefaultLimitsAreTheDocumentedOnes pins 4 min and the forwarder's 240.
  • The new test passed 50/50 under -race -cpu 1 -count=50.
  • Mutations, each killed: blind tunnel at a fixed 90 s, at Go's ratio of the limit (3/8), or at handshakeTimeout; keepAliveTimeout at 90 s; the forwarder default at 90.

🤖 Generated with Claude Code

kanat and others added 4 commits November 20, 2026 16:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round-1 review at 83c3417, merged onto main 18eb148 in a scratch tree. CI: the PR is a draft, so the go, unit and integration checks were skipped. They run at gh pr ready.

Checked: the base repro fails 5/5 with Post "https://outside.test/echo": EOF, and the head passes. The new test passed 50/50 under -race -cpu 1 -count=50. The egress unit and integration suites pass (TunnelSuite included), and so does cmd/router -run 'Proxy|IdleTimeout'. A control probe printed the same limits on base and head: handshake 10s, keep-alive 4m, header 1m, dry-run 30s. Options.IdleTimeout and the 90 s constant have no other reader. tunnel.py's 240 s is at or above the proxy limit. Mutations: 7 checked, 1 survived (R1.1).

[Question] R1.2 proxy.go:249-255: nothing caps how many blind tunnels are open, per sandbox or in total. The router's proxy server has no LimitListener and no MaxConns, and egress keeps no per-session count. The idle limit caps only how long one silent tunnel lives, and that goes from 90 s to 240 s. Grant tunnels and kept proxy connections already live 4 min, so this is not a new kind of exposure. Ticket: decide whether to add a per-session tunnel cap.

Verdict: request changes, for R1.1 (a surviving mutation of the new limit).

// The tunnel's idle clock starts when the proxy reads a byte, before the sandbox's client has
// it, so its limit must outlast the client's pooling one as a kept connection's does: a
// client that pools the connection reaches its own limit first (AI-2346).
idle := &idleTunnel{timeout: p.keepAliveTimeout, ends: [2]net.Conn{conn, upstream}}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.1 The mutant timeout: p.keepAliveTimeout * 2 survives every blind-tunnel test and DefaultsSuite. With it, a tunnel nobody uses would close at 8 min, not at the new 4 min limit. TestATunnelToAnAllowedDomainThatCarriesNothingIsClosedAtTheIdleTimeout checks only >= idle and the 5 s read deadline (proxy_test.go:1132, 1150-1151). Fix: add an upper bound in that test, after s.GreaterOrEqual(hostEnd, idle):

s.Less(sandboxEnd, 2*idle, "closed at the idle timeout, not a multiple of it")
s.Less(hostEnd, 2*idle, "closed at the idle timeout, not a multiple of it")

Checked: head with this change passes 20/20, and 30/30 under -race -cpu 1. The * 2 mutant with this change fails 5/5 (closed at 0.60 s).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2d5fff2: TestATunnelToAnAllowedDomainThatCarriesNothingIsClosedAtTheIdleTimeout now asserts both ends close below 2x the idle timeout. The mutant timeout: p.keepAliveTimeout * 2 at proxy.go:255 fails it 5/5 (both ends closed at 0.60 s); head passes 20/20, and 20/20 under -race -cpu 1.

- A `Delete` ends the tunnels to a grant's host. A blind tunnel to an allowed domain carries no token and lasts until either end closes it, or until no byte has moved in either direction for the idle timeout (90 s).
- A `Delete` ends the tunnels to a grant's host. A blind tunnel to an allowed domain carries no token and lasts until either end closes it, or until no byte has moved in either direction for 4 min (`keepAliveTimeout`).
- A connection to the proxy, or a grant's tunnel, that sends no next request is closed after 4 min (`keepAliveTimeout`). That is past the longest idle limit of a client in the sandbox, libcurl's 118 s (curl, git), so the client closes it first. urllib3 (requests) keeps a connection with no limit: a request it sends on one just as the proxy closes it fails.
- A blind tunnel's idle limit is the same 4 min, for the same reason: its clock starts when the proxy reads the host's last byte, before the client has it, so a limit equal to the client's (Go's 90 s) closed the tunnel while the client still pooled it, and a POST the client sent on it then failed (AI-2346). The TLS tunnel (`tunnel.py`) waits as long for a proxy that stays silent after its client has stopped sending (`IDLE_AFTER_EOF`).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nit] R1.3 This bullet and line 76 both say the blind tunnel's idle limit is 4 min (keepAliveTimeout). Line 78 only adds the reason. Merge them (ticket).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 26cef9b: the two bullets are merged into one at AGENTS.md:76, which states the 4 min limit once.

kanat and others added 3 commits November 20, 2026 17:53
A tunnel nobody uses must close at the idle timeout. The test checked
only that it closed no earlier, so a limit of a multiple of it passed.
Assert both ends close before twice the idle timeout.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Two bullets said the same limit. Merge them into the one that
already names the blind tunnel.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@kanat
kanat marked this pull request as ready for review November 20, 2026 16:57
@kanat
kanat merged commit 175bce1 into main Nov 21, 2026
15 checks passed
@kanat
kanat deleted the ai-2346-tunnel-idle branch November 21, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant