Repository navigation
[AI-2346] Keep sandbox blind tunnels past client idle limits - #2161
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kanat
left a comment
There was a problem hiding this comment.
Round-1 review at 83c3417, merged onto main 18eb148 in a scratch tree. CI: the PR is a draft, so the go, unit and integration checks were skipped. They run at gh pr ready.
Checked: the base repro fails 5/5 with Post "https://outside.test/echo": EOF, and the head passes. The new test passed 50/50 under -race -cpu 1 -count=50. The egress unit and integration suites pass (TunnelSuite included), and so does cmd/router -run 'Proxy|IdleTimeout'. A control probe printed the same limits on base and head: handshake 10s, keep-alive 4m, header 1m, dry-run 30s. Options.IdleTimeout and the 90 s constant have no other reader. tunnel.py's 240 s is at or above the proxy limit. Mutations: 7 checked, 1 survived (R1.1).
[Question] R1.2 proxy.go:249-255: nothing caps how many blind tunnels are open, per sandbox or in total. The router's proxy server has no LimitListener and no MaxConns, and egress keeps no per-session count. The idle limit caps only how long one silent tunnel lives, and that goes from 90 s to 240 s. Grant tunnels and kept proxy connections already live 4 min, so this is not a new kind of exposure. Ticket: decide whether to add a per-session tunnel cap.
Verdict: request changes, for R1.1 (a surviving mutation of the new limit).
| // The tunnel's idle clock starts when the proxy reads a byte, before the sandbox's client has | ||
| // it, so its limit must outlast the client's pooling one as a kept connection's does: a | ||
| // client that pools the connection reaches its own limit first (AI-2346). | ||
| idle := &idleTunnel{timeout: p.keepAliveTimeout, ends: [2]net.Conn{conn, upstream}} |
There was a problem hiding this comment.
[Should fix] R1.1 The mutant timeout: p.keepAliveTimeout * 2 survives every blind-tunnel test and DefaultsSuite. With it, a tunnel nobody uses would close at 8 min, not at the new 4 min limit. TestATunnelToAnAllowedDomainThatCarriesNothingIsClosedAtTheIdleTimeout checks only >= idle and the 5 s read deadline (proxy_test.go:1132, 1150-1151). Fix: add an upper bound in that test, after s.GreaterOrEqual(hostEnd, idle):
s.Less(sandboxEnd, 2*idle, "closed at the idle timeout, not a multiple of it")
s.Less(hostEnd, 2*idle, "closed at the idle timeout, not a multiple of it")Checked: head with this change passes 20/20, and 30/30 under -race -cpu 1. The * 2 mutant with this change fails 5/5 (closed at 0.60 s).
There was a problem hiding this comment.
Fixed in 2d5fff2: TestATunnelToAnAllowedDomainThatCarriesNothingIsClosedAtTheIdleTimeout now asserts both ends close below 2x the idle timeout. The mutant timeout: p.keepAliveTimeout * 2 at proxy.go:255 fails it 5/5 (both ends closed at 0.60 s); head passes 20/20, and 20/20 under -race -cpu 1.
| - A `Delete` ends the tunnels to a grant's host. A blind tunnel to an allowed domain carries no token and lasts until either end closes it, or until no byte has moved in either direction for the idle timeout (90 s). | ||
| - A `Delete` ends the tunnels to a grant's host. A blind tunnel to an allowed domain carries no token and lasts until either end closes it, or until no byte has moved in either direction for 4 min (`keepAliveTimeout`). | ||
| - A connection to the proxy, or a grant's tunnel, that sends no next request is closed after 4 min (`keepAliveTimeout`). That is past the longest idle limit of a client in the sandbox, libcurl's 118 s (curl, git), so the client closes it first. urllib3 (requests) keeps a connection with no limit: a request it sends on one just as the proxy closes it fails. | ||
| - A blind tunnel's idle limit is the same 4 min, for the same reason: its clock starts when the proxy reads the host's last byte, before the client has it, so a limit equal to the client's (Go's 90 s) closed the tunnel while the client still pooled it, and a POST the client sent on it then failed (AI-2346). The TLS tunnel (`tunnel.py`) waits as long for a proxy that stays silent after its client has stopped sending (`IDLE_AFTER_EOF`). |
There was a problem hiding this comment.
[Nit] R1.3 This bullet and line 76 both say the blind tunnel's idle limit is 4 min (keepAliveTimeout). Line 78 only adds the reason. Merge them (ticket).
There was a problem hiding this comment.
Fixed in 26cef9b: the two bullets are merged into one at AGENTS.md:76, which states the 4 min limit once.
A tunnel nobody uses must close at the idle timeout. The test checked only that it closed no earlier, so a limit of a multiple of it passed. Assert both ends close before twice the idle timeout. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Two bullets said the same limit. Merge them into the one that already names the blind tunnel. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Why
A blind tunnel (a CONNECT to an allowed domain) closed after 90 s with no byte (
idleTimeout,acceleration/internal/sandbox/egress/proxy.go:47on base). That is the same 90 s as Go'shttp.DefaultTransportIdleConnTimeout. The proxy's clock starts when it reads the host's last byte, and the client's clock starts later, once that byte has crossed the network and the client has read it. So the proxy closed first while the client still pooled the connection. A POST the client wrote while the close was on its way failed withEOF, and net/http does not send a POST again (net/http/transport.go,shouldRetryRequest). #1513 (AI-2318) fixed the same race for kept connections withkeepAliveTimeout(4 min). It left the blind tunnel at 90 s (AI-2346).How it works
idleTimeout)keepAliveTimeout)tunnel.pyIDLE_AFTER_EOF(client stopped sending, proxy silent)Changes
proxy.go: the blind tunnel useskeepAliveTimeoutfor its idle limit. TheidleTimeoutconstant andOptions.IdleTimeoutare removed. Now one limit, with one source, covers every connection the sandbox can pool. A tunnel that carries no byte still closes at that limit.tunnel.py:IDLE_AFTER_EOFstays equal to the proxy's tunnel limit, as its comment says (240 s), so the forwarder does not cut an answer that the proxy would still send.AGENTS.md, comments, andcmd/router/sandboxproxy_test.go(no longer sets the removed option).Sibling sweep, every other limit in the package:
handshakeTimeout10 sproxy.go:44headerTimeout60 sproxy.godryRunTimeout30 sbigquery.goIdleTimeoutproxy.gotunnelkeepAliveTimeoutIdleTimeoutcmd/router/sandboxproxy.go:100proxy.KeepAliveTimeout()CONNECT_TIMEOUT10 stunnel.pyIDLE_AFTER_EOFtunnel.pyValidation
TestAGoClientReusesItsPooledTunnelToAnAllowedDomainUpToItsOwnIdleLimit(proxy_test.go): a Go client 600 ms of latency from the proxy (distantConn) pools its tunnel to an allowed domain whose host keeps the connection. It POSTs again 300 ms before its own idle limit. Durations are scaled so that client idle : proxy limit = 90 s : 4 min. Base with its own default ratio (proxy idle = client idle) fails 10/10 withPost "https://outside.test/echo": EOF. Base with the proxy at the 4 min ratio passes 5/5. Head passes.KeepAliveTimeout, so a tunnel nobody uses still closes at the new limit.TestTheDefaultLimitsAreTheDocumentedOnespins 4 min and the forwarder's 240.-race -cpu 1 -count=50.handshakeTimeout;keepAliveTimeoutat 90 s; the forwarder default at 90.🤖 Generated with Claude Code