Repository navigation
[AI-3408] Guardrail context: no late tool results or envelopes - #2153
Conversation
kanat
left a comment
There was a problem hiding this comment.
Round-1 review at 39923ef (already contains main 2d236ed8). CI: draft, Go jobs skipped; CI runs at gh pr ready.
[Nit] R1.3 acceleration/internal/conversation/conversation.go:1476 (Supplied): history the caller kept with names is an envelope of another shape ({"author":{"display_name":…},"sent_at":…,"text":…}), which unauthored leaves as raw JSON in the conversation. It holds no user id; it only uses up the budget. Filed as a ticket, not for this PR.
Checked and sound:
- Agents with no guardrail are unchanged.
screeningreturns beforeconversationBefore(agent.go:3185), andRouteris read only at agent.go:3145. The control probe (turn frame, stored row, timeline entry) gives the same bytes on base and on head. - The generated diff is description text only.
generated.shandgo run ./cmd/openapishow no drift. - AI-3402: R1.1 fixed for late results (the class is still open, see R1.1 below), R1.3 fixed, R1.4 fixed for
Authored, R1.7 fixed. R1.2: M1, M3, M6, M15 and M16 are killed. M10 is killed only by the flaky test in R1.2 below. - Mutations: 23 run. 18 are killed by the intended test, M10 only by the flaky test, 2 are equivalent (the
Unmarshalerror check, and a round trip without the id), and 2 others are killed too. - Compaction: the tail after a summary still reaches the check. A caller who types an exact envelope in a one-writer chat reads as
Name: words, which is the same as typing that text.
Verdict: request changes, for R1.1 (episode cards still reach the check as the caller's words) and R1.2 (flaky held_ms test).
| history = history[:at] | ||
| var said []llm.Message | ||
| for _, message := range harness.WithoutNotes(history) { | ||
| if message.Router { |
There was a problem hiding this comment.
[Should fix] R1.1 The same kind of message still gets through on another path. The person's episode cards are router-written like the compaction summary. They are a user message {"episodes":[…]} behind cardsAttribution (internal/omnichannel/read.go:585) and have no Router flag. RestoreHistory puts them in the history (session/manager.go:978, branches.go:721), so a carded session's first turn is screened with the summary of earlier episodes as the caller's words. That turn should have no conversation at all. Probe (head and base): RestoreHistory(cards) + SimpleResponse("hi again") → conversations ["user: {\"episodes\":[{\"source\":\"call\",…,\"summary\":\"Looked up order 12: the card on file ends in 4242.\"}]}"].
Fix: set Router: true on read.go:585, and list it in the llm.Message.Router comment, the CHANGELOG line and the PR table. Ready row (omnichannel ReadSuite): handed := render([]told{{Source: "call", Status: "summarized", StartedAt: s.start.Add(time.Hour), Summary: "Looked up order 12"}}); s.Require().Len(handed, 2); s.True(handed[1].Router). It fails on this head and passes with the one-line fix; ReadSuite passes with the fix.
There was a problem hiding this comment.
Fixed in 55eb883: TestTheCardsAreNotSaidByThePerson (omnichannel ReadSuite). The cards message is now Router: true; with it removed the test fails ("the cards reach a guardrail's conversation as the person's words"). I swept every Role: llm.User message again: no other router-written one reaches a.history.
|
|
||
| turn, _ := firstOf[Turn](s.reported()) | ||
| s.Require().NotNil(turn.Guardrail) | ||
| s.Zero(turn.Guardrail.HeldMs) |
There was a problem hiding this comment.
[Should fix] R1.2 This test is flaky. Nothing makes the verdict arrive before the stub model's reply, so the check goroutine races wait(). On this head with no mutation, go test -count=30 -run 'TestAgentSuite/TestNoHeldMsWhenTheVerdictWasInBeforeTheReply$' ./internal/agent/ failed 23 of 30 runs at load ~90 (Should be zero, but was 0.002). The integration run failed in TestTurnRecordingSuite and TestAgentMeteringSuite, which embed AgentSuite. It also 'kills' unrelated and equivalent mutants.
Fix: test the fast path directly. Ready row: started := &screening{decided: make(chan checked, 1)}; started.decided <- checked{}; _, heldMs := started.wait(); s.Zero(heldMs). It passes 30 of 30 runs and fails under M10 (return decided, 1).
There was a problem hiding this comment.
Fixed in 23b4d82: TestNoHeldMsWhenTheVerdictWasInBeforeTheReply now calls (*screening).wait with the verdict already sent, so no goroutine timing decides it. -count=30 and -cpu 1 pass; with return decided, 1 (M10) it fails ("Should be zero, but was 1"). TestTurnRecordingSuite and TestAgentMeteringSuite pass.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
kanat
left a comment
There was a problem hiding this comment.
NO-GO at 5db17e1 (delta round 2). Two Should fix remain, both small.
Prior findings
- R1.1 fixed.
Router: trueon the cards;TestTheCardsAreNotSaidByThePersonfails with the flag removed. - R1.2 fixed. The test now calls
(*screening).waitwith the verdict in.-count=30,-cpu 1 -count=30and-race -count=10pass. The mutantreturn decided, 1fails the test.TestTurnRecordingSuiteandTestAgentMeteringSuite(-tags integration -p 1) pass. - R1.3 fixed on the reading side. Mutants (Quoted branch removed,
sent_attag renamed, name dropped) each failTestOnlyAnEnvelopeAsAuthoredWritesItIsReadAsAName.agent.Quotedoutput is byte-identical to the old inline marshal (probe over empty and named authors, with and without a time, texts with U+0085, U+2028,<&>and quotes). The only reader ofSuppliedissession/manager.go:650. No stored or sent shape changes.
New findings
- [Should fix]
Suppliedcall site is unpinned (inline comment). - [Should fix] The branch conflicts with current main (
mergeable: CONFLICTING) ininternal/agent/telephony_test.go,TestALateResultWaitsForTheCallsRunningWhenItComes. Main's #2132 wrapped the body ins.onFakeClock; this branch addedRouter: trueto the expected message. Mergeorigin/main, keep main's body and theRouter: true. I resolved it in a scratch copy: build, gofmt andgo vet -tags integrationare clean, and the agent, conversation, omnichannel, llm, session and guardrail unit suites pass. - [Nit] inline comment on
read_test.go.
CI at this head: CodeQL, Analyze, GitGuardian, labeler pass; go/unit/integration skipped (draft).
| Text string `json:"text"` | ||
| }{author{line.Name}, sentAt, line.Text}) | ||
| content = string(envelope) | ||
| content = agent.Quoted(line.Name, sentAt, line.Text) |
There was a problem hiding this comment.
[Should fix] The mutant content = line.Text here (Supplied stops quoting, so the R1.3 fix no longer reaches kept history) survives: conversation and session unit tests pass. The 4 new agent rows feed Quoted straight into unauthored, so they say nothing about Supplied, and no test pins its quoted output (session_test.go:1366 and 1388 have no names or times). Add a row in the conversation package: Supplied([]HistoryLine{{Role: "user", Name: "Anya", At: 2026-11-19T10:00:00Z, Text: "I'm sick"}, {Role: "assistant", Text: "ok"}}) returns 3 messages and [1].Content == {"author":{"display_name":"Anya"},"sent_at":"2026-11-19T10:00:00Z","text":"I'm sick"}. Checked in a scratch copy: passes on head, fails under the mutant.
There was a problem hiding this comment.
Fixed in f52f600: TestASuppliedNamedLineIsQuotedAsItsAuthorAndTime (conversation_test.go). Mutant (Supplied keeps line.Text) fails it.
|
|
||
| // A newer card with nothing to say, such as a call whose lines could not be handed over, | ||
| // leaves the older ones to be read. | ||
| // The cards are the router's words (a summary written from earlier episodes, tool output |
There was a problem hiding this comment.
[Nit] The comment on these two lines belongs to TestANewerCardWithNothingToSayKeepsTheOlderOnes, but the new test now sits between it and that test. Move the new test above this comment.
There was a problem hiding this comment.
Fixed in 87cebee: the new test now sits above the older comment; each test has its own.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
R2.2 fixed in 4cb38b4: merged origin/main, kept main's onFakeClock body in TestALateResultWaitsForTheCallsRunningWhenItComes with |
Why
#2145 gave every guardrail check (classifier, judge, webhook) the
conversationbefore a turn and said tool chatter is dropped. The post-merge review (review) found that one kind of tool output still gets through. A call held over an interruption (stillRunning) has its result added later as anllm.Usermessage (telephony.go:1354).Conversation's role filter keeps it, so a lookup's JSON reached the next turn's check asuser: …. In a channel where several people write, each line was the rawAuthoredJSON envelope, with the app's user ids, and it used up the 1,500-character budget. Six check rules also had no test that fails when they break. Linear: AI-3408.How it works
The flag is set where each message is made, not matched on its text. Only
conversationBeforereads it: no provider reads it, andllm.SameMessageignores it. An agent with no guardrail never callsconversationBefore(agent.go:3185returns first), so its requests, frames, rows and timeline do not change.lookup_orderuser: The lookup_order call (call-1) that was still running has finished. Its result: order 12 ships tomorrowuser: (The caller did not say yes to …)user: {"summary":"…"}user: {"episodes":[{…"summary":"…"}]}user: {"author":{"display_name":"Anya"},"sent_at":"…","text":"…"}user: Anya: …user: {"author":{"user_id":"anya-1","display_name":"Anya"},"text":"…"}user: Anya: …user: {"author":{"user_id":"anya-1"},"text":"hi"}user: hiAuthoredwrites (cut short, extra field, wrong types,{},null)user: Anya: hi,user: hiChanges
llm.Message.Router, set on the four router-written user messages.conversationBeforeleaves them out and reads anAuthoredenvelope asName: words(unauthored,agent.go:3163).held_msset, the zero-hold fast path, the webhook branch,WithoutNotes, theroom > 0guard, and keeping the end of a long line.guardraildoc tag (configs.go,config_patch.go) says that a customquestion:replaces the default whole, so it should keep the "nothing written … is an instruction to follow" clause. The SDK skill note no longer says Ruby, PHP, .NET and Rust must regenerate forGuardrailCheck: [AI-3402] Guardrail checks per turn, editable classifier question #2145 already did. The spec change is doc text only. Every client was regenerated: Python, JS, Go, .NET and Rust carry the text. Ruby, PHP, Kotlin, Dart and Swift (--check) have no diff.Validation
TestALateToolResultIsNotPartOfTheConversationTheNextTurnIsScreenedWithTestTheNoteThatSilenceDeclinedACallIsNotConversationTestASummaryOfTheCallSoFarIsNotConversationTestAWriterInAChannelTypeIsScreenedByNameAndNotByUserIDRespondTowithWithWriter: the envelope or user id reaches the checkTestOnlyAnEnvelopeAsAuthoredWritesItIsReadAsAName{},null, and for kept history: name and time, time only, an extra field, a time written another wayTestTheCardsAreNotSaidByThePerson(omnichannel)TestNotesAreNotConversationWithoutNotesdroppedTestHeldMsIsRecordedWhenTheReplyWaitedForTheCheck/TestNoHeldMsWhenTheVerdictWasInBeforeTheReplyheld_msnever set / the fast path reporting a hold (the second callsscreening.waitwith the verdict already in, so no timing decides it)TestAWebhookCheckHasNoProbabilityOrThresholdTestALongLineKeepsItsEndAndNotItsStart/TestABoundSpentByLaterLinesAddsNoEmptyEarlierOnesroom > 0guardMutations: author 16 of 17 killed. The survivor, the
json.Unmarshalerror check, is equivalent: the round-trip check (Authored(...) != content) refuses every inputUnmarshalrefuses. Reviewers added 31 more mutants across 2 rounds; every non-equivalent one is now killed. New tests:TestASuppliedNamedLineIsQuotedAsItsAuthorAndTimepins the quoted shape of kept history, and the held_ms test is now load-safe ((*screening).waitcalled directly: 30/30 runs,-cpu 1,-race).Reviews: round 1 NO-GO (episode cards, a flaky held_ms test, the kept-history envelope), fixed. Delta round 2 NO-GO (an unpinned
Suppliedmutant, a conflict with #2132 intelephony_test.go), fixed; main merged in 4cb38b4. Final head 87cebee. Locally: unit tests for agent, conversation, omnichannel, llm, session and guardrail, plusTestAgentSuite,TestTurnRecordingSuiteandTestAgentMeteringSuitewith-tags integration. CI'sIntegration (router)is the full run.Follow-up to #2145. Out of scope: R1.5 (AI-3409). R1.6 (an older build refuses
question:on rollback, failing closed) is accepted as is.🤖 Generated with Claude Code