Skip to content

Avoid echoing denied shell commands into model context - #256

Merged
qduc merged 3 commits into
mainfrom
codex/efficient-file-guidance
Nov 12, 2026
Merged

qduc merged 3 commits into
mainfrom
codex/efficient-file-guidance

Conversation

@qduc

@qduc qduc commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

An automatically denied shell command is already present in the tool call, but the non-interactive policy repeats the entire command in the rejection reason sent back to the model. A long heredoc can therefore be carried twice through subsequent requests. Keep the RED/YELLOW decision and explanation while omitting the duplicate command from generated heuristic reasons. Human-supplied reasons are unchanged. The CLI retains its existing tool preview, and the full call remains in JSON events and session history; the rejection line itself no longer contains the full command.

This is a bounded context reduction, not proof of end-to-end parity with Pi. In one concurrent Sonnet comparison, recorded heuristic-rejection strings totaled 130,993 characters across baseline requests and 8,232 with this change. Both Term2 samples made seven denials and 27 requests. Across three concurrent Term2 pairs, generated tokens were 16,976 / 20,010 / 15,276 for baseline and 17,998 / 17,190 / 13,946 with the change. The small sample is variable; it does not establish a causal or quality-preserving improvement.

In the latest comparisons, all outputs built, typechecked and passed the two recovered ACP wire regressions. One Term2 baseline and the Pi control passed 18/18 rebuilt classification cases; both changed outputs and the other baseline passed 16/18. Four other shell tests failed with the same signatures on the historical reference in the nested sandbox; the original SDK exception fixtures were unavailable. The PR remains a draft while the end-to-end benefit and quality remain uncertain.

The normal application launcher uses the changed policy. The exported headless fallback also avoids repeating the command; user-provided and injected reasons are passed through unchanged. The benchmark runners also included the Sonnet compatibility fix from #255; this change itself does not depend on that fix. An independent verifier checked the fallback decision and direct-connection output, including custom reasons and both risk levels.

Validation: focused CLI-to-provider RED/YELLOW tests pass and fail against the baseline; existing approval-reuse tests, typecheck, lint, provider black-box suite (270 passed, 5 skipped), human rejection-reason coverage and independent safety verification pass.

The earlier generation environment lacked the socat executable needed by Term2’s sandbox. A repaired isolated grader reran all five outputs from the latest comparison and the full recovered ACP wire suite passed; the original runs were still generated in the impaired environment. The previously reported model comparisons therefore do not establish general quality or cost parity.

A later smoke evaluation used three different small tasks (retry policy, invoice aggregation, and path containment) and two models (Sonnet and Sol). The Term2 baseline, this change, and Pi passed all 18 held-out checks. No run hit an approval denial, so this is cross-task quality smoke evidence, not evidence of savings from this change. Pi had skills disabled while Term2 retained its skills. Provider cache sharing also differed across some runs; no stable cost-parity claim is made.

@qduc
qduc changed the base branch from codex/sonnet-adaptive-thinking to main November 12, 2026 01:05
@qduc
qduc force-pushed the codex/efficient-file-guidance branch from 968f523 to c5541db Compare November 12, 2026 01:05
@qduc
qduc marked this pull request as ready for review November 12, 2026 01:29
@qduc
qduc merged commit 156e190 into main Nov 12, 2026
14 of 16 checks passed
@qduc
qduc deleted the codex/efficient-file-guidance branch November 12, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant