Skip to content

Agent harness adoption Part B: tool loop on @qduc/agent-core (flagged) + provider replay e2e - #42

Merged
qduc merged 6 commits into
mainfrom
ccr-c2e20a05-677q07
Nov 10, 2026
Merged

qduc merged 6 commits into
mainfrom
ccr-c2e20a05-677q07

Conversation

@qduc

@qduc qduc commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Part B of #40. When AGENT_HARNESS_TOOL_LOOP=true (off by default), the streaming tool loop runs on @qduc/agent-core. The PR also adds end-to-end replay tests that run both tool loops against recorded provider traffic, with no real API calls. While building the tests I found and fixed a real bug in the upstream logging tee.

Changes

Harness loop (212cc13)

  • backend/src/lib/harness/harnessToolsStreaming.ts runs the flagged loop. openaiProxy.js routes to it when the flag is on.
  • providerModel.ts retries a reasoning-only response twice, then emits the existing incomplete-reasoning warning.

Provider replay e2e (50542e5, 9e73ce2)

  • integration/providerReplay.test.js replays sanitized real recordings from Term2 for Chat Completions, Responses, Anthropic and Gemini. Each one goes through the real route, adapters, tools and persistence, once per loop.
  • Parallel tool calls use synthetic fixtures derived from those recordings. Sources are listed in fixtures/PROVENANCE.md.
  • integration/syntheticFixtureOracle.test.js serves every fixture to the official openai, @anthropic-ai/sdk and @google/genai SDKs:
    • real recordings must parse to their recorded call (calibration);
    • synthetic fixtures must parse to both calls.
  • The replay server can stall or drop a turn mid-stream. This drives the Stop and upstream-drop scenarios in both loops.

Bug fix (92c2016)

  • upstreamLogger.js: pipe() doesn't forward errors from the source stream. When a provider connection dropped mid-answer, the client waited for the 30s idle timeout. The fix destroys the passthrough with the source error, so the drop is now detected in about 100ms.
  • toModelFailure now reads error messages by duck typing, because errors from another realm fail instanceof Error.

Install @qduc/agent-core from npm (2b0cfff)

  • 0.2.0-alpha.1 is published. Backend and electron pin that exact version; both lockfiles resolve from registry.npmjs.org.
  • The registry tarball has the same sha512 integrity as the tarball previously vendored here (packed by qduc/agent-runtime 3e322ba), so the installed code is identical.
  • backend/vendor/ and the COPY vendor lines in both Dockerfiles are gone.

Verification

  • Backend: npm run lint and npm run typecheck are clean. The full jest suite passes: 1352 passed, 3 skipped, 15 todo.
  • npm run test:provider-replay: 60/60.
  • A clean npm ci in electron installs 0.2.0-alpha.1 from npm.
  • Mutation checks:
    • the oracle rejects a reused Anthropic block index and a missing Chat id;
    • the drop test fails with the tee fix reverted.

Known limits

  • The parallel fixtures are a best guess at each provider's wire format. They are valid for the vendors' own parsers, but they aren't captures. Replace each one with a real capture when one is available.
  • The harness loop doesn't send previous_response_id and ignores provider_stream:false.
  • Streamed tool_output is a string in the harness loop and an object in the legacy loop.

Related

  • qduc/agent-runtime and qduc/term2 PRs from ccr-c2e20a05-677q07. term2 still installs a vendored tarball and needs the same switch to npm.

Generated by Claude Code

…agent-core behind a flag

Installs @qduc/agent-core 0.2.0-alpha.1 from a vendored tarball
(backend/vendor/, packed from qduc/term2) until the alpha is on npm, and
deletes the local contracts mirror so the Part A wrappers compile against
the package's own declarations.

AGENT_HARNESS_TOOL_LOOP=true routes streaming tool requests to
harness/harnessToolsStreaming.ts: it builds a TurnSpec from the request,
runs startTurn, and feeds every accepted event to TurnEventSink. Default
off; the legacy loop is unchanged.

- Per-user max iterations -> TurnSpec.limits.maxModelCalls.
- Parallel tool settings -> TurnDeps.maxParallelToolCalls.
- Reasoning-only responses are re-asked inside the model wrapper (2x, then
  the legacy warning). The engine's model_behavior recovery refuses to
  replay once reasoning has streamed, so it cannot cover this case.
- Requests are self-contained for now (no previous_response_id).
- Interaction requests are denied rather than left waiting.

Packaging: both Dockerfiles copy vendor/ before npm ci, and Electron (which
repackages backend dependencies) declares the package too. The backend
lockfile's root devDependency specs are also synced to package.json.

A parity test drives both loops through /v1/chat/completions against the
same scripted upstream: tool call then answer, reasoning-only retry and
exhaustion, iteration cap, upstream error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqbJBAaQKKFnHRJoDCGExb
Replays sanitized recordings of real provider traffic, copied from
qduc/term2's provider black-box fixtures, from a loopback server through
the real /v1/chat/completions route, ChatForge's provider adapters, tool
execution and persistence. No real API calls.

Recordings: OpenAI Chat Completions (OpenCode Go), OpenAI Responses
(o3-mini), Anthropic Messages (OpenCode Go) and Gemini, all the two-turn
tool-continuation probe. Each runs through the legacy loop and the agent
harness loop and is checked against what the recording says (tool name,
arguments and call id, answer text) and against the other loop
(stream, persisted message and tool rows).

The wire checks pin the known continuation difference: legacy chains
Responses-API turns with previous_response_id, the harness sends full
history. A mutation that stops the harness streaming tool calls fails 8 of
the 12 tests.

Run: npm run test:provider-replay (also part of npm test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqbJBAaQKKFnHRJoDCGExb
pipe() does not forward source errors, so a provider connection that
dropped mid-answer left consumers waiting for the idle timeout (~30s)
instead of failing in ~100ms. Destroy the passthrough with the source
error.

Also read error messages by duck typing in toModelFailure: errors from
Node internals can come from another realm and fail instanceof Error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqbJBAaQKKFnHRJoDCGExb
- Synthetic parallel-call fixtures derived from the real recordings for
  Chat (sequential and interleaved), Responses, Anthropic and Gemini,
  shaped after the official SDKs' own tests and parsers (sources in
  fixtures/PROVENANCE.md).
- Oracle test serving every fixture to the official openai,
  @anthropic-ai/sdk and @google/genai SDKs; real recordings act as
  calibration.
- Replay server can stall or drop a turn mid-stream; Stop and drop
  scenarios run through both tool loops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqbJBAaQKKFnHRJoDCGExb
The harness graduated from Term2 to qduc/agent-runtime. Replace the
tarball packed from Term2 with the one packed there (same version and
code; repository metadata and README differ) and update the lockfile
integrity in backend and electron.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqbJBAaQKKFnHRJoDCGExb
0.2.0-alpha.1 is now published. The registry tarball has the same
sha512 integrity as backend/vendor/qduc-agent-core-0.2.0-alpha.1.tgz,
so the installed code is identical.

- backend and electron pin the exact npm version
- lockfiles resolve from registry.npmjs.org (integrity unchanged)
- delete backend/vendor and the COPY vendor lines in both Dockerfiles
- update the harness doc and tick the checklist item

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqbJB9PPzDrqCr5EPj6yNc
@qduc
qduc merged commit 19d0c65 into main Nov 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants