Skip to content

feat(legal): add an agent for negotiation follow-ups against the playbook - #31

Merged
rparundekar merged 3 commits into
mainfrom
feat/agent-legal
Nov 11, 2026
Merged

rparundekar merged 3 commits into
mainfrom
feat/agent-legal

Conversation

@rparundekar

Copy link
Copy Markdown
Collaborator

What this does

Adds agent.py beside the playbook review. A lawyer mid-negotiation asks follow-up questions:

  1. What does the playbook allow when the other side rejects our position?
  2. Does their new wording meet it?
  3. Draft the reply.

The tool, not Claude, puts a revised clause back into the full NDA before checking it. Corrin's new Section 7 passes in context, and on its own the check can't settle it (unclear). Code checks every answer and reply draft. The term position gets one fallback in rules.yaml. The workflow ignores that fallback, and its results are unchanged.

Live results (5 runs each, using SIE Serverless, 10 Nov 2026)

14 of 15 conversations came out as expected. The first batch was 10 of 15 because of a bug in the tool, which was fixed in the tool rather than the prompt. The README reports both batches.

  • The trade-secret tail: the verifier passed "for so long as it remains a trade secret" 10 of 10 times, and Claude flagged it 10 of 10 times. It's reported as a gap in the workflow.
  • One reply claimed an action nobody took. No check here catches that.

Depends on the docs PR.

🤖 Generated with Claude Code

rparundekar and others added 3 commits November 10, 2026 19:50
…book

agent.py lets a lawyer ask follow-up questions on one NDA: what the playbook
allows when the other side rejects a position, whether new wording meets it,
and a reply draft. Its tools wrap the workflow, and the tool, not Claude,
puts a revised clause back into the full NDA before checking it. Code checks
every answer and reply. The term position gains one fallback in rules.yaml,
which the workflow ignores; its results are unchanged.

14 of 15 conversations came out as expected over 5 runs each. The verifier
passes a "for so long as it remains a trade secret" tail; the README reports
it as a gap in the workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The post isn't published yet, so its links returned 404, and its claims will
change in the blog update. The README now links the root README's test and
states each point on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Checked wording counts only for a position whose cited section it
  replaced or whose status it turned into a match, and only when no
  position came out worse than in the review; every position open in the
  review or the latest clause check must be in the draft.
- lawyer_to_confirm is refused when the review or the latest check says
  deviates.
- A reply body that accepts a position (agent.acceptance words, not
  negated just before the word, not a question) fails unless the table
  marks it agreed or proposed; README narrowed to what this catches.
- A clause check grounds a duration or place only from rows that came out
  a match.
- Single-quoted spans are checked as quotes too.
- Draft bodies and answers are fenced; table cells, warnings, questions
  and tool arguments go through md_safe.
- Tool arguments are validated against the signature; errors inside a
  tool raise instead of becoming "Bad arguments".
- The note's clause quote uses agent.quote_min_words; asking about a
  position not checked for our role is a check problem, not a crash.
- An answer or draft bundled with tool calls goes back to Claude unshown.
- The status comes from a draft for the latest draft request.
- Transient Claude errors (incl. timeouts, 408, 409) stop the turn; any
  other stops the run, message first; cost priced on response.model.
- README: "recipe.decide twice" only when a fallback applies; agent
  numbers from five live reruns after the fixes (14 of 15).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rparundekar

Copy link
Copy Markdown
Collaborator Author

Self-review

Two review passes ran on this PR. One read the code: correctness, silent failures, tests and comments. The other covered security and checked the README against the code. Every finding below was fixed in cab8e86. Each fix has a selfcheck case that fails if the fix is undone, and the reviewers' own repro scripts now raise a warning on every case.

Important (5, all fixed)

  • Checked wording counted for every position. A checked text counted as allowed wording for any position that matched in that check, even when it made another position worse. For Corrin, a survival clause with Texas law bolted on was accepted as the term, and Orlaith's Singapore sentence was accepted as permitted disclosures. Fixed: checked wording counts only for the position whose section it replaced or turned into a match, and only if no position got worse. Every position open in the review or the latest check has to appear in the draft.
  • A deviating row could be hidden as lawyer_to_confirm after a hypothetical check came back as a match. Fixed: this is refused while the review or the latest check says the row deviates.
  • The reply body wasn't checked against the table. "We accept Singapore law" passed while governing law was marked ask_approver. Fixed: an acceptance sentence about a position fails unless the table marks that position agreed or proposed. The check handles negation, and the README names exactly what it catches and what gets past, such as "Singapore works for us".
  • A clause check that rejected a value still grounded it. "Offer four years" on Corrin passed after a check came back deviates at four years. Fixed: only a check whose status was a match grounds a value.
  • The invented-quote check read only double quotes. Fixed: single-quoted spans are checked too.

Suggestions (all fixed)

  • Escaping: draft bodies are fenced and table cells escaped.
  • Tool arguments: checked against each tool's signature, so real errors now raise instead of reading as "bad arguments".
  • Note quote check: the minimum quote length comes from rules.yaml, and an approver on a position that doesn't apply no longer crashes the run.
  • Bundled answers: an answer sent in the same response as lookups goes back to Claude unshown.
  • Status: it comes from a draft made for the latest draft request.
  • Claude errors: classified as transient or fatal, and tokens are priced on the model that answered.
  • README: the tools table no longer says recipe.decide runs twice; that only happens when a fallback applies.

Reruns after the fixes (5 live runs per conversation)

14 of 15 came out as expected.

  • Lowfield: 5 of 5, up from 4 of 5. Claude flagged the trade-secret tail every time.
  • Corrin: 5 of 5.
  • Orlaith: 4 of 5. In one run Claude wrote the note to the approver a turn early and submitted nothing on the turn that asked for it, so the stricter status rule gave no_draft. It's reported as a miss.

@rparundekar
rparundekar marked this pull request as ready for review November 11, 2026 02:50
@rparundekar
rparundekar requested a review from a team as a code owner November 11, 2026 02:50
@rparundekar
rparundekar merged commit 21eb64d into main Nov 11, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant