Skip to content

feat(finance): add an overnight agent for counterparty news the workflow sends to review - #30

Merged
rparundekar merged 4 commits into
mainfrom
feat/agent-counterparty
Nov 11, 2026
Merged

rparundekar merged 4 commits into
mainfrom
feat/agent-counterparty

Conversation

@rparundekar

Copy link
Copy Markdown
Collaborator

What this does

Adds agent.py beside the counterparty news workflow. This is the guide's overnight fallback: articles the workflow sends to review go to Claude. Claude reads the borrowers' credit files and reruns the workflow's own event check with the link fixed. A judge in code then checks every quote, borrower and event, and allows only the outcomes that each review reason permits. Everything goes to the credit analyst's queue, and nothing becomes an alert.

Live results (5 runs each, using SIE Serverless, 10 Nov 2026)

10 of 15 jobs came out as expected.

  • N-04, the shared name: 5 of 5.
  • N-07, the ticker conflict: 5 of 5.
  • A-01: 0 of 5. It was written to land between the alert and filed scores, but the workflow filed it as routine on every run, so it never reached the agent. It's reported as a miss, not tuned. It also means the guide's "unsure score" example hasn't fired on our data yet.

recipe.py is unchanged, and its outputs are identical before and after. Depends on the docs PR.

🤖 Generated with Claude Code

rparundekar and others added 4 commits November 10, 2026 19:32
…low sends to review

agent.py runs the news workflow, then gives Claude each article in the review
queue to dig into: the borrowers' credit files and the workflow's own event
check with the link fixed. A judge in code checks every quote, borrower and
event, and allows only the outcomes each review reason permits; everything
lands in the credit analyst's queue. recipe.py and its results are unchanged.

Over 5 runs each, N-04 and N-07 came out as expected every time. A-01, written
to land between the alert and filed scores, was filed by the workflow on every
run, so it never reached the agent; the README reports it as a miss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The post isn't published yet, so its links returned 404, and its claims will
change in the blog update. The README now links the root README's test and
states each point on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The judge runs the fixed-link event check for every candidate on a suggested
  link's row; finding_checked only when the pick's check differs from every
  other candidate's and agrees with the agent. Identical checks (N-04) are
  call_not_checked; N-04's expected outcome changes for that reason.
- Quotes are whole words, at least quote_min_words (4), from one field value
  or note, never field labels; a link's article quote contains the mention.
- suggest_not_a_borrower needs an article quote and one from the candidate's
  file.
- Scores on other outcomes must be null or match the workflow's own number.
- A retry where every SIE call fails keeps the first pass; those articles go
  to the analyst as sie_failed. A refusal still stops the run.
- read_credit_file, lookups and quote sources reach only the row's candidates
  and borrowers a lookup returned for a name or ticker the article prints.
- review_notes.md escapes text through md_safe and fences the note; the judge
  rejects links and HTML in the note.
- Tool input is validated against each tool's fields instead of catching
  TypeError.
- Each borrower's judge check runs once per job; result.checks holds only the
  final submission's checks.
- Claude errors: 408/409 and timeouts are transient, anything else stops the
  run with the error's message first; cost is priced on response.model.
- README: retry path in the chart, figures of two or more digits, honest job
  count, new numbers from 5 live runs after the fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A response from a fallback model now shows up in cost.json as unpriced,
not priced at figures nobody checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rparundekar
rparundekar force-pushed the feat/agent-counterparty branch from 87b3974 to abf77ee Compare November 11, 2026 02:48
@rparundekar

Copy link
Copy Markdown
Collaborator Author

Self-review

Two review passes ran on this PR. One read the code: correctness, silent failures, tests and comments. The other covered security and checked the README against the code. Every finding below was fixed in 551eb47 and abf77ee. Each fix has a selfcheck case that fails if the fix is undone.

Important (5, all fixed)

  • The judge couldn't tell shared-name candidates apart. For N-04, the event check with the link fixed reads the same sentences for CP-1004 and CP-1005, so a finding that picked the wrong Northwind still came out finding_checked. Fixed: the judge now runs the check for every candidate. finding_checked needs the pick's result to differ from every other candidate's; otherwise the status is call_not_checked. N-04's expected outcome changed to call_not_checked because review found this gap, not because of a live result, and the README says so.
  • Quotes were a bare substring match, so "a" or a field label counted as evidence. Fixed: a quote must be at least 4 whole words, taken from a field value. For a link, the article quote must contain the name in question.
  • suggest_not_a_borrower needed no evidence. Fixed: it now needs an article quote and a quote from the candidate's file.
  • Scores were only checked on links, so a made-up 0.99 could appear in the analyst's notes. Fixed: on other outcomes the score must be empty or match the workflow's own number.
  • A retry where every call failed stopped the whole overnight run. Fixed: the first-pass results are kept, and those articles go to the analyst as sie_failed.

Suggestions (all fixed)

  • Tool scope: credit files and lookups reach only the row's candidates, and names or tickers that appear in the article.
  • review_notes.md: text is escaped, the note is fenced, and the judge rejects a note containing a link or HTML.
  • Tool input: checked against each tool's schema. Real errors are no longer reported to Claude as "bad arguments".
  • Judge checks: cached per job, and only the final submission's checks are written out.
  • README: the Mermaid chart shows the retry path, "every figure" now says figures of two or more digits, and the job count is honest.
  • Anthropic errors: classified as transient or fatal. Tokens are priced on the model that answered, and a fallback model with no listed price shows as unpriced rather than guessed.

Reruns after the fixes (5 live runs)

  • N-04 came out call_not_checked on 5 of 5 runs.
  • N-07 came out left_for_person on 5 of 5 runs.
  • A-01 was still filed by the workflow, so it never reached the agent, and it stays a reported miss.
  • No live job reached finding_checked. That path is exercised only in selfcheck.py, and the README says so.

@rparundekar
rparundekar marked this pull request as ready for review November 11, 2026 02:48
@rparundekar
rparundekar requested a review from a team as a code owner November 11, 2026 02:48
@rparundekar
rparundekar merged commit 16e06a2 into main Nov 11, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant