revise: replace the diff-probe sweep and rule-probes with scripts and a backtest #86

Skill: revise, diff-probe, rule-probe · Plugin commit: 4495e1441dbd

Extends #50 and #54.

What happens.

• prepare.md runs one diff-probe per merged pull request, up to thirty, against every rule and documented claim (:94-110).
• harvest.md The bar runs one rule-probe per example: anchor of each drafted rule.

Effect. In the reference run:

• 34 of about 60 agents were probes, against 20 that read threads.
• The sweep's 62 "breaks" were mostly template claims judged on pull requests opened before the template existed (revise: the sweep judges template claims on PRs opened before the template #51). It produced one ticket-worthy pattern.
• Rule-probes were shown their entry's own example: line, and confirmed it (revise: rule-probe input is hand-assembled and can leak the example #54).
• 5 probes hit their turn limits (diff-probe: turn limit too low, and a verdict that contradicts itself #50).

Change.

• Mechanical drift checks become scripts and git grep, run per pull request since its document was adopted: template lines, changelog entries, TODO without a ticket, history in comments, a logging pattern. The output is the same doc/ and ticket/ proposals, deterministic and free.
• A backtest replaces rule-probe. Run the drafted rule's wording over the last N merged diffs, and report how often it would fire and on what, so the group sees false positives. If that is too costly, drop the probe and rely on rule_lint plus the group's mark.
• Keep one agent sweep only for claims no script can judge, capped well below the subagent limit (revise: "all in one message" exceeds Claude Code's 20-subagent limit #46).
• Spend the freed budget on reading (thread-analyst: replies and review prose are never read as asks #75, thread-analyst: one paraphrased line per thread loses the quote, rationale and generality #77).

Done when.

• On the reference window, the scripts reproduce the sweep's genuine findings, and the template-claim false positives are gone.
• A revision run uses fewer than half the agents.

Tracked in #89.