Repository navigation
[3442 by Claude Opus 5.5] Flag automation runs that repeat the same report or act without changing anything - #3454
Conversation
…wner gets one Inbox notification after N unproductive scheduled runs in a row. Typecheck and lint pass, and every test suite I ran passes. I didn't run the full test suite, and I only ran the migration against SQLite, not Postgres. How runs are flagged - Report novelty: when a report is stored, the run gets a hash of the normalised report text and a `report_novelty` of `new`, `unchanged` or `empty`. Normalisation collapses whitespace and replaces dates, times, weekdays, epoch timestamps, "3 hours ago"-style times, UUIDs and run ids. A report under 40 characters after normalisation is `empty`. The previous run's report is re-hashed rather than read from its stored hash, so runs from before this change compare correctly. Version numbers like `4.17.21` are left alone. - Acting steps: the revoke endpoint now also returns how many mutating operations the grant performed. That's the count of `mcp_operations` rows for the grant, minus those that definitely failed. The acting job now revokes the grant before it marks the run completed, so the count is stored while the run is still `acting`. If the count can't be read it stays `null`, which is not treated as a no-op. - What counts as unproductive: an empty or unchanged report, or an acting step with 0 changes. An acting step that changed something makes the run productive, whatever its report said. The state machine is unchanged. - Where it shows: the REST run payload, MCP `get_automation_run` / `list_automation_runs`, the CLI (a new Change column in `runs`, a `change:` line in `report`), and a "No change" badge in the runs list and run detail. I also added a derived `noChangeReason` field to the payload so every client shows the same badge. Streak, notification and auto-pause - Only scheduled runs count. Manual runs are flagged but neither extend nor reset the streak. - The notification fires once, when the streak reaches exactly N. A retry can't create a second one, and the next productive scheduled run resets the streak. - N is a new instance setting, `automation_unproductive_run_threshold` (1–50, default 3). It's available through the API, MCP and the settings catalog, but not in the Settings UI, the same as the existing usage-pause setting. - The per-automation option `pauseWhenUnproductive` (off by default) appears as a checkbox under the schedule. When the notification fires, it turns the schedule off through `disableAgentDefinitionSchedule`, logs the reason, and says so in the notification. - The migration adds three nullable run columns and the definition flag, and reverses cleanly; existing runs keep null flags. Decisions you may want to revisit - Automations without a repository: the existing notification code skips these. I didn't want this one silently dropped, so they're notified under the `system_failure` category instead. Say if you'd rather they get no notification. - Previous report always read: the novelty check reads the latest previous report even when "include previous reports" is off. I changed the one existing test that expected no read in that case. - Layout fix: the badge pushed the Duration and Report columns out of line in the wide runs list, so I widened the State column. The automation-runs e2e specs still pass. - Line cap: `processAgentRunJob.ts` was already at the 400-line lint limit, so I moved `agentReportRecap` into its own file and kept the old export. Tests and docs - New test files cover hashing and normalisation, the migration, the streak rules, the notification, auto-pause and the setting's validation. - I extended the job, grant client, internal route, REST, MCP, CLI and UI tests, including the acting-step count. - The automations docs page has a new section, and the settings-locations page was regenerated. Previews of the runs list (desktop and phone), the run detail, and the new schedule checkbox were captured with Playwright.
|
/ultrafix goal=8 max=10 |
|
🔄 Ultrafix loop started (goal: 8/10, max cycles: 10) First action:
|
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds productivity flags across storage, workers, APIs, CLI and UI, with substantial focused test coverage. It needs changes before merge because streak evaluation can act on unfinished runs and miss qualifying no-op runs or notifications. The supplied issue specification is truncated; this review uses its visible requirements and the correctness of the changed behavior. Verification was by static trace only; no commands were run. Current checks show 37 passed, none failed and one pending. ✅ Consistent presentation — REST and MCP expose the stored flags, and the UI uses the shared no-change classification. ✅ Backward-compatible storage — Existing runs retain nullable flags, and automatic schedule pausing defaults to off. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F1: 🔴 Awaiting approval is treated as final
F2: 🔴 Known action counts require unrelated report flags
F3: 🔴 A failed threshold notification is permanently skipped
SuggestionsThese are optional follow-ups and are not sent to S1: 🟢 Align mutation-count documentationThe new documentation says counting occurs before grant revocation, while the endpoint counts afterward. Align the description with the implementation. This wording correction is optional and does not itself establish a runtime defect. ScoreThe implementation covers the main surfaces well, but three streak correctness defects remain despite the passing checks. These require focused corrections before merge. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit 60bfaec
Summary of ChangesI fixed F1, F2 and F3. Typecheck and zero-warning lint pass for the root, core and API. The streak, autonomy/approval route, both job processors, the automation REST routes and the MCP run tests pass: 202 tests, 0 failures. To check the new tests, I put back the old logic for each finding, confirmed the tests then fail, and restored the fix. I didn't run the full test suite. F1 – a run waiting for approval was treated as final
F2 – known action counts needed a report comparison
F3 – a failed notification was skipped for good
Tests and docs
Separate issues I found but didn't fix (out of scope)
This doesn't change anything visible in the UI, so I didn't capture previews. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
… for the root, core and API. The streak, autonomy/approval route, both job processors, the automation REST routes and the MCP run tests pass: 202 tests, 0 failures. To check the new tests, I put back the old logic for each finding, confirmed the tests then fail, and restored the fix. I didn't run the full test suite. I fixed F1, F2 and F3. Typecheck and zero-warning lint pass for the root, core and API. The streak, autonomy/approval route, both job processors, the automation REST routes and the MCP run tests pass: 202 tests, 0 failures. To check the new tests, I put back the old logic for each finding, confirmed the tests then fail, and restored the fix. I didn't run the full test suite. **F1 – a run waiting for approval was treated as final** - `awaiting_approval` now counts as pending, alongside `report_ready` and `acting`. This covers preview runs and auto runs held back by the cost gate, so they no longer extend a streak, notify the owner or pause the schedule. - Once the acting step finishes, the action job evaluates the run as before. Rejection now also triggers an evaluation: the reject route (which MCP `reject_automation_run` also goes through) checks the streak after the run moves to `rejected`. A failure in that check never fails the reject request. **F2 – known action counts needed a report comparison** - A known `actingMutations` count now decides the verdict by itself, whether or not the report could be compared. Zero makes the run unproductive and more than zero makes it productive. A run is only pending when neither the count nor the report comparison gives an answer. - The streak query now includes runs where either flag is set, not only `report_novelty`. **F3 – a failed notification was skipped for good** - The streak has a stable identity: the productive scheduled run just before it, or `start` if there is none. It no longer changes as the streak grows, and the notification's deduplication key is built from it. - Any unproductive scheduled run in a streak at or past the threshold now checks whether that notification was actually saved, and sends it if not. So a failed write is retried by the next run, and once it is saved, later runs neither notify nor pause again. - The schedule is now paused only after the notification is saved. In the old order, a paused schedule meant no later scheduled run would come along to retry a lost notification. - The streak lookup now pages back through up to 1,000 flagged scheduled runs. A streak longer than that gets an identity from the oldest run read, so the owner could get a second notification after 1,000+ unproductive runs in a row. **Tests and docs** - New tests in `test/agentRunProductivity.test.ts` cover: waiting runs that are later approved or rejected, runs whose report couldn't be compared, a stable streak identity, and a notification that fails and is then retried with the pause applied only after it's saved. One test uses the real notification service on SQLite to show a saved notification is never repeated. - `test/agentAutonomy.test.ts` checks that rejecting a run hands it to the streak check. - The automations docs page now says a failed notification is retried and that runs waiting for approval aren't counted. **Separate issues I found but didn't fix (out of scope)** - If the notification is saved but pausing the schedule then fails, nothing retries the pause, and the notification still says the schedule was paused. - Runs that fail or are cancelled after their report is stored are counted using their report flag, but no streak check runs at that moment. With the F3 retry, the next unproductive scheduled run picks them up. This doesn't change anything visible in the UI, so I didn't capture previews. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThis PR adds report novelty, acting-operation counts, client-visible flags, and notifications with optional schedule pausing. It needs changes before merge: notification deduplication breaks for long streaks, and a failed schedule pause cannot recover. ✅ Consistent presentation — REST, MCP, CLI, and UI expose productivity flags while preserving run states. ✅ Unknown counts remain unknown — Revocation failures retain a null mutation count instead of falsely recording zero. ✅ Approval-aware classification — Pending acting steps do not receive a final streak verdict until completion or rejection. The issue specification is explicitly truncated; this review uses its visible requirements and the supplied changed behavior. The authoritative checks show 37 passed, none failed, and one pending. No commands or tests were run. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F4: 🔴 Recover an unsuccessful schedule pause
F5: 🔴 Keep streak identity stable beyond the scan limit
SuggestionsThese are optional follow-ups and are not sent to No suggestions. ScoreThe implementation covers the main surfaces and includes substantial tests, but two persistent-state defects undermine notification and pause guarantees. Both appear fixable without broad redesign. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit e0dc8ec
Summary of ChangesI fixed F4 and F5. Typecheck and zero-warning lint pass for the root, core and API. The streak tests, the two job processors, the autonomy, definition-route, MCP-run and grant tests all pass: 233 tests, 0 failures. To check the new tests, I put back the old logic for each finding, saw 9 of them fail, and restored the fix. I didn't run the full test suite, and I only ran the migration against SQLite, not Postgres. F5: streak identity past the 1,000-run scan limit
F4: a pause that didn't finish
I also updated the pausing paragraph in Something else I noticed: a notification couldn't be updated to mark the pause as done, because saved notifications can't be changed after they're written. That's why the record lives on the definition instead. Nothing here changes what users see, so there are no previews. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
… the root, core and API. The streak tests, the two job processors, the autonomy, definition-route, MCP-run and grant tests all pass: 233 tests, 0 failures. To check the new tests, I put back the old logic for each finding, saw 9 of them fail, and restored the fix. I didn't run the full test suite, and I only ran the migration against SQLite, not Postgres. I fixed F4 and F5. Typecheck and zero-warning lint pass for the root, core and API. The streak tests, the two job processors, the autonomy, definition-route, MCP-run and grant tests all pass: 233 tests, 0 failures. To check the new tests, I put back the old logic for each finding, saw 9 of them fail, and restored the fix. I didn't run the full test suite, and I only ran the migration against SQLite, not Postgres. **F5: streak identity past the 1,000-run scan limit** - When the scan runs out before finding the run that ended the previous streak, `findUnproductiveScheduledStreak` now looks that run up with one extra query: the newest productive scheduled run older than the scan window. The `scan:<oldest-id>` fallback is gone. A streak longer than 1,000 runs keeps the same key it had when it was shorter: `start`, or the id of the productive run before it. - That query restates in SQL the same "productive" rule the code applies: more than zero changes from the acting step, or no acting count and a new report once no acting step can still run. - New tests: - streaks of 999, 1,000, 1,001 and 1,150 runs, with and without a productive run before them. - pending and manual runs beyond the window, which don't change the key. - a test that checks every combination of run state, report novelty and acting count, so the SQL rule and the code can't drift apart. - a full evaluation across the boundary, which sends exactly one notification. **F4: a pause that didn't finish** - The definition now records which streak's pause was carried out, in a new `agent_definitions.unproductive_paused_streak` column. I added it to this PR's existing migration rather than a new one. - `pauseScheduleForUnproductiveStreak` writes that record in the same transaction as turning the schedule off. The write is conditional, so if two evaluations run at once only one of them pauses. - When a saved notification says the schedule was paused, the evaluator no longer just stops. It tries the pause again, and the record makes that a no-op once the pause has happened. So: - a disable that failed, or a worker that stopped after the notification was saved, gets finished by the next unproductive run, with no second notification. - an owner who turns the schedule back on after a successful pause isn't overridden. - an owner who turns pausing off after a failed pause isn't overridden either. - a notification that didn't say "paused" never leads to a pause later. - To know what the saved notification said, the evaluator now reads its `schedulePaused` value through `findUnproductiveStreakNotification`, instead of only checking that it exists. The REST route's version of the evaluator now uses the new functions too. - New tests cover the failed disable, the interrupted worker, the owner turning the schedule back on, the owner turning pausing off, a notification that didn't claim a pause, and two pauses racing. I also updated the pausing paragraph in `docs/docs/features/agents.md` to describe the retry, and adjusted the existing streak tests to the renamed parts. The new tests are in `test/agentRunUnproductiveStreakRecovery.test.ts`. **Something else I noticed:** a notification couldn't be updated to mark the pause as done, because saved notifications can't be changed after they're written. That's why the record lives on the definition instead. Nothing here changes what users see, so there are no previews. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds report novelty, acting-operation counts, shared presentation, and owner notifications with optional schedule pausing. It needs changes before merge: normalization can hide meaningful report differences, and schedule pausing can override a concurrent opt-out. ✅ Consistent presentation — REST, MCP, CLI, and UI expose the productivity flags, with shared logic for the no-change verdict. ✅ Durable pause recovery — Notification deduplication and transactional pause records cover interrupted pauses and subsequent schedule re-enablement. ✅ Broad regression coverage — Added tests exercise migration compatibility, unknown mutation counts, approval states, notification failures, and long streaks. The issue specification is explicitly truncated; this review assesses its visible requirements and the correctness of the supplied changes. Review was static, with no commands executed. Current checks show 37 passed, none failed, and one pending. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F6: 🔴 Preserve meaningful UUIDs in reports
F7: 🔴 Recheck opt-in when pausing
SuggestionsThese are optional follow-ups and are not sent to S2: 🟢 Correct mutation-count timing documentation
ScoreThe implementation has substantial coverage and no current failed checks, but two correctness defects can incorrectly pause productive or opted-out automations. Both admit localized fixes. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit 63e86a5
Summary of ChangesI fixed F6 and F7, and each fix has regression tests. The 6 related test files pass (192 tests, 0 failures), and typecheck and zero-warning lint pass for the root and core. I didn't run the full test suite, and nothing ran against Postgres. To check the new tests, I put back the old logic for each change and confirmed they fail, then restored the fix. F6: other UUIDs in reports are kept
F7: the pause checks the opt-in at the moment it runs
I also added a sentence to each of the two matching paragraphs in Found but not changed:
🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
… related test files pass (192 tests, 0 failures), and typecheck and zero-warning lint pass for the root and core. I didn't run the full test suite, and nothing ran against Postgres. To check the new tests, I put back the old logic for each change and confirmed they fail, then restored the fix. I fixed F6 and F7, and each fix has regression tests. The 6 related test files pass (192 tests, 0 failures), and typecheck and zero-warning lint pass for the root and core. I didn't run the full test suite, and nothing ran against Postgres. To check the new tests, I put back the old logic for each change and confirmed they fail, then restored the fix. **F6: other UUIDs in reports are kept** - **Change:** `agentRunNovelty.ts` no longer replaces every UUID. It only replaces the run ids it is given, ignoring upper/lower case, and longest first so one id never cuts into another. Any other id is kept, so a report naming a different incident or task is `new`. - **Same problem one step back:** the report prompt shows the current run id plus up to 5 earlier run ids. So the previous report can quote a run one further back than the new report can see. Only the blanket UUID rule had been hiding those ids. `reportNovelty.ts` now reads the last 6 reported runs, compares against the newest, and treats all their ids as run ids. Without this, reports that only quote a different run id would have turned `new`. - **Tests:** - a different incident or task UUID gives `new`, while a different run id stays `unchanged`. - run ids in different upper/lower case are still treated as the same run, and other UUIDs are kept. - in the job processor, the previous report can quote a run from beyond the current prompt and is still `unchanged`. A different incident is `new`, and the test fails if the lookup is cut back to one report. **F7: the pause checks the opt-in at the moment it runs** - **Change:** `pauseScheduleForUnproductiveStreak` now only claims the streak if `pause_when_unproductive` is still true, inside the same transaction. If it isn't, it returns a new result, `opted_out`, and changes nothing: no claim, the pending slot is kept, and the schedule stays on. - **Both paths covered:** the check is inside the transaction, so it applies to the first pause after the notification and to the later retry of a failed pause. In `unproductiveStreak.ts`, `opted_out` now gives the outcome `notified` on the first path and `null` on the retry. A pause that was already made still counts as made, even if the owner opts out afterwards. - **Tests:** - the owner opts out during notification delivery, through the real `NotificationService` with an opt-out hooked into its update-publishing step. The schedule stays on, the pending slot is kept and the streak is not claimed. - the owner opts out between the retry reading the definition and the pause. - the pause function on its own: opting out, opting back in, and opting out after the pause was made. - **Lint fix:** the new branch pushed `evaluateUnproductiveStreak` over the complexity limit (21, max 20). I replaced the branching with two small outcome tables, one per path. I also added a sentence to each of the two matching paragraphs in `docs/docs/features/agents.md`. Nothing here changes what users see in the UI, so there are no preview images. **Found but not changed:** - If the owner opts out while the Inbox item is being delivered, the item has already been saved saying the schedule was paused. Its text will then be wrong, even though the schedule correctly stays on. Fixing that would mean editing a saved notification, which is outside F7. The owner caused this themselves during a very short window. - The only per-run ids the agent can see in its prompt are run ids, so I found no other changing id that needs replacing. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds productivity flags across storage, workers, API/MCP, CLI and UI, plus streak notifications and optional schedule pausing. It needs changes before merge because report selection can misclassify repeated reports, and a stale streak evaluation can pause a schedule after a productive run. ✅ Consistent client presentation — REST and MCP expose the derived no-change reason, with CLI output and UI badges covered by tests. ✅ Durable pause recovery — The schedule change and pause marker share a transaction; tests cover interrupted pauses, opt-out changes and manual re-enabling. The supplied checks show 37 passed, none failed and one pending. This review is a static analysis of the supplied code; no commands were run. The issue specification is explicitly truncated, so requirements beyond its visible portion could not be assessed. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F8: 🔴 Compare against the actual previous reported run
F9: 🔴 Revalidate the streak before pausing
SuggestionsThese are optional follow-ups and are not sent to No suggestions. ScoreThe implementation has broad integration and recovery coverage, and current checks show no failures. Two correctness defects remain in the decisions that classify reports and disable schedules. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit ef36871
Summary of ChangesI fixed F8 and F9, and each fix has regression tests. Root, core and API typecheck pass, and zero-warning lint passes for root One test-run problem: the MCP-run suite passed all 16 tests but then stayed open until my 500s timeout. With F8: novelty compares with the report stored last
F9: the streak is rechecked inside the pause transaction
Separate findings, not fixed:
There are no visual changes, so I made no preview. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
… core and API typecheck pass, and zero-warning lint passes for root `src/`, core and the API files I touched. The 9 related root test files pass (223 tests, 0 failures), and so do the API definition-route, grant and MCP-run suites (28, 15 and 16 tests). To check the new tests, I put back the old logic for each finding: the new tests then fail, and pass again with the fix. I didn't run the full test suite, and nothing ran against Postgres; production here is SQLite only. I fixed F8 and F9, and each fix has regression tests. Root, core and API typecheck pass, and zero-warning lint passes for root `src/`, core and the API files I touched. The 9 related root test files pass (223 tests, 0 failures), and so do the API definition-route, grant and MCP-run suites (28, 15 and 16 tests). To check the new tests, I put back the old logic for each finding: the new tests then fail, and pass again with the fix. I didn't run the full test suite, and nothing ran against Postgres; production here is SQLite only. One test-run problem: the MCP-run suite passed all 16 tests but then stayed open until my 500s timeout. With `--test-force-exit` it ends normally with the same 16 passes. My changes don't touch that file or add timers. I didn't run it on a clean checkout, so I haven't confirmed the hang is already there without my changes. **F8: novelty compares with the report stored last** - New `listLatestStoredReports` in `packages/core/src/services/agents/agentRunReports.ts` picks reports by when they were stored, newest first, with run id as the tie-breaker. It has no creation-time cutoff and keeps runs whose acting step later failed. - The prompt-history lookup (`listPreviousReports`) is unchanged and still feeds the prompt. - `reportNoveltyPatch` compares against the first result of the new lookup. It uses both lookups only to collect the run ids a report may quote. - New tests in `test/agentRunReportNoveltyPrevious.test.ts` run against a real SQLite store: - A report B from a run whose acting step then failed makes the next B `unchanged`, including after repeated failures. - An older, deferred run that reports after a newer run is compared with that newer run's report. - Prompt history still leaves out failed runs. - Reports stored at the same moment are ordered by run id. - There is also a processor-level test in `test/processAgentRunJob.test.ts`. **F9: the streak is rechecked inside the pause transaction** - `pauseScheduleForUnproductiveStreak` now receives the streak the evaluation found (its identity and threshold). After the opt-in claim, it re-reads the streak inside the same transaction. If the streak changed or no longer meets the threshold, the transaction rolls back, including the claim, and returns a new result, `streak_ended`. That result maps to `notified` after a new Inbox item and to nothing after a stored one. - The claim is the transaction's first write, so it takes SQLite's write lock before the re-read. A run that completes productively either commits before the re-read, and is seen, or waits until the pause commits. - This applies to both the first pause and a recovered one. The default caller and the API route caller pass the streak through. - New tests in `test/agentRunUnproductiveStreakRecovery.test.ts`: - An acting run that completes with changes during Inbox delivery keeps the schedule on, with no claim and the pending slot kept. - The same completion during a recovered pause keeps the schedule on. - A direct transaction check covers a changed streak and an unmet threshold. - I updated the existing direct pause tests to build real streaks, and added one sentence each for F8 and F9 to `docs/docs/features/agents.md`. **Separate findings, not fixed:** - When the pause is skipped after the Inbox item was created (`opted_out`, and now `streak_ended`), the item has already said the schedule was paused. - Two runs of the same automation that store reports at almost the same moment can each classify before the other's report is stored. Both can then come out `new`. The window is only between the lookup and the store; closing it would need the classification inside a locked write, so I didn't pursue it. There are no visual changes, so I made no preview. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds report novelty, acting-operation counts, consistent client flags, and streak notifications with optional schedule pausing. It needs a targeted correction before merge: notifications can permanently claim that a schedule was paused when the pause was explicitly refused. ✅ Consistent presentation — REST, MCP, CLI, and UI expose the new productivity information without adding run states. ✅ Pause recovery — The transactional pause record prevents repeated disabling after an owner re-enables the schedule, while allowing interrupted pauses to be retried. ✅ Focused coverage — Added tests exercise migration rollback, report ordering, streak identity, notification persistence, and opt-out races. This is a static review; no commands were run. No current-head checks were reported. The supplied issue excerpt is truncated, so requirements beyond its visible portion could not be assessed. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F10: 🔴 Notify the actual pause outcome
SuggestionsThese are optional follow-ups and are not sent to S3: 🟢 Exercise overlapping report storage
ScoreThe implementation has broad integration coverage and thoughtful recovery tests, but the verified notification inconsistency needs correction before merge. Runtime validation remains unperformed as requested. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit b81c76c
Summary of ChangesI fixed F10. An Inbox item now says a schedule was paused only after the pause has actually turned the schedule off. Typecheck passes for root, core and API, zero-warning lint passes on the two changed source files, and the six related test suites pass (213 tests, 0 failures). I didn't run the full test suite. Why the first item can't be corrected later. Inbox items are immutable: a database trigger rejects any update. The streak's item also has to be saved before the pause is tried, so that a failed pause can be retried by the next run. So the only way to keep it truthful is to word it so it stays true whatever the pause does. What changed (
One trade-off to decide on: the issue asks for one Inbox notification per streak. Automations that opt into pausing now get a second, push-free item when the pause happens. If you'd rather keep exactly one item, the alternative is to drop the paused item and leave the automation page as the only place showing the schedule is off. Tests:
Separately (not changed): the comment on I didn't make a visual preview: the Inbox screen itself is unchanged, only the wording the server stores. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
…nly after the pause has actually turned the schedule off. Typecheck passes for root, core and API, zero-warning lint passes on the two changed source files, and the six related test suites pass (213 tests, 0 failures). I didn't run the full test suite. I fixed F10. An Inbox item now says a schedule was paused only after the pause has actually turned the schedule off. Typecheck passes for root, core and API, zero-warning lint passes on the two changed source files, and the six related test suites pass (213 tests, 0 failures). I didn't run the full test suite. **Why the first item can't be corrected later.** Inbox items are immutable: a database trigger rejects any update. The streak's item also has to be saved before the pause is tried, so that a failed pause can be retried by the next run. So the only way to keep it truthful is to word it so it stays true whatever the pause does. **What changed** (`src/jobs/agentRuns/unproductiveStreak.ts`): - **First item:** the title is now "Automation X keeps producing no change". The body says the automation is set to pause its schedule, "so the schedule will be turned off unless pausing is turned off or a newer scheduled run produces something first." That stays true whether the pause succeeds, is refused, or fails. - **Stored flag:** this item's metadata now records only `pauseIntended`, which a later run uses to finish a failed pause. It no longer carries `schedulePaused`, so nothing stored claims an outcome before it happens. The core reader in `agentRunProductivity.ts` reads the new field. - **Second item:** only the evaluation that actually turned the schedule off saves an item titled "Automation X paused: no change in N runs", with `schedulePaused: true`. It has its own key and goes to the Inbox without a push, because the first item already pushed. No paused item is sent when: - pausing was turned off first. - a newer scheduled run produced something first. - the schedule was already off. - another evaluation made the pause. - **If the paused item fails to save:** the schedule stays paused, a warning is logged, and the item isn't retried. A late one could arrive after the owner had turned the schedule back on, and the first item is still true without it. - **Docs:** `docs/docs/features/agents.md` now describes the two items. **One trade-off to decide on:** the issue asks for one Inbox notification per streak. Automations that opt into pausing now get a second, push-free item when the pause happens. If you'd rather keep exactly one item, the alternative is to drop the paused item and leave the automation page as the only place showing the schedule is off. **Tests:** - The two race tests the finding cites now also check the saved text: no "paused" claim and no paused item. Pausing turned off during delivery is one; a newer run finishing productively during delivery is the other. - The failure-and-recovery tests check that the paused item appears only once the next run actually makes the pause, and that the owner gets a push for the first item but not the second. - New tests cover: a schedule the owner turned off during delivery, a paused item that fails to save, and two evaluations racing to pause. - With the old code put back, 12 tests fail, and both race tests fail on the "paused" title. With the fix restored they pass. **Separately (not changed):** the comment on `unproductivePauseReason` says the reason is "repeated in the Inbox item", but it only goes to the log. I didn't make a visual preview: the Inbox screen itself is unchanged, only the wording the server stores. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds report novelty, acting-operation counts, client-visible badges, and scheduled-run notifications with optional pausing. It needs corrections before merge. ✅ Consistent presentation — REST, MCP, CLI, and UI expose the productivity flags, with a shared rule for deriving “no change.” ✅ Pause recovery — The pause transaction rechecks the opt-in and streak, and records completed pauses so re-enabling a schedule is respected. ✅ Focused coverage — Tests cover migration compatibility, notification persistence failures, and competing pause evaluations. The supplied issue specification is truncated; this review relies on its visible requirements and the changed implementation. Current checks show 33 passed, four pending, and no failures. Verification below is static; no commands were run. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F11: 🔴 Concurrent reports can both be classified as new
F12: 🔴 Failed acting steps lose performed mutations
SuggestionsThese are optional follow-ups and are not sent to No suggestions. ScoreThe implementation has substantial coverage and careful pause recovery, but concurrent classification and failure-path accounting can produce incorrect streak decisions. These localized correctness defects should be fixed before merge. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit 5129966
Summary of ChangesI fixed F11 and F12, and each fix has regression tests. Root, core and API typecheck pass, and zero-warning lint passes on the changed files. The 9 related root test files pass (226 tests, 0 failures). I didn't run the full suite, and nothing ran against Postgres. To check the new tests, I put back the old logic for each finding: they fail, and pass again with the fix. F11: two runs storing the same report at once
F12: acting steps that end without completing lose their count
Decision for you: a failed or cancelled acting step whose grant changed nothing now stores Related gaps I left alone (not in scope):
Nothing here is visible in the UI, so there are no previews. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
…t, core and API typecheck pass, and zero-warning lint passes on the changed files. The 9 related root test files pass (226 tests, 0 failures). I didn't run the full suite, and nothing ran against Postgres. To check the new tests, I put back the old logic for each finding: they fail, and pass again with the fix.
I fixed F11 and F12, and each fix has regression tests. Root, core and API typecheck pass, and zero-warning lint passes on the changed files. The 9 related root test files pass (226 tests, 0 failures). I didn't run the full suite, and nothing ran against Postgres. To check the new tests, I put back the old logic for each finding: they fail, and pass again with the fix.
**F11: two runs storing the same report at once**
- The worker still compares the report with the latest stored one. The store now commits the report and its flags only if that latest report hasn't changed since the comparison. This step is serialized per definition: on Postgres it locks the definition row, and on SQLite the transaction already holds the write lock. The new `storeAgentRunReport` is in `agentRunReports.ts`.
- If another run stored a report in between, the worker compares again. After 3 failed attempts it stores the report with no flags rather than a classification that may be out of date. If the run is no longer running, the report is discarded as before.
- The new test runs two identical reports concurrently against real SQLite. The old logic gives `new, new`; the fix gives `new, unchanged`.
**F12: acting steps that end without completing lose their count**
- The acting step's grant now ends as soon as the agent finishes, before the run's final state is written. The count is saved with that state whether the run completes, fails or is cancelled from the Tasks UI. The run's state itself doesn't change.
- I found the same problem in one more place: a run cancelled through the API while its agent is running. The count now gets written onto the cancelled run. That write only touches a failed or cancelled run claimed by this acting step, and never replaces a count already stored.
- An end-to-end test covers the finding's case. Two unchanged scheduled runs are followed by an acting step that makes a change and then crashes, and then a third unchanged run. With the old code that third run paused the schedule. Now the failed run breaks the streak and the schedule stays on. The same holds when the run is cancelled elsewhere.
- I replaced the existing test "a failed acting step stores no count", because it asserted the bug.
**Decision for you:** a failed or cancelled acting step whose grant changed nothing now stores `0`. It therefore shows the "no change" badge and extends the streak, even if its report was new. This matches the acceptance criterion ("zero is recorded as a no-op acting step") and how completed runs already behave. If failures shouldn't count that way, store only counts above zero on that path.
**Related gaps I left alone (not in scope):**
- If the API cancels an acting run and the scheduler's leftover-grant cleanup revokes the grant before the worker does, the worker gets no count and the run's count stays unknown. Closing this would mean having the cleanup record counts too.
- If the database write that fails the run itself errors, the run stays acting. The redelivery that then fails it has no count to store.
Nothing here is visible in the UI, so there are no previews.
PR: #3454
Comment by: @propr-ultrafix (ID: 0)
Model: Claude Opus 5.5
|
✅ Applied the requested follow-up changes in commit 72337ab
Summary of ChangesI fixed F20. A cancelled run whose acting step made changes no longer gets stuck with its count marked as pending when the database write for that count fails. The run stays cancelled, the delivery fails so BullMQ retries it, and the retry saves the count. The next unchanged scheduled run then sees the productive run and does not count it toward a streak. The fix:
The same defect in a second place: if the claiming delivery's final write (the run's completed or failed state) failed after its grant ended, the retry marked the run as failed with no count. That threw away a known count, and the run's unchanged report then made it look unproductive. The saved job data now covers this case too, and the retry stores the count. Tests:
What this doesn't cover:
Separate issue, not fixed: while a run's count is pending, the streak query skips it rather than letting it end the streak. If another unchanged scheduled run finishes during the retry backoff, the streak can still be counted across the productive run and reach the threshold. This comes from the earlier F17 design, not from F20, so I left it alone. There's no visual change, so I didn't make previews. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
… no longer gets stuck with its count marked as pending when the database write for that count fails. The run stays cancelled, the delivery fails so BullMQ retries it, and the retry saves the count. The next unchanged scheduled run then sees the productive run and does not count it toward a streak. I fixed F20. A cancelled run whose acting step made changes no longer gets stuck with its count marked as pending when the database write for that count fails. The run stays cancelled, the delivery fails so BullMQ retries it, and the retry saves the count. The next unchanged scheduled run then sees the productive run and does not count it toward a streak. **The fix:** - **Failed write** (`keepMutationsOfEndedRun`, `src/jobs/processAgentActionJob.ts`): it used to swallow the error. Now the task is still closed to match the cancelled run, but the delivery then throws a new `AgentRunMutationsSettlementError` instead of returning normally. - **Keeping the count**: the revoked grant can't report the count a second time, so it is saved in the job's own data (`job.updateData`, new optional `actingMutationsOwed` field on `AgentRunJobData`). This happens whenever a delivery that claimed the step is about to fail, after the grant has ended. - **Retry**: a retry of the same job reads that field, takes over the claim and saves the count. A separate delivery for the same run doesn't have the field, so it still can't settle the count. **The same defect in a second place:** if the claiming delivery's final write (the run's completed or failed state) failed after its grant ended, the retry marked the run as failed with no count. That threw away a known count, and the run's unchanged report then made it look unproductive. The saved job data now covers this case too, and the retry stores the count. **Tests:** - **New regressions:** three in `test/processAgentActionJob.test.ts`. One covers a failed write, then the retry settling the count. One checks that a separate delivery without the saved count does not settle it. One covers the failed final write. One in `test/agentRunActingMutationsStreak.test.ts` runs against the real SQLite store: two unchanged runs, a third cancelled during its step that made one change, a failed count write, the retry, then another unchanged scheduled run. The run is marked productive, the streak holds only the new run, no Inbox item is sent and the schedule stays on. - **Checking the tests:** with the old processor code put back, all four new tests fail; with the fix they pass. - **Results:** the 8 related root test files pass (221 tests, 0 failures). Root and core typecheck and zero-warning lint on the changed files pass. I didn't run the full suite, and nothing ran against Postgres. **What this doesn't cover:** - **Retry limit:** agent run jobs get 3 attempts with exponential backoff (5s, then 10s). If the count write fails on every attempt, the run stays pending, though the count is still in the failed job's data, so manually retrying that job would settle it. If saving to the job's data fails too (Redis unavailable), the count is lost and only an error is logged. **Separate issue, not fixed:** while a run's count is pending, the streak query skips it rather than letting it end the streak. If another unchanged scheduled run finishes during the retry backoff, the streak can still be counted across the productive run and reach the threshold. This comes from the earlier F17 design, not from F20, so I left it alone. There's no visual change, so I didn't make previews. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds report novelty, acting-operation counts, client badges, and streak notifications with optional schedule pausing. It needs a localized recovery fix before merge. ✅ Consistent presentation — REST, MCP, CLI, and UI expose the productivity flags, with a shared rule for deriving “no change.” ✅ Ordered report comparison — Report storage checks the predecessor under a definition lock and maintains strictly increasing report timestamps. ✅ Substantial regression coverage — Tests cover cancellation, mutation-count persistence, concurrent report storage, notification deduplication, and pause recovery. This review is a static trace of the supplied code; no commands were run. The issue specification is explicitly truncated, limiting validation of its unseen requirements. Current checks show 33 passed, four pending, and no failures. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F21: 🔴 Retry skips the streak evaluation
SuggestionsThese are optional follow-ups and are not sent to S7: 🟢 Align grant-counting documentationThe new documentation says operations are counted before revocation, while ScoreThe implementation has broad coverage and careful persistence handling, but terminal-task recovery can bypass the notification and pause behavior. The remaining pending checks are not themselves blockers. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit 58a883d
Summary of ChangesI fixed F21. If a task write fails after a run's end has been stored, the retry now runs the streak check, so the owner still gets the Inbox notification and an opted-in schedule is still paused. The fix covers the completed and failed acting runs from the finding, plus the same gap in the report step. Action step (
Report step (
A repeated check is safe: the stored notification is looked up first, and a pause already made is not made again. Tests: I added 8 tests across
Test setup changes:
Verification:
Separate finding, not fixed: in 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
…en stored, the retry now runs the streak check, so the owner still gets the Inbox notification and an opted-in schedule is still paused. The fix covers the completed and failed acting runs from the finding, plus the same gap in the report step. I fixed F21. If a task write fails after a run's end has been stored, the retry now runs the streak check, so the owner still gets the Inbox notification and an opted-in schedule is still paused. The fix covers the completed and failed acting runs from the finding, plus the same gap in the report step. **Action step (`src/jobs/processAgentActionJob.ts`)** - Once `taskFollowsRun` has brought the task in line with the run, it now checks the run's streak if this delivery recorded no count but the run is already final. "Final" means completed, failed or cancelled, with `actingMutationsPending` false. - That covers the case from the finding, where the retry records nothing new because a completed run takes no count and a stored count is never replaced. - It also covers a retry that couldn't keep the count with its job, so it doesn't own the claim. - The pending-count guard still holds. A run whose count is still owed is skipped; the delivery that holds the count checks it once it saves it. If saving the count fails, the delivery still throws before any check. - The existing check that runs straight after a count is recorded stays where it was, so a later failed task write doesn't delay it. **Report step (`src/jobs/processAgentRunJob.ts`), same defect** - A run can fail right after its report is stored, for example when the acting step can't be queued. If the task write then failed, the retry went through `reconcileEndedRun`, which ended the task but never checked the run. - It now checks the run once the task has been ended. - The file was already at the 400-line lint limit, so I moved that function's existing return earlier to stay under it. A repeated check is safe: the stored notification is looked up first, and a pause already made is not made again. **Tests:** I added 8 tests across `processAgentActionJob`, `agentRunActingMutationsStreak` and `processAgentRunJob`: - **Completed and failed runs whose task write fails once:** the first attempt fails and checks nothing; the retry checks the run after its task is settled. - **Retry with no saved count:** it still checks the run. - **Run whose count is still owed:** it is not checked. - **Count recorded before the task write failed:** it is checked straight away. - **Real SQLite store and real streak check:** the third unproductive completed or failed run notifies the owner and pauses the schedule on the retry, and a further delivery doesn't notify again. - **Report step:** the failed-after-report retry is checked. **Test setup changes:** - The action test setup now marks a run's count as owed when the step claims it, as the real store does. - One existing test comment said an "already stored" count needs no check. That was the wrong assumption, so I corrected it. - In the SQLite test, the record of sent notifications used to start empty on every check, so a repeated check could look like a duplicate notification. It is now kept per inbox. **Verification:** - With the old source files put back, 7 of the new tests fail; the one that checks the pending guard passes on both versions, as it should. - With the fix, these nine suites pass, 250 tests and 0 failures: action 54, report 79, mutation streak 12, recovery 27, cancel streak 4, productivity 21, novelty 14 + 13, autonomy 26. - Root typecheck and zero-warning lint on `src/` pass. - I typechecked the edited test files separately because the root typecheck skips `test/`: they add no new type errors (21 are already there at HEAD). - I didn't run the full suite or anything against Postgres. There's no visual change, so no preview. **Separate finding, not fixed:** in `approveRun` (`packages/api/routes/automationDefinitionRoutes.ts`), if queuing the acting step fails, `enqueueAgentRunActionOrFail` fails the run and nothing checks its streak. An approved run with an unchanged report that ends up failed this way only counts when a later scheduled run is checked. No task write is involved, so it's outside F21. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds report novelty, acting-operation counts, notifications, and optional schedule pausing across the backend and clients. It needs a localized correctness fix before merge. ✅ Durable settlement — Acting counts survive task-write retries through ✅ Consistent presentation — REST, MCP, CLI, and UI expose productivity information without changing completion states. ✅ Pause recovery — Persisted notification intent and pause records support retrying interrupted pauses while respecting subsequent owner changes. The issue specification is explicitly truncated; this review assesses its visible requirements and the changed behavior. Current checks show no failures and two pending checks. No commands were run. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F22: 🔴 Normalize whitespace before matching timestamps
SuggestionsThese are optional follow-ups and are not sent to S8: 🟢 Reduce streak query payloads
ScoreThe implementation has substantial recovery coverage and consistent client integration, but timestamp normalization still violates a central acceptance criterion. The remaining checks are pending, with no reported failures. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit 8b7c483
Summary of ChangesI fixed F22. A report whose only differences from the previous one are its date and the whitespace inside the timestamp is now classified as Fix ( Same defect elsewhere in the patterns: the time-zone suffix ( Already-stored reports: no migration is needed. The previous report is re-hashed at every comparison rather than reusing its stored hash, so older runs are compared under the new normalisation automatically. No test pins a literal hash. Regression tests (
Verification:
The change has no visible effect in the UI, so I made no previews. I found nothing else to report. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
…ous one are its date and the whitespace inside the timestamp is now classified as `unchanged`. I fixed F22. A report whose only differences from the previous one are its date and the whitespace inside the timestamp is now classified as `unchanged`. **Fix** (`packages/core/src/services/agents/agentRunNovelty.ts`): `normalizeAgentReport` now collapses whitespace to single spaces and trims before it replaces run ids and applies the timestamp patterns. Before, it did that last. The ISO pattern's literal-space date/time separator now matches no matter how the timestamp was wrapped. Both reports in the finding now normalise to `Checked <date>. Dependency watch: …`. **Same defect elsewhere in the patterns:** the time-zone suffix (`22:00 UTC`), the clock-time `am`/`pm` and zone suffixes, and the named-month dates (`Nov\n12,\n2026`) had the same problem. The reordering fixes all of these. I also collapse whitespace in the run ids before matching, the same way as the report text, so run-id replacement works as before. **Already-stored reports:** no migration is needed. The previous report is re-hashed at every comparison rather than reusing its stored hash, so older runs are compared under the new normalisation automatically. No test pins a literal hash. **Regression tests** (`test/agentRunNovelty.test.ts`): - **Normaliser test:** single-space, multiple-space, newline, CRLF and tab layouts of ISO timestamps (with `UTC` and `+01:00` suffixes), clock times with `pm UTC`, and named-month dates all normalise to the same string as their single-spaced form. - **Classifier test:** this is the F22 scenario at the default 40-character minimum. The next night's report with a newline, two spaces or one space before the time is `unchanged`, and its hash equals the previous report's. **Verification:** - Both new tests fail when I put back the old normaliser and pass with the fix. - The novelty, previous-report and process-agent-run/action suites pass: 162 tests, 0 failures. They need `PROPR_DEMO_MODE=true`; without it they stop at the known GitHub-auth banner. - Core `tsc --noEmit` and zero-warning ESLint on the changed source file pass. - I didn't run the full suite. The change has no visible effect in the UI, so I made no previews. I found nothing else to report. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Opus 5.5
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR adds report novelty, acting-operation counts, consistent client flags, and notifications with optional schedule pausing. It needs changes before merge because a worker crash can leave an ended run’s productivity permanently unsettled. ✅ Consistent presentation — REST, MCP, CLI, and UI expose productivity flags, with shared logic for the no-change reason. ✅ Careful report ordering — Report storage checks the predecessor under the definition lock and assigns increasing report timestamps. ✅ Recovery coverage — Added tests cover cancellation, failed persistence, notification deduplication, and interrupted schedule pausing. The supplied checks show 37 passed, none failed, and one pending. The issue specification is explicitly truncated; this review assesses its visible requirements and the changed behavior. No commands were run. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F23: 🔴 Recover pending counts after worker crashes
SuggestionsThese are optional follow-ups and are not sent to No suggestions. ScoreThe implementation has substantial coverage and strong handling of ordinary retries, but abrupt worker termination leaves a durable recovery gap that can affect automatic schedule pausing. Current checks show no failures. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit ef3306d
Summary of ChangesF23 is fixed in the acting-step job, with regressions. No visual preview applies, since the change is worker-side settlement behavior with no UI effect. What was wrong. A delivery only took ownership of a claimed step's count from the obligation its own catch handler saved to the job. After a worker crash that handler never ran, so a redelivery for a run cancelled or failed while the lost worker held the grant reached The correction in
Protection preserved. A duplicate delivery while the step executes in this process is still left alone, since the active-run check gates adoption. Across processes, a redelivery only happens after the executing worker lost its lock. If that worker is in fact still alive, its revoke by grant id finds the grant gone and reports no count, and the store never writes a count over one already stored, so the adopter's count, taken at the moment the grant died, stands. Tests in
Verification. The action job suite passes at 59 tests, the run job and streak recovery suites pass at 106, the root typecheck is clean, and the source file passes zero-warning lint. The report step holds no outstanding-count obligation, so it has no equivalent gap. One residual window, not pursued. If a worker only lost its lock but is still running, a takeover can settle an unknown count first and the claimant's retry then stores the known count, producing two evaluations. That is a race between BullMQ redelivery and a worker the queue considers dead, and the store's write ordering already prevents a known count being lost. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
…isual preview applies, since the change is worker-side settlement behavior with no UI effect. F23 is fixed in the acting-step job, with regressions. No visual preview applies, since the change is worker-side settlement behavior with no UI effect. **What was wrong.** A delivery only took ownership of a claimed step's count from the obligation its own catch handler saved to the job. After a worker crash that handler never ran, so a redelivery for a run cancelled or failed while the lost worker held the grant reached `taskFollowsRun` without ownership. Nothing settled the count, and the run stayed pending and excluded from every streak query. **The correction** in `src/jobs/processAgentActionJob.ts`: - **New claim takeover.** `adoptAbandonedClaim` revokes whatever action grant the run's phase still holds, by phase rather than grant id. It takes the count the API reports from that revoke, settles unknown when the grant is already gone or the revoke fails, and marks the context as owing the settlement. Ending the grant also stops any container the lost worker left behind before the count is fixed. - **Ended-run path.** When the run already ended with its outstanding marker set and no delivery in this process is executing the step, the redelivery adopts the claim before `taskFollowsRun`, which then records the count and evaluates the run. - **Sibling occurrence.** `recoverClaimedRun`, which fails a still-acting run left behind by a stopped worker, had the same gap: it cleared the pending marker without ending the grant or recovering its count. It now adopts the claim first, so the failed run carries the recovered count, and a cancel racing its write still lands the count through the existing settlement path. **Protection preserved.** A duplicate delivery while the step executes in this process is still left alone, since the active-run check gates adoption. Across processes, a redelivery only happens after the executing worker lost its lock. If that worker is in fact still alive, its revoke by grant id finds the grant gone and reports no count, and the store never writes a count over one already stored, so the adopter's count, taken at the moment the grant died, stands. **Tests** in `test/processAgentActionJob.test.ts`: - Redelivery for a cancelled and for a failed run with a crashed claimant settles the recovered count, evaluates the run once, and never re-runs the step. - Grant already gone, or the revoke failing, settles the run with an unknown count. - A recovered count whose write fails is kept with the job, and the retry settles it without ending a grant again. - The abandoned still-acting run is failed with its recovered count, including the owner-cancel race. - The earlier test asserting a redelivery never settles a pending run was reworked to the real protection: a duplicate delivery while the step executes here leaves it to that step. - The earlier F20 test was updated. Since action jobs are deduplicated per run, a delivery without the kept obligation is always a post-lock-loss redelivery. It now settles unknown, and the claimant's retry still stores the known count over that. **Verification.** The action job suite passes at 59 tests, the run job and streak recovery suites pass at 106, the root typecheck is clean, and the source file passes zero-warning lint. The report step holds no outstanding-count obligation, so it has no equivalent gap. **One residual window, not pursued.** If a worker only lost its lock but is still running, a takeover can settle an unknown count first and the claimant's retry then stores the known count, producing two evaluations. That is a race between BullMQ redelivery and a worker the queue considers dead, and the store's write ordering already prevents a known count being lost. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Fable 5.1
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR implements productivity flags across storage, workers, REST, MCP, CLI and UI, with substantial recovery coverage. It needs changes before merge because mutation counts can be lost during a worker crash, and the PostgreSQL pause transaction does not fully serialize against productive completions. ✅ Consistent presentation — Shared productivity rules drive the server’s ✅ Ordered report comparison — Report storage checks the predecessor under a definition lock and assigns strictly increasing report timestamps. The issue excerpt is truncated; this review uses its visible requirements and the changed behavior. The supplied checks show 35 passed, none failed and two pending. No commands were run. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F24: 🔴 Preserve mutation counts across revocation crashes
F25: 🔴 Serialize pausing against productive completions
SuggestionsThese are optional follow-ups and are not sent to S9: 🟢 Read lighter streak records
ScoreThe implementation has strong surface coverage and extensive tests, but two recovery and concurrency defects can incorrectly pause productive automations. Pending checks alone do not lower the score. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit c6fcadc
Summary of ChangesBoth findings are fixed, each with regressions that fail on the old logic and pass on the new one. No visual preview applies: both changes are server-side settlement and locking behaviour with no UI effect. F24: the mutation count survives a crash after revocation
F25: pausing is serialized with productivity-final writes
Verification
Setup change. The Independent observations, not acted on. The pre-existing API test asserted that an already-gone grant reports no count. That assertion was the F24 gap itself, so I updated it. The store's header comment says it is the only module writing 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
… the old logic and pass on the new one. No visual preview applies: both changes are server-side settlement and locking behaviour with no UI effect. Both findings are fixed, each with regressions that fail on the old logic and pass on the new one. No visual preview applies: both changes are server-side settlement and locking behaviour with no UI effect. **F24: the mutation count survives a crash after revocation** - The API now remembers which grant a phase ended. Revocation in `packages/api/mcp/agentRunGrants.ts` writes an ended-grant record in the same transaction that deletes the phase record, so the grant's identity never has a gap. The record is purged once the grant TTL has passed since the revoke. - A revoke that finds no live grant now answers with the grant the phase ended before, with `revoked: false` and the count of that grant's operations. A cleanup fenced by expiry is never answered with it, and a stale grant id names nothing. The sweep only reads the live kind, so ended grants are not revoked again every pass. - The worker's claim adoption in `src/jobs/processAgentActionJob.ts` takes the count whenever the API names a grant, ended now or before. The crash window itself is now a unit test: the first delivery stops right after its revoke, and a second processor instance recovers the count of 1 and fails the run as productive instead of as the third unchanged run. **F25: pausing is serialized with productivity-final writes** - A new module, `packages/core/src/services/agents/agentRunDefinitionLock.ts`, locks the run's definition row through a subquery on the run id. Every terminal transition, every transition carrying an acting count, and the late settlement in `recordAgentRunActingMutations` now take that lock first, in a transaction. The pause transaction already holds the same row lock from its claim update, so a productive write either commits before the streak read or waits until the pause commits. - Lock order is definition then run everywhere, matching the existing report store, so no deadlock cycle is introduced. - The interleaving test runs against a real PostgreSQL 15 that I extracted from Debian packages into /tmp. It injects a competing completion after the pause's streak read: the competitor hits a lock timeout, the pause commits, and the write lands afterwards. The reverse order yields `streak_ended`. The same holds for a late count settlement on a cancelled run. The test skips unless `PROPR_TEST_POSTGRES_URL` is set, because the repository ships only better-sqlite3 and its migration chain is SQLite-only, so the Postgres tables are built from the SQLite schema. An always-on SQLite test asserts the lock statement and transaction boundaries instead. **Verification** | Check | Result | |---|---| | Root, core, API typecheck | pass | | Zero-warning lint on changed source files | pass (core store needed the helpers split out to stay under 400 lines) | | 9 related root suites plus the new locking suite | 244 tests, 0 failures | | API grant, MCP run and definition-route suites | 60 tests, 0 failures | | Postgres interleaving suite | 4 tests, 0 failures | | Negative checks | all 4 locking tests and the F24 worker test fail with the fix reverted | **Setup change.** The `pg` driver is added as an agent-only helper in `.propr/package.json`, and `.propr/setup.sh` installs it. I ran only that new install step, not the whole setup script. A PostgreSQL server is a Debian system package, so it belongs in the agent image at the installation level. Without both, the Postgres suite skips, as it does in CI today. **Independent observations, not acted on.** The pre-existing API test asserted that an already-gone grant reports no count. That assertion was the F24 gap itself, so I updated it. The store's header comment says it is the only module writing `agent_runs`, which the earlier `recordAgentRunActingMutations` already made untrue. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Fable 5.1
|
✅ AI Code Review Complete requested by @propr-ultrafix Posted 1 review: |
🔍 AI Code Review — codex:gpt-6-astraOverall EvaluationThe PR implements productivity flags across storage, REST, MCP, CLI and UI, with substantial recovery and concurrency coverage. It needs changes before merge because mutation accounting can still lose evidence and incorrectly pause productive automations. ✅ Consistent presentation — REST and MCP expose a shared no-change classification, which the UI also uses. ✅ Serialized report storage — The predecessor check under the definition lock prevents concurrent identical reports from both being classified as new. ✅ Recoverable schedule pauses — Notification persistence and pause completion have separate durable records, preserving retries and respecting schedules subsequently re-enabled by owners. The supplied issue specification is truncated; this review assesses its visible criteria and the changed behavior. No commands were run. Reported checks show 37 passed, none failed, one pending and 11 neutral/skipped. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F26: 🔴 Revocation does not finalize mutation admission
F27: 🔴 Recovery evidence expires before settlement
SuggestionsThese are optional follow-ups and are not sent to S10: 🟢 Exercise PostgreSQL interleavings in CIThe PostgreSQL tests skip unless ScoreBroad implementation and substantial tests are strengths, but two mutation-accounting recovery defects can produce incorrect flags and automatic schedule pauses. The pending check alone does not lower the score. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
|
✅ Applied the requested follow-up changes in commit 9c73778
Summary of ChangesNothing else is pending; everything left depends on that run finishing. 🤖 Implementation Details
💡 ProPR Slash Commands
🤖 Generated by ProPR |
…ld logic and pass on the new one. No visual preview applies: both changes are server-side grant settlement with no UI effect. Both findings are fixed with regressions that fail on the old logic and pass on the new one. No visual preview applies: both changes are server-side grant settlement with no UI effect. **F26, revocation now fences receipt admission.** A mutating MCP call writes its receipt inside a transaction that first re-reads the grant row under the same row lock revocation takes. Revocation therefore either waits for an in-flight receipt to commit, so the count taken after it sees that receipt, or commits first, in which case the late request is refused with a `GRANT_REVOKED` error before it runs and leaves no receipt. The worker's normal end-of-step revoke and the abandoned-claim takeover both go through this same route, so both get a complete count. Connect delegations have no stored grant row and are unaffected. - `packages/api/mcp/oauth.ts`: `admitGrant` plus a locked read shared with `revokeGrant`. - `packages/api/mcp/policy.ts`: `admit` entry point used by tool execution. - `packages/api/mcp/operations.ts` and `toolExecution.ts`: the receipt insert runs under the admission transaction. **F27, ended-grant records survive while the count is owed.** The TTL purge now skips an action grant's ended record while its run still has `acting_mutations_pending` set. A redelivery hours later still learns the grant id and recovers the count from its receipts. Records for settled runs, deleted runs, and report grants purge by TTL as before. **Tests.** The grant suite gained an `executeTool` regression with a real grant for F26 and a past-TTL retention regression for F27, in `packages/api/test/agentRunGrants.test.ts`. Its harness now has a minimal `agent_runs` table. The merge-preconditions policy mock gained an `admit` stub. Verification: | Check | Result | |---|---| | API typecheck | pass | | Zero-warning lint on changed files | pass | | Grant suite | 18 pass | | All tool-executing MCP suites plus OAuth and delegation suites | 335 pass, 0 fail, 1 pre-existing skip | | New tests on old logic | both fail | Not covered: nothing ran against Postgres, so the `FOR UPDATE` ordering is verified by inspection only. SQLite serializes the two transactions on its single connection. One residual window is intentional: a receipt admitted just before revocation whose mutation then fails counts as mutating. That errs toward productive, never toward a false no-op. Independent observation, not changed: the existing "an action grant may plan, execute and review" test logs a `no such table: system_configs` error on this branch. It passes, and it predates this follow-up. PR: #3454 Comment by: @propr-ultrafix (ID: 0) Model: Claude Fable 5.1
|
|
|
ProPR runs only on SQLite (
Constraints:
|
|
✅ Applied the requested follow-up changes in commit 7a4a7f5 Summary of ChangesI've removed everything in this PR that existed only for PostgreSQL. One 1. Agent setup
2. PostgreSQL-only test
3. Row locks
Nothing in the PR's diff (docs, changelog or comments) mentions PostgreSQL, Verification
There's no visual preview, since nothing here changes the UI. 🤖 Implementation Details
Undo Changes • View Task Execution 💡 ProPR Slash Commands
🤖 Generated by ProPR |
…stgreSQL. One `.forUpdate()` remains, and two SQLite tests had to change because they checked for the lock query itself; details below. I've removed everything in this PR that existed only for PostgreSQL. One `.forUpdate()` remains, and two SQLite tests had to change because they checked for the lock query itself; details below. **1. Agent setup** - `.propr/setup.sh` now matches `main` exactly, so there is no extra `npm ci` step. - `.propr/package.json` and `.propr/package-lock.json` are deleted. I also deleted the ignored `.propr/node_modules` that held the `pg` packages. **2. PostgreSQL-only test** - In `test/agentRunProductivityLocking.test.ts` I removed the `PostgreSQL: ...` describe block, its proxy helper, `PROPR_TEST_POSTGRES_URL`, both `client: 'pg'` connections and the fixture helper and imports only that block used. - **Changed tests:** the two SQLite tests that stay in that file asserted that the definition lock query ran before the write. That query is now gone, so they check instead that the write runs inside its own transaction (begin before it, commit after it). That transaction is what `BEGIN IMMEDIATE` applies to. **3. Row locks** - **`agentRunReports.ts`:** removed the `agent_definitions … forUpdate()` query, whose result was never used, from the report-storing transaction. The helper is renamed `storeReportUnderWriteLock`. - **`agentRunDefinitionLock.ts`:** replaced by `agentRunFinalTransaction.ts`. The lock query is gone, and `inTransactionIfFinal` still wraps writes that can make a run's verdict final in a transaction. - **`agentRunProductivity.ts`:** - `recordAgentRunActingMutations` keeps its transaction without the lock query. - The comments in `pauseScheduleForUnproductiveStreak` now describe only the write lock. - **`oauth.ts`:** `lockGrantRow` is now `readGrantRow`, without `.forUpdate()`. Its result is still used, both when an operation is admitted under a grant and when a grant is revoked. - **Comments:** I rewrote the ones in `policy.ts`, `agentRunInternalRoutes.ts`, `agentRunStore.ts` and `test/processAgentRunJob.test.ts` to describe only SQLite's write lock (`BEGIN IMMEDIATE`). - **Still on the branch:** one `.forUpdate()` in `revokeAgentRunPhaseGrant` (`packages/api/mcp/agentRunGrants.ts`). It was already on `main` (the PR only rewrote that line), and `main`'s `lockPhaseRecord` uses the same lock. I left it alone because you asked me to remove only what this PR added. Say if you want it taken out too; it does nothing on SQLite. Nothing in the PR's diff (docs, changelog or comments) mentions PostgreSQL, `pg`, `FOR UPDATE` or row locks any more. SQLite behaviour is unchanged: these transactions already start with `BEGIN IMMEDIATE`. **Verification** - Typecheck passes for root, core and api, and eslint passes on the changed files. - The root suites all pass, 266 of 266. That covers report ordering, mutation-count settlement, streak evaluation and pause recovery, including the test where a productive write lands between the streak read and the pause commit. None of those tests were edited. - On the API side, `agentDefinitionRoutes` exited cleanly with no failures. `agentRunGrants`, `mcpAgentRuns` and `mcpMergePreconditions` passed every test, then hung at exit and were killed by the timeout. That exit hang is a known issue with these test files. There's no visual preview, since nothing here changes the UI. PR: #3454 Comment by: @integry (ID: 6828835260) Model: Claude Opus 5.5
|
/review astra fable |
|
✅ AI Code Review Complete requested by @integry Posted 2 reviews: |
🔍 AI Code Review — astraOverall EvaluationThe PR adds report novelty, acting-operation counts, consistent client flags, and recoverable streak notifications and schedule pauses. It needs changes before merge because recovery can finalize productivity while an abandoned acting grant remains usable. ✅ Consistent presentation — REST, MCP, CLI, and UI expose the productivity flags without changing successful runs’ terminal state. ✅ Durable recovery — Ended-grant records preserve mutation-count recovery, and the pause marker prevents repeatedly disabling a schedule the owner re-enabled. ✅ Report ordering — Report storage checks the predecessor transactionally and assigns increasing storage timestamps. The original objective is explicitly truncated; this review assesses its visible requirements and the complete authorized follow-up. Supplied checks show 30 passed, six pending, and no failures. No commands were run. Merge blockersEvery finding below was introduced by this PR and must be resolved before merging. F28: 🔴 Keep settlement pending when revocation fails
SuggestionsThese are optional follow-ups and are not sent to S11: 🟢 Assert the actual transaction modeThe new locking tests accept any statement beginning with ScoreThe implementation has substantial coverage and coherent cross-surface behavior, but failed-revocation recovery can produce an incorrect durable verdict and schedule pause. The pending checks are not themselves blockers. Score: 6/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
🔍 AI Code Review — fableOverall EvaluationThis PR implements #3442 end to end: a normalised report hash and The design is unusually careful about the awkward paths (owner cancel while the step holds its grant, lost task writes, lost count writes, worker crash between revoke and count store, pause interrupted after the item is stored), and each is backed by a test against the real SQLite store. I could not drive any path to an incorrect persistent state. The main residual risks are verification gaps (three test shards and Validate Changes still pending) and a few deliberate design trade-offs noted below as suggestions. ✅ Compare-and-store under one write lock — ✅ Count survives the revoke→store crash window — ✅ Pause evidence separated from notification intent — the immutable Inbox item records only Verdict: merge-ready within scope, conditional on the pending shards and Validate Changes finishing green. Merge blockersNo merge blockers. SuggestionsThese are optional follow-ups and are not sent to S12: 🟢 Retry instead of settling an unknown count
S13: 🟢 Confirm no generic expiry sweep deletes ended-grant rowsF27 retention relies on S14: 🟢 API route depends on a worker module
S15: 🟢 Badge while the acting step is undecided
S16: 🟢 Normalisation false positives
S17: 🟢 Cosmetic outcome for a deleted definition
S18: 🟢 Watch file growth against the lint gate
S19: 🟢 Fixture fakes outside the changed tests
ScoreThe effective objective (including the PostgreSQL removal) is met, the changed paths hold up under tracing, and the new behaviour is thoroughly tested against the real store; readiness is conditional only on the pending shards and Validate Changes. Score: 8/10 🤖 Review Details
Copy and edit to choose records from this review: 🤖 Review by ProPR |
AI Implementation Summary
Closes #3442
Branch:
3442/claude-opus-5-5-flag-automation-runs-that-20261111-2200-km0Commits: ✅ Changes committed (60cb085)
AI Processing Completed
Execution Details:
Summary:
Runs that produce nothing are now flagged on every surface, and the owner gets one Inbox notification after N unproductive scheduled runs in a row. Typecheck and lint pass, and every test suite I ran passes. I didn't run the full test suite, and I only ran the migration against SQLite, not Postgres.
How runs are flagged
report_noveltyofnew,unchangedorempty. Normalisation collapses whitespace and replaces dates, times, weekdays, epoch timestamps, "3 hours ago"-style times, UUIDs and run ids. A report under 40 characters after normalisation isempty. The previous run's report is re-hashed rather than read from its stored hash, so runs from before this change compare correctly. Version numbers like4.17.21are left alone.mcp_operationsrows for the grant, minus those that definitely failed. The acting job now revokes the grant before it marks the run completed, so the count is stored while the run is stillacting. If the count can't be read it staysnull, which is not treated as a no-op.get_automation_run/list_automation_runs, the CLI (a new Change column inruns, achange:line inreport), and a "No change" badge in the runs list and run detail. I also added a derivednoChangeReasonfield to the payload so every client shows the same badge.Streak, notification and auto-pause
automation_unproductive_run_threshold(1–50, default 3). It's available through the API, MCP and the settings catalog, but not in the Settings UI, the same as the existing usage-pause setting.pauseWhenUnproductive(off by default) appears as a checkbox under the schedule. When the notification fires, it turns the schedule off throughdisableAgentDefinitionSchedule, logs the reason, and says so in the notification.Decisions you may want to revisit
system_failurecategory instead. Say if you'd rather they get no notification.processAgentRunJob.tswas already at the 400-line lint limit, so I movedagentReportRecapinto its own file and kept the old export.Tests and docs
Previews of the runs list (desktop and phone), the run detail, and the new schedule checkbox were captured with Playwright.
Detailed Logs:
77f36b09-bc1b-45e7-a1dd-661b41e959f2Log files stored at:
/usr/src/app/logs/executions/issue-3442-2026-11-11T22-38-41-631Z-conversation.json/usr/src/app/logs/executions/issue-3442-2026-11-11T22-38-41-631Z-output.txtLatest Conversation Messages
This PR was created automatically by ProPR after processing issue #3442.
💡 Need changes?
Comment on this PR to request refinements — the AI agent monitors comments and will update the implementation based on your feedback. Keep iterating until you're satisfied!
Visual preview
Run detail of an unchanged run
The run header shows the same "No change" badge next to the Completed state.
Connect sign-in required. Original retained until 2027-02-09T22:38:46.148Z.
View the full-resolution original in ProPR Connect
Run history with "No change" badges (desktop)
Scheduled runs whose report was unchanged or empty, or whose acting step changed nothing, still show Completed but carry a "No change" badge; productive runs have none. Columns stay aligned.
Connect sign-in required. Original retained until 2027-02-09T22:38:47.565Z.
View the full-resolution original in ProPR Connect
Run history on a 390px phone
The badge sits beside the state on the stacked phone layout without horizontal panning.
Connect sign-in required. Original retained until 2027-02-09T22:38:48.991Z.
View the full-resolution original in ProPR Connect
Schedule option to pause unproductive automations
New per-automation checkbox under the cron schedule (off by default) that turns the schedule off when the unproductive-run notification fires.
Connect sign-in required. Original retained until 2027-02-09T22:38:50.156Z.
View the full-resolution original in ProPR Connect