Repository navigation
fix(handler,mcp): a COMMIT whose connection broke says unknown on every write path, the confirm notes its write at the commit, and the remaining writes name write_outcome (#483) - #490
Merged
Conversation
…own, not rolled_back, and a test per write path holds its classifier Part of #483 (point 1). pgx v5.11 reports a connection that breaks while a COMMIT waits for its answer as pgconn.ErrConnClosed, the lock error of a call on a closed connection, marked safe to retry (MultiResultReader.NextResult drops the read error). writeOutcome and statementOutcome read that as "never sent" and said rolled_back for a write the database then committed - on every labeled bracket (MCP store, update, archive, guard_resolve, store_batch, the staged batch, the confirmed store/update/archive/batch, REST manage update, guard-resolve with ids, ingest block mode) and on the issue tools. Both classifiers now say unknown for it. A TCP proxy between the handlers' pool and a real PostgreSQL cuts the connection while a deferred constraint trigger holds the commit; every write path's failure line and MCP tool line must say unknown and the held commit must land afterwards. A second test runs probe B of the #482 review: a COMMIT whose deadline passes while it waits ends with pgconn's "timeout: context deadline exceeded", statementOutcome says unknown, and writeOutcome on an unlabeled bracket would say rolled_back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmit, and a failed execute names the block or blob of the card Part of #483 (points 2 and 13). mcpConfirmHandler noted the confirm's outcome after executeConfirm returned, so a panic in the afterwork of a confirmed store (the classification after the upsert) left the tool line at write_outcome=none for a block that stands. The confirm cores now note where the write ends, as the other write tools do: committed right after the commit, before the afterwork, and a failed execute with the classifier of its store function next to the failure line - together with the block (update, archive) or blob (blob_link) the card names, which the line lacked on confirmExecErr. A confirmed batch names its item count also when it fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ame it, and the census allows none Part of #483 (point 8). manage delete and dream-link-resolve take writeOutcome (labeled brackets); guard-resolve of one id, the chunk ingest, the blob delete and the chat's staged store and update take statementOutcome (one statement each); the REST issue writes (manage issue-create, issue-update, issue-comment-create and the three project issue routes) take statementOutcome through writeIssueWriteError (the issue transaction carries no commit label); a purge says committed when an earlier batch removed something, else what its failing statement left behind. The census's list of paths without an outcome is empty. Each path has a cut test: its commit is held and its connection cut, and the failure line must say unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…largest batch whose afterwork fits its one follow-up bound Part of #483 (point 3). A batch classifies and enriches all its written blocks after the commit under one 10 s bound inside the 15 s shutdown drain, and nothing classifies a block later but its next write (no scheduler, guard or dream backfill for type_name). The setting had no upper limit. Measured on a test database on a loaded host: about 3 ms per block - 100 blocks 0.29 s, 1000 blocks 3.0 s, 5000 blocks 16.1 s, past the drain, so a bound tied to the batch size could not have covered it either. V67 refuses a value above 1000 (boot abort for the environment value, 422 for a settings write, a psql row dropped with a WARN), and the handler test now measures a batch of the ceiling's size inside the bound (2.7-2.8 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the follow-up bound, and a failed reload answers that the toggle is written Part of #483 (point 4). The backend pool reload and the settings reload after the eject/gaming toggle shared one 10 s follow-up context: a backend reload that hung for the whole bound ended the settings reload before it started, and the client got a generic 500 "internal error" for a toggle that had committed. Each reload now runs detached on 5 s of its own (the pair stays within the 10 s bound inside the 15 s drain), and a failed settings reload answers 500 with persisted=true and a text that says the toggle is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not as the client spelled its id Part of #483 (point 5). store.BlobWriteScope, the stage-time scope lookup of a blob_link card, also returns the blob's id as stored; the card carries it and the MCP line of the stage names it, as the direct path names blob.ID since 0f31fed. A test stages a link with an upper-case and a braced uuid and reads both the stage line and the confirm line. The cut tests' proxy refuses new connections until the held backend is gone: pgx sends a cancel request on a new connection after a broken one, which could cancel the held COMMIT before the test released it and made the landed check race. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #483 (point 6). A COMMIT whose backend is terminated (57P01) while it waits for a synchronous standby has committed locally although the client receives an error. Not built (no such deployment); the write_outcome paragraph says it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and under an aliased store import Part of #483 (point 7). The census matched only `x, err := store.F(...)`, `err = store.F(...)` and `return store.F(...)` under the name "store". It now also follows `if x, err := pkg.F(...); err != nil {...}` (the if's body is the failure branch) and reads the name each file imports the store package by. A synthetic file holds both forms; the package still has 33 write calls, none of them in the new forms. `return store.F(...)` stays counted, not followed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…RROR line with its write_outcome Part of #483 (point 9). Ten admin write routes answered a bare 500 with no line: tenant-create, tenant-update, tenant-delete, tenant-limit-set, scope-create, tenant-grant-create, tenant-grant-delete, block-grant-create, block-grant-revoke and tenant-quota-set - rights and budgets that changed or did not, with no trace. Each now writes "manage: <action> failed" with the error, write_outcome (writeOutcome for the labeled tenant bootstrap and scope assignment, statementOutcome for the single statements and the unlabeled quota bracket, unknown for tenant-delete, whose statements commit one by one) and the request id; no request argument, name or key. The backend-* (4) and embed-migration-* (8) routes are left for a follow-up: their failures run through helpers without the error (writeBackendTxFailure) or answer 4xx and 500 from one mapper (embedMigrationMechErr). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… client that leaves is seen leaving Part of #483 (point 11). The route takes no body, but the CLI sends {"trigger":"manual"}, and net/http watches a connection for its client going away only once the request body is consumed: such a client was noticed only when the answer was written. The handler now drains the body first (bounded by the route's 1 MB MaxBodySize). The digest tests send the CLI's body; on the base the server saw the client go only when the handler returned, past the classification bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… gaming toggle answers persisted, the admin paths' classifiers and the deadline test hold under a cut Part of #483. - The gaming toggle's backend pool reload carries the toggle; its failure now answers 500 with persisted=true as a failed settings reload does (reloadBackendPool returns its error), and the text names the NOTIFY listeners. - The cut tests cover the seven admin writes that take statementOutcome (tenant-update, tenant-limit-set, tenant grants, block grants, quota). - TestStatementOutcomeDeadlineAtCommit runs on the proxied pool and refuses pgx's cancel request until the held commit went on. - The census also walks switch and select cases; V67's comment states the measurement and the live setting; the docs say a digest run ends with its client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 19 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A COMMIT whose connection broke says
unknown, notrolled_back(point 1). pgx v5.11 reports a connection that breaks while a COMMIT waits for its answer aspgconn.ErrConnClosed— the lock error of a call on a closed connection, marked safe to retry (itsMultiResultReader.NextResultdrops the read error).writeOutcomeandstatementOutcomeread that as "never sent" and saidrolled_backfor a write the database then committed: on every labeled bracket (MCP store, update, archive, guard_resolve, store_batch, the staged batch, the confirmed store/update/archive/batch, REST manage update, guard-resolve with ids, ingest block mode) and on the issue tools. Both classifiers now sayunknownfor it (client_abort.go,mcp_write_line.go). pgx also sends a cancel request on a new connection after such a break, so whether the held COMMIT lands is a race in production;unknownis the only honest word.A test per write path holds its classifier (point 1).
write_outcome_cut_integration_test.go: the handlers' pool runs through a TCP proxy; a deferred constraint trigger holds the commit of the path's write on an advisory lock; once it waits, the proxy closes both sides and refuses new connections. Every path's failure line and MCP tool line must sayunknown, and the held commit must land once the lock goes away — 12 statement paths, 13 labeled paths, the 11 paths of point 8 and the 7 admin writes of point 9 that takestatementOutcome. A second test runs probe B of the #482 review: a COMMIT whose deadline passes while it waits ends with pgconn'stimeout: context deadline exceeded(not safe to retry),statementOutcomesaysunknown, andwriteOutcomeon an unlabeled bracket or single statement would sayrolled_back. A type split between the two classifiers was checked and is not small:pgxdb.WriteOptspassesfn's error through unchanged (its documented contract), so a bracket marker would change the identity of everyfnerror in every package.The confirm tool notes its write at the commit (points 2, 13).
mcpConfirmHandlernoted the outcome afterexecuteConfirmreturned, so a panic in the afterwork of a confirmed store left the line atwrite_outcome=nonefor a block that stands. The confirm cores note where the write ends — committed right after the commit, before the afterwork; a failed execute with its store function's classifier next to the failure line, and the block (update, archive) or blob (blob_link) the card names, which the line lacked onconfirmExecErr. Sweep of the eleven write tools: only the confirm had code between its commit and its note.write_outcomeon the eight remaining paths (point 8). manage delete and dream-link-resolve (labeled brackets,writeOutcome); guard-resolve of one id, chunk ingest, blob delete, the chat's staged store and update (one statement each,statementOutcome); the REST issue writes — manageissue-create,issue-update,issue-comment-createand the three project issue routes — throughwriteIssueWriteError(statementOutcome, the issue transaction has no commit label); a purge sayscommittedwhen an earlier batch removed something. The census allows no path without an outcome.writes.batch_max_itemsat most 1000 (point 3). No path classifies a block later but its next write (no scheduler, guard or dream backfill fortype_name). Measured afterwork on a test database on a loaded host: about 3 ms per block — 100 blocks 0.29 s, 1000 blocks 3.0 s, 5000 blocks 16.1 s, past the 15 s drain, so a bound tied to the batch size could not cover it either. V67 refuses a larger value (boot abort for the environment value,422for a settings write, a psql row dropped with a WARN), like V43 did fordispatch.external_max_wait.Gaming toggle (point 4). The backend pool reload and the settings reload after the toggle each run detached on 5 s of their own (together inside the 10 s bound and the 15 s drain); a hung backend reload no longer ends the settings reload before it starts. Either reload failing — the backend pool's carries the toggle on this instance — answers
500 {"success": false, "persisted": true, "error": "persisted but reload failed — the toggle is written …"}instead of a bare "internal error" (and a failed backend reload no longer answers200).The staged blob_link names the stored id (point 5).
store.BlobWriteScopealso returns the blob's id as stored; the staged card and the stage line carry it (the answer text keeps the client's spelling).Census (point 7). It now sees
if x, err := pkg.F(…); err != nil {…}, an aliased store import and writes in switch/select cases;return store.F(…)stays counted, not followed.Admin write failures log (point 9, tenants, scopes, grants, quotas).
tenant-create,tenant-update,tenant-delete,tenant-limit-set,scope-create,tenant-grant-create,tenant-grant-delete,block-grant-create,block-grant-revokeandtenant-quota-setwrite one ERROR linemanage: <action> failedwith the error,write_outcome(unknownfortenant-delete, whose statements commit one by one) andrequest_id— no request argument, name or key.POST /api/digestdiscards its body (point 11). The CLI sends{"trigger":"manual"}; net/http watches the connection only once the body is consumed, so such a client was noticed only when the answer was written. The run now ends with a client that leaves.Docs (point 6 and the above).
docs/operations.md:rolled_backassumes no synchronous standby; the paths, the batch ceiling, the gaming answer, the admin lines, the digest.docs/api.md,docs/mcp.md: the ceiling and the gaming answer.Why
Part of #483. Open: point 9 for
backend-*andembed-migration-*, points 10 and 12 (listed in the issue comment). Rejected alternative for point 3: a bound proportional to the batch size, capped inside the drain — the measured 5000-block afterwork (16.1 s) exceeds any cap inside the 15 s drain.Behaviour changes (intended)
write_outcome=unknown(wasrolled_back).writes.batch_max_itemsabove 1000 is refused (an environment value aborts the boot). The live.envsets no value (default 100).persisted: trueand a new error text (500 as before for a failed settings reload; a failed backend-pool reload answered 200 and now answers 500).POST /api/digestduring the run is seen leaving: the run ends with it, as every request does (its classifications after the upsert stay detached).write_outcome; ten admin routes gain a failure line.Consumers of the changed contracts
writeOutcome,statementOutcome(ErrConnClosed)write_outcomesite ininternal/handler(census: 33 write calls)TestWriteOutcome,TestStatementOutcome,TestWriteOutcomeBrokenCommit, the cut testsnoteConfirmOutcomeremovedmcpConfirmHandler,ConfirmHandler(REST, notes are no-ops there)TestMCPConfirmLine,TestMCPWriteLineEveryWriteToolCommitsstore.BlobWriteScopereturns the stored idmcpStageBlobLink; store testTestMCPStagedBlobLinkNamesTheStoredID,TestBlobBlockRef…writeIssueStoreError(…, attrs),writeIssueWriteErrorProjectIssuesHandlerTestWriteOutcomeCutRemainingPathswriteBootstrapError(w, r, err)handleTenantCreateTestAdminWriteFailureLinesconfig.WritesBatchMaxItemsCeiling, V67config.Validate(boot, settings PUT, reload)TestWritesBatchMaxItemsCeiling,TestFollowupBoundBatchAfterworkFitsreloadBackendPoolreturns its errorcli/eject.go) readsuccess/erroronly;reloadAfterMutationdrops the error as beforeTestFollowupBoundGamingReloadFailureSaysWritten,TestFollowupBoundGamingReloadsHaveTheirSharesCost
No new statement on a write path.
BlobWriteScopeselects one more column. The digest route reads at most the 1 MB its middleware allows. The new log lines are written only on failure.Verification
168b7840, green after: the cut tests — 13 labeled paths and the 3 issue tools saidrolled_backfor a commit that then landed (TestWriteOutcomeBrokenCommit: both classifiersrolled_back); the 11 point-8 paths had nowrite_outcome;TestMCPConfirmLine(write_outcome=noneafter a panic past the commit, noblock_id/blob_idon a failed execute);TestMCPStagedBlobLinkNamesTheStoredID(upper-case and braced id on the stage line);TestFollowupBoundGamingReloadsHaveTheirShares(settings reload ran on an ended context) and…FailureSaysWritten(bare "internal error");TestAdminWriteFailureLines(10/10 without a line);TestWritesBatchMaxItemsCeiling(1001 and 5000 accepted);TestFollowupBoundDigestwith the CLI's body (the server saw the client go only when the handler returned). Unit:TestWriteOutcome,TestStatementOutcomewith pgx's ErrConnClosed shape. Green on the base by design (they pin correct code):TestStatementOutcomeDeadlineAtCommitand the single-statement cut paths.DeadlineExceededcounted as rolled back instatementOutcome; the confirmed blob ops noted withnoteToolWrite; the stagedblob_linknoted as labeled) and 25 of our own (each classifier swap on a statement path, the ErrConnClosed branches, the confirm notes, the purge outcome, V67, the gaming shares and answer, the stored blob id, the digest drain, the admin lines and classifiers, the census forms).bash go/gates.sh wave --final --base 168b7840: rc 0 — build, vet, lint, doccheck, short (231 s), census, race (319 s), integration of./internal/handler/(579 s),./internal/store/(659 s),./cmd/ctxd/,./internal/config/,./internal/handler/v1door/,./internal/testdb/(6/6 ok), python, deadcode, deadcode-testonly.-count=5, with and without-race(the two slow digest/batch tests-count=2).level=ERROR msg="mcp: store error" error="store: upsert block: commit: conn closed" write_outcome=unknown request_id=…level=INFO msg="mcp tool write" tool=store write_outcome=unknown duration_ms=20 request_id=…level=ERROR msg="manage: tenant-grant-create failed" error="store: create tenant grant: ERROR: probe fault (SQLSTATE P0001)" write_outcome=rolled_back request_id=sample-reqNot verified
writes.batch_max_itemson the live instance was not read (needs an admin key).writeIssueWriteErrorwith the cut-tested manage routes; they have no cut test of their own.🤖 Generated with Claude Code