Repository navigation
fix(thrift): fail operations after unrecoverable fetch errors - #631
Draft
vuanhphung wants to merge 4 commits into
Draft
vuanhphung wants to merge 4 commits into
vuanhphung wants to merge 4 commits into
Conversation
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thrift could advance its Arrow/file cursor and then reject a fetch before returning the accumulated rows. Retrying the operation silently returned only a suffix (RESULTFETCH-024/025 in databricks/databricks-driver-test).
Any failed Thrift
FetchResultscall or CloudFetch download latches the original fetch error: every later row read rethrows that same error instance, even if cleanup fails. Callers re-execute instead of fetching again (FetchResultsis not retried). Cleanup is unchanged: it runs in the background, shares one close request with explicitclose(), safely contains logger failures, and leaves unfinished explicit close failures retryable. Argument handling, SEA, retries, downloads, and reader policy are unchanged; main's late-invalid-argument and overlapping-reader gaps remain follow-ups.Verification: all 1,817 unit tests, lint, prettier, types, and build pass. Fifteen focused tests verify original-error identity, pending/failed cleanup, throwing loggers, close races/retries, and real Arrow trailing prefetch; seven identity assertions fail on e683624 before this change. The inherited agent-marker failure was already reproduced on main; clearing only those markers gives the clean unit run.
CLOUDFETCH-017passes as written on driver-test main 261bb38dc for both backends: Thrift repeat drain returns the original download error immediately, with 2 downloads; SEA is unchanged. At new tripwire-only head 5c87979f, both Thrift FetchAfterError cases yield requiredXPASS(RESULTFETCH-024)/XPASS(RESULTFETCH-025)3/3, each with zero recovered rows and three typed later errors. SEA 025 passes; 024 is Thrift-only. Seven repository live e2e tests and fifteen shared assertion guards pass. Full replay exits 0: Thrift 610 passing / 628 pending; SEA 697 passing / 539 pending (pending cases are not live passes).No conformance contract change is required. driver-test #2184 adds only the FetchAfterError tripwires. A full live CI run against driver-test main is dispatched with
commit_sha=68d3f73cdd1ea8e554bc013daf923ef3a5f318d5,proxy_mode=passthrough, and no PR-number check.Prior-head (1319102) fresh-process
fetchAllmedians in milliseconds (100,000 rows, three runs per cell; alternating rebuild/run order; shared warehouse):The unchanged SEA control also moves; shared-warehouse measurements establish no causal improvement or regression. Subsequent deltas affect failed-fetch handling and close, outside the timed successful fetch path.
Cross-driver state
DatabricksResultSet.java:410@97dac8d rethrows;LazyThriftInlineArrowResult.java:322then latches EOF, not repeated typed errors. Potential completeness gap; source-only audit. No failure-policy option.result_set.py:327,334,384,387@087b2bf advances before refill without a failure latch;use_kernel=False. Exhausted RPCs raiseRequestError; potential gaps, source-only audit.resultPageIterator.go:164,238@44b21a6 retains a typed request error; publicdatabase/sqlstops on iteration errors. Raw Arrow iterators clear a pending error. No failure-policy option.ThriftOperationBackend.ts:217,244@68d3f73 retains and rethrows the original fetch error instance/type/message, even after cleanup failure. Existing SEA givesCLOSEDafter successful cleanup but resetsisClosedwhen cleanup fails (KernelOperationLifecycle.ts:175,270@fc085d4); unchanged, separate gap.useKernel=false.napi/src/statement.rs:321@dac32e2 requires close/discard after fetch failure. This binding contract does not guarantee adapter safety when cleanup fails; no Rust change.Textual overlap audit: #623 needs a rebase in
ThriftOperationBackend.tsandCHANGELOG.md; #605, #622 (dropped), and #627 conflict only in the Unreleased changelog. #610 merges cleanly with this change. No overlapping PR was modified.Adversarial review: round-two cleanup delta converged; the replacement original-error design is pushed at 68d3f73 for delta review. R2-1/R2-7 remain deferred per orchestrator; PR stays draft.
This pull request and its description were written by Isaac.