Skip to content

fix(thrift): fail operations after unrecoverable fetch errors - #631

Draft
vuanhphung wants to merge 4 commits into
mainfrom
vuanhphung/thrift-fetch-after-error
Draft

vuanhphung wants to merge 4 commits into
mainfrom
vuanhphung/thrift-fetch-after-error

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Thrift could advance its Arrow/file cursor and then reject a fetch before returning the accumulated rows. Retrying the operation silently returned only a suffix (RESULTFETCH-024/025 in databricks/databricks-driver-test).

Any failed Thrift FetchResults call or CloudFetch download latches the original fetch error: every later row read rethrows that same error instance, even if cleanup fails. Callers re-execute instead of fetching again (FetchResults is not retried). Cleanup is unchanged: it runs in the background, shares one close request with explicit close(), safely contains logger failures, and leaves unfinished explicit close failures retryable. Argument handling, SEA, retries, downloads, and reader policy are unchanged; main's late-invalid-argument and overlapping-reader gaps remain follow-ups.

Verification: all 1,817 unit tests, lint, prettier, types, and build pass. Fifteen focused tests verify original-error identity, pending/failed cleanup, throwing loggers, close races/retries, and real Arrow trailing prefetch; seven identity assertions fail on e683624 before this change. The inherited agent-marker failure was already reproduced on main; clearing only those markers gives the clean unit run. CLOUDFETCH-017 passes as written on driver-test main 261bb38dc for both backends: Thrift repeat drain returns the original download error immediately, with 2 downloads; SEA is unchanged. At new tripwire-only head 5c87979f, both Thrift FetchAfterError cases yield required XPASS(RESULTFETCH-024) / XPASS(RESULTFETCH-025) 3/3, each with zero recovered rows and three typed later errors. SEA 025 passes; 024 is Thrift-only. Seven repository live e2e tests and fifteen shared assertion guards pass. Full replay exits 0: Thrift 610 passing / 628 pending; SEA 697 passing / 539 pending (pending cases are not live passes).

No conformance contract change is required. driver-test #2184 adds only the FetchAfterError tripwires. A full live CI run against driver-test main is dispatched with commit_sha=68d3f73cdd1ea8e554bc013daf923ef3a5f318d5, proxy_mode=passthrough, and no PR-number check.

Prior-head (1319102) fresh-process fetchAll medians in milliseconds (100,000 rows, three runs per cell; alternating rebuild/run order; shared warehouse):

Backend / query main This branch
Thrift / inline 392.819 415.795
Thrift / CloudFetch-sized 1174.499 1026.405
SEA / inline 87.425 92.867
SEA / CloudFetch-sized 178.061 172.564

The unchanged SEA control also moves; shared-warehouse measurements establish no causal improvement or regression. Subsequent deltas affect failed-fetch handling and close, outside the timed successful fetch path.

Cross-driver state

Driver State Notes
JDBC Partial DatabricksResultSet.java:410 @97dac8d rethrows; LazyThriftInlineArrowResult.java:322 then latches EOF, not repeated typed errors. Potential completeness gap; source-only audit. No failure-policy option.
Python Partial result_set.py:327,334,384,387 @087b2bf advances before refill without a failure latch; use_kernel=False. Exhausted RPCs raise RequestError; potential gaps, source-only audit.
Go Partial resultPageIterator.go:164,238 @44b21a6 retains a typed request error; public database/sql stops on iteration errors. Raw Arrow iterators clear a pending error. No failure-policy option.
Node.js Partial → Supported (Thrift); SEA Partial ThriftOperationBackend.ts:217,244 @68d3f73 retains and rethrows the original fetch error instance/type/message, even after cleanup failure. Existing SEA gives CLOSED after successful cleanup but resets isClosed when cleanup fails (KernelOperationLifecycle.ts:175,270 @fc085d4); unchanged, separate gap. useKernel=false.
Kernel Not applicable napi/src/statement.rs:321 @dac32e2 requires close/discard after fetch failure. This binding contract does not guarantee adapter safety when cleanup fails; no Rust change.

Textual overlap audit: #623 needs a rebase in ThriftOperationBackend.ts and CHANGELOG.md; #605, #622 (dropped), and #627 conflict only in the Unreleased changelog. #610 merges cleanly with this change. No overlapping PR was modified.

Adversarial review: round-two cleanup delta converged; the replacement original-error design is pushed at 68d3f73 for delta review. R2-1/R2-7 remain deferred per orchestrator; PR stays draft.

This pull request and its description were written by Isaac.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

This branch was successfully deployed

1 active deployment
azure-prod — 68d3f73c Deployed Nov 19, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #2149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant