Repository navigation
fix(thrift): honor per-statement rowLimit - #623
vuanhphung wants to merge 7 commits into
Conversation
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Medium
Looks good — 1 low concern. Thrift now forwards rowLimit as resultRowLimit, and its Number coercion matches the kernel path (KernelSessionBackend.ts:311). Unset requests stay uncapped, and only statementConf still triggers the warning.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
Other findings
- 🟡 Medium — The new tests inspect a stubbed request but the repository's e2e suite does not exercise
rowLimit, so it cannot catch regressions in actual capped results or fetch exhaustion. Add a warehouse e2e test for capped and uncapped results, including zero and a multi-fetch case. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
|
Re: the Medium e2e-coverage finding: live warehouse coverage is in the shared conformance suite, not just the mocked unit tests: RESULTFETCH-020..023 at the tested coverage head. All four XPASS on final head f0fa993 after checking wire caps, exact ordered rows, inline/CloudFetch delivery, payload integrity, and fetch exhaustion (including cap 100003 over 200000). Main reproduces all four bugs. Eight additional live checks on this head confirm numeric-string and zero/null/NaN/fraction behavior across Thrift and SEA. I’m retaining the assignment’s shared-conformance coverage rather than duplicating it in this repository; the expected-failure wrappers will be flipped separately after merge. No uncovered runtime regression is demonstrated by this finding. This comment was generated with GitHub MCP. |
…nversion Enforce rowLimit across result pages and all Thrift result formats, including legacy columnar overflow. Validate the documented normalized range before constructing Int64 so large inputs cannot wrap into a different limit. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Medium
Looks good, no blocking issues found. rowLimit is validated before any RPC, sent as resultRowLimit, and capped locally in ThriftOperationBackend for both fetchChunk and hasMore. Unit tests cover coercion, range limits and multi-page capping, and e2e tests cover inline Arrow and columnar results. Nit: the e2e test never restores its getConfig stub.
This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.
Other findings
- 🟡 Medium — For Arrow results, the converter prefetches another batch before returning the current one, so a failure in data beyond the cap can reject a fetch even when the requested rows are already available. Stop prefetching once the remaining row limit is satisfied and test this with a server that ignores the wire cap. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
🔀 1 finding considered and dismissed by the judge
- 🟡 Medium
lib/thrift-backend/ThriftOperationBackend.ts:176— A zero limit skips validation of an already-received direct-results fetch response, silently turning its error status into an empty successful result. Validate
Judge's evidence:if (this.remainingRows === 0) { return []; }
|
Round-2 review follow-up for 36cfd70:
Both maintainer findings are fixed; independent round-2 re-review remains pending. This comment was generated with GitHub MCP. |
Carry the statement budget into Arrow conversion and CloudFetch link scheduling, exempt automatic staging instructions, and preserve numeric fetch-size coercion. Clarify backend-specific validation and cover oversized Arrow pages, failing later pages, bounded downloads and zero-limit staging. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
Superseding my earlier Arrow-prefetch pushback: 2f1696a fixes all six requested round-2 follow-ups. Thrift budgets now stop Arrow prefetch and CloudFetch link scheduling at the cap, including oversized server pages; internal staging instructions remain readable at zero when automatic staging is enabled. Validation docs are explicitly Thrift-scoped, fetch-size Number coercion is preserved, and the coverage citation is refreshed to 21d55463f. 69 new real-stack regressions pass (29 failed before the fix), including failing later Arrow pages, real HTTP CloudFetch download counts and a zero-limit staging DELETE. All 1,912 units, local checks, 24 live cases and the four refreshed-head XPASS tripwires pass their assertions. PR CI is green; test-merge replay is running at https://github.com/databricks/databricks-driver-test/actions/runs/44398849306. Keeping this PR a draft for re-review. This comment was generated with GitHub MCP. |
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Medium
Looks good, no findings. The row cap is tracked consistently at three layers: the operation backend, the Arrow converter and CloudFetch link scheduling. Tests cover direct results, buffered and unbuffered paths, prefetch and staging. Not verified: whether a real server honoring resultRowLimit=0 still returns the staging-instruction row; only mocks test this.
This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.
Other findings
- 🟡 Medium — The staging test always returns an instruction row regardless of the forwarded
resultRowLimit, so it cannot detect a server applying the zero cap before the client exemption. Add a live staging transfer withrowLimit: 0or an integration fake that applies the wire cap. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
🔀 1 finding considered and dismissed by the judge
- 🟡 Medium
lib/thrift-backend/ThriftOperationBackend.ts:181— WithrowLimit: 0, this return silently discards an error status already present in the direct-results fetch response, which is normally checked byRowSetProv<br>Judge's evidence:if (this.remainingRows === 0) { return []; }`
Floor an explicitly set wire row limit at one when staging is enabled, while retaining the requested client cap for ordinary results. Cover an Arrow server that honors the wire cap and a default-Arrow live transfer. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
17153bb fixes the staging finding. The default-Arrow live repro failed before the change: resultRowLimit=0 removed the instruction before the client could exempt it. With stagingAllowedLocalPath, set normalized wire limits now have a floor of one; unset stays omitted, and ordinary results retain the original client cap. New Arrow stubs apply the actual wire limit before returning instructions, across direct/fetched results and both row modes. The default-Arrow zero-limit PUT/GET/REMOVE e2e now verifies content and deletion. All 1,926 units, local checks, 28 live cases and four latest-head tripwires pass their assertions. Prior replay 44398849306 passed both backends. Current-head test-merge replay is queued at https://github.com/databricks/databricks-driver-test/actions/runs/44405072122. The PR description is corrected, and this PR remains ready for the user's review. This comment was generated with GitHub MCP. |
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 1 Nit
Looks good, no blocking issues. The cap is enforced consistently in ThriftOperationBackend, ArrowResultConverter and CloudFetchResultHandler, and the staging exemption works because DBSQLSession loads metadata before the result handler is built.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
Other findings
Validate an already-received direct-results status before the zero-cap shortcuts can skip it. Cover both fetchAll and hasMoreRows, without another fetch, and add the PR reference to the row-limit changelog entries. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
390bf96 adds (#623) to all three row-limit CHANGELOG entries, addressing the review-body nit. It also fixes the prefetched-status regression with two failing-first tests; both inline findings now have evidence-based replies. The no-result-set polling behavior matches main and is left for a separate completion-semantics fix. All 1,928 units, local checks, 28 live cases and the four latest-head tripwires pass their assertions. PR readiness is unchanged. This comment was generated with GitHub MCP. |
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Medium
Looks good, with 1 low concern. I traced the cap through every Thrift path: columnar, Arrow, CloudFetch, ResultSlicer and the staging metadata flow. The total is enforced consistently, and staging instructions are exempted from the cap before any fetch.
Validate queued response statuses through the existing row-set provider only when an exhausted operation is read. Preserve the operation handle for callers to close after a failed fetch, with lifecycle regressions for fetchAll and hasMoreRows. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Low
Looks good, no blocking concerns. I traced validation (before any RPC), the staging metadata-first ordering in DBSQLSession, the zero-cap prefetched-status check and the cap handling across the Columnar, Arrow and CloudFetch paths. Nit: the getConfig stub in the e2e test is never restored.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
|
Re the review-body Round 4 converged at 94420a8 with no open high finding. The PR now documents the agreed zero-cap direct-results status caveat and cites replay 44407390595 as green on both backends at the final head. The PR stays ready for the user's review. This comment was generated with GitHub MCP. |
Thrift accepted
executeStatement(sql, { rowLimit })but dropped the server cap. This PR forwards it and enforces a total across direct results and later fetches, including legacy columnar overflow. Arrow prefetch and CloudFetch scheduling stop at the cap. On Thrift, the documentedNumber/NaN-to-zero/truncation coercion precedes range validation (0..2147483646); invalid limits raiseParameterErrorbefore Int64 conversion or an RPC. Kernel behavior and server-side validation are unchanged.Round 3 fixes the default-Arrow staging regression: with
stagingAllowedLocalPath, an explicitly set, normalized limit is sent asMath.max(limit, 1)so the server preserves the instruction row. Unset stays omitted. Ordinary results still obey the requested client cap, and internal staging instructions bypass it. The earlier client exemption alone failed because the server had already removed the instruction at zero. CHANGELOG and both option references now describe this exactly. Already-returned direct-fetch failures at zero are reported when reading results, while preserving an operation the caller can close. At cap 0, the driver doesn't read results, so aFINISHEDoperation with a failed direct-results status reports differently than on main; no real server trigger is known.Lint, Prettier, type-check, build and 1,928 units pass (only inherited
CODEX_CIunset; its unrelated failure also reproduces on main). Wire-honoring Arrow staging tests cover direct/fetched results and both row modes; 28 live e2e cases pass without skips across Thrift columnar/Arrow, kernel and staging, including default-Arrow zero-limit PUT/GET/REMOVE with content verification. This staging case failed before the follow-up. Columnar caps 3/0 return exactly 3/0 rows; main returned 20/20, and the forwarding-only head returned 4/1. At latest coverage headd9ca1d981eca10159752b8fede6fb3950fa4e3fb(databricks/databricks-driver-test#2161), all four Thrift RESULTFETCH-020..023 tripwires XPASS after wire, ordered-row, delivery/download and exhaustion assertions: cap/rows7/7,100003/100003,11/11,10/7.Fresh-process execute+fetchAll medians measured at
17153bb, baseline mainbd8cb804→ branch, six samples per cell with alternating build order on the shared warehouse: Thrift 100k inline 481.5 → 516 ms, 200k CloudFetch 2423 → 2573.5 ms; kernel 450.5 → 436 ms, 1747 → 1820.5 ms. Counts and delivery are verified. These runs measure +7.2%/+6.2% on Thrift; warehouse variance limits attribution. All samples and both three-run batches are retained. The wire floor and final zero-budget status validation do not run on these uncapped queries; that execution path is unchanged from the measurement.Final-head replay 44407390595 passed Thrift and SEA at
94420a8via test-merge3412bd41(parents mainfc085d49and94420a8). It ran driver-testmainwithcommit_shaand replay mode, without a PR number; one unrelated integration run was active before dispatch. Final-head Node 22/24/26 unit/e2e, lint, security, coverage and DCO checks passed.Cross-driver state
setMaxRows(default 0/unlimited), plus local total enforcement:DatabricksThriftServiceClient.java:313,DatabricksSdkClient.java:961,DatabricksResultSet.java:392@d5bdaa3b.cursor(row_limit=None); Thrift forwards it (thrift_backend.py:1162); classic SEA sends only positive values (sea/models/requests.py:48), so zero differs @b8259e4b.WithMaxRowsis page sizing, not a statement cap (connector.go:429,internal/backend/backend.go:120@d9008a9e).StatementSpec.row_limit: Option<i64>; inline reader trims look-ahead overflow (spec.rs:217,reader/inline/provider.rs:87,napi/src/connection.rs:217@dac32e2b).Adversarial review: converged in round 4 at
94420a8with no open high finding; the remaining zero-cap error-status caveat is documented, and the final-head bot review is approved.This pull request and its description were written by Isaac.