Skip to content

Complete configured Volume transfers through the native backend - #613

Open
aakash-saravanan-db wants to merge 37 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 37 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Configured staging operations need to finish their Volume transfer through the native backend before executeStatement resolves.

What this PR addresses

Delegate marked file PUT, GET and REMOVE to the shared native core, preserve raw results when staging is omitted, and map staging and local-file errors through the Node API. Native completion declarations and enum exports are generated from the binding source.

Dependencies

Requires the Volume APIs in databricks/databricks-sql-kernel#465 and a compatible coordinated native release. Platform package pins and locks, plus clean installed-driver validation, must be updated before merge.

Tests and validation

The focused suites pass: 306 tests. TypeScript build, production ESLint and formatting pass. Native declarations use NAPI-RS CLI 3.8.2.

@github-actions

Copy link
Copy Markdown

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/volume-restart-20261113 branch from ad49367 to 7b276d7 Compare November 14, 2026 00:40
@github-actions github-actions Bot removed the kernel-e2e Preview-run the SEA Kernel E2E suite on this PR label Nov 14, 2026
@aakash-saravanan-db aakash-saravanan-db added the kernel-e2e Preview-run the SEA Kernel E2E suite on this PR label Nov 14, 2026
@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/volume-restart-20261113 branch from 7b276d7 to 12876fc Compare November 14, 2026 00:52
@aakash-saravanan-db aakash-saravanan-db added kernel-e2e Preview-run the SEA Kernel E2E suite on this PR and removed kernel-e2e Preview-run the SEA Kernel E2E suite on this PR labels Nov 14, 2026
@aakash-saravanan-db
aakash-saravanan-db marked this pull request as ready for review November 14, 2026 02:18

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly sound, with one medium issue: on the kernel path, the new StagingError wrapping can also catch the statement's own execution errors, not just transfer errors. Nits: there is a stray blank line in the CHANGELOG entry, and the constructor comment in KernelOperationBackend still says cancel() "always routes to the cancellable execution". Follow-up (not caused by this PR): executeStatement waits for the transfer before it returns the operation, so a caller can only interrupt an in-flight transfer by closing the session.

Comment thread lib/kernel/KernelOperationBackend.ts Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@github-actions

Copy link
Copy Markdown

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium · 2 Low

Mostly sound: one medium and one low concern. The dispatch, error classification and cancel routing look correct and have tests. The gap is that the published native binding has no Volume completion method, and nothing in the code or the changelog accounts for that.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • 🟡 Medium — The new completion tests stub the native method, while the existing staging integration tests use only the default Thrift backend, leaving actual kernel transfers and symlink enforcement untested in the suite. Add kernel end-to-end coverage for PUT, GET, REMOVE and allowed versus escaping symlinks. (Found by GPT-6.1 Terra · the judge couldn't confirm it)

Comment thread CHANGELOG.md
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry claims kernel Volume support, but the pinned native packages (1.4.0) don't include it. Note the required native package bump and the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Unreleased entry describes the implemented source behavior. The actual release still requires a compatible native package set; the PR description records the kernel source selection and the package/pin/lock/install merge gate. Keep this release-note/package dependency open until the real versions exist. No currently published 1.4.0 compatibility is claimed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at c89456f: CHANGELOG.md:9 still doesn't say a native package bump is required or reference the kernel change.

failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The optional native packages are pinned to 1.4.0, which lacks completeVolumeOperation. A kernel statement with stagingAllowedLocalPath then throws a raw TypeError, even for ordinary SQL. Guard for the missing method with a clear error or ignore fallback, and bump the native packages.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid required merge gate: all eight platform packages and lock entries must select a native release with the completion APIs. Clean installed ordinary SELECT, metadata and Volume checks are required before merge. The PR description explicitly records this prerequisite; source-built CI is not installed-package proof.

The test-coverage observation is also bounded correctly: the repository staging suite currently selects Thrift. Retained source-built native/driver comparisons verify transfer bytes and removal; the shared core verifies symlink authorization. Named non-skipped scheduled execution with compatible released inputs remains open. Additional duplicate connector coverage is optional test placement, and is not claimed present.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at c89456f: package.json:95-102 pins 1.4.0, and KernelOperationBackend.ts:322 has no missing-method guard.

Comment thread lib/kernel/KernelErrorMapping.ts Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 2 Low

Needs changes: 1 high concern, 1 low. The staging delegation, error classification and cancel routing look correct. However, the code calls a native API that the pinned 1.4.0 binding does not ship.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Comment thread CHANGELOG.md
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry doesn't say which native package version is required or link the kernel change, even though published 1.4.0 lacks the API. Add the required follow-up package bump and a kernel reference.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retained with the native release gate. A compatible published version has not been selected or released, so the changelog cannot accurately name one yet. Before merge, the native release, all eight platform pins/locks and installed checks must be completed together, and the final changelog must describe the actual required version. No invented version or older-native fallback is being used.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 52c78ca: CHANGELOG.md:9 doesn't name the required native package bump or link the kernel change; it should at least note that the follow-up bump is pending.

Comment thread lib/kernel/KernelErrorMapping.ts Resolved
failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — package.json pins the kernel platform packages at 1.4.0, which has no completeVolumeOperation. Any kernel statement run with stagingAllowedLocalPath would fail with a TypeError instead of the old no-op warning. Bump the pins together with this change, or feature-detect the method and raise a clear error.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed; this remains an open merge gate. The compatible native release must be published and all eight platform pins/locks updated before merging this consumer, followed by clean installed SELECT/metadata/Volume checks. The current source-built native proof does not make released 1.4.0 compatible. Coordinated release is the selected contract, so no silent older-native fallback is added.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 52c78ca: package.json:95-102 pins all eight kernel platform packages at 1.4.0, which has no completeVolumeOperation. Needs the published release.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 2 High · 1 Low

Mostly solid, with 1 medium and 1 low concern. The new staging path calls native APIs that the published binding doesn't have, with no capability check. The new cancel routing doesn't cover the execution arms executeStatement actually uses.

This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • 🟠 High — For a query that outlives the inline wait, this parks awaitResult() before exposing the operation, while concurrent session close queues behind that same await and cannot cancel the query. Cancel the pending async statement before closing it and test session teardown during a configured long-running query. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
🔀 1 finding considered and dismissed by the judge
  • 🟡 Medium lib/kernel/KernelErrorMapping.ts:330 — Safe integers outside the signed 32-bit range pass validation but make getSystemErrorName throw a RangeError, masking the original file error and path. Restri
    Judge's evidence: typeof envelope.localOsError === 'number' && Number.isSafeInteger(envelope.localOsError) ? envelope.localOsError : undef…

failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeVolumeOperation is called with no feature-detection, and package.json still pins the native kernel packages at 1.4.0. A binding without the method fails every statement run with stagingAllowedLocalPath with a raw TypeError, where it used to be ignored with a warning. Check typeof handle.completeVolumeOperation === 'function' and throw a clear error naming the required native version, and bump the platform pins together with this change.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the compatible-package prerequisite. Published platform packages remain 1.4.0; all supported platform pins and locks must move to an actual compatible release before merge/release, with clean installed SELECT/metadata/Volume validation. The source selector alone does not close that gate. The coordinated release contract does not support older native packages, so feature detection is not a substitute for the required package bump. This finding stays open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 2870cfe: package.json:95-102 still pin every native platform package at 1.4.0, and KernelOperationBackend.ts:322 calls completeVolumeOperation without a guard.

Comment thread lib/kernel/KernelOperationBackend.ts Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The pending-result session-close finding is valid. Controlled execution through the actual Node session/backend and exported native API kept configured close pending while SQL remained RUNNING; the no-staging control closed promptly. Fixed in c7a9e46: use detached native cancellation once before queuing native close. A rejected preparatory cancel falls back to waiting for SQL and closing, without setting the public cancelled state.

The existing session lifecycle group now covers successful and failed cancel responses, single statement/session disposal and repeated close. All 144 affected tests, TypeScript build, lint and formatting pass. The native failure control also completes disposal after independent SQL success, with exactly one statement-delete and one session-delete. Compatible packages and clean installed validation remain separate open gates.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 4 Low

Needs changes: 1 medium and 2 low concerns. The Volume completion path calls a native method that the published binding lacks, so with today's 1.4.0 package, kernel users who pass stagingAllowedLocalPath get errors where they used to get a warning.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Comment thread lib/kernel/KernelErrorMapping.ts Resolved
failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The pinned 1.4.0 native packages appear to predate completeVolumeOperation, so every kernel statement with stagingAllowedLocalPath would fail with a TypeError. Check typeof handle.completeVolumeOperation === 'function' and throw a clear unsupported-binding error, or fall back to the old ignore path. Land the package pin bump with this change.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing required package gate. The latest native release remains v1.4.0, which lacks the completion API. Before merge/release, all eight platform pins/locks must select an actual compatible release and clean installed SELECT, metadata and Volume checks must pass. KERNEL_REV validates source integration only. Coordinated release is the selected contract, so feature detection or silently retaining the old ignore behavior would not satisfy it. This finding remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: pins remain 1.4.0, which lacks completeVolumeOperation (KernelOperationBackend.ts:324). Merge is blocked until the release is coordinated.

Comment thread lib/kernel/KernelOperationBackend.ts Resolved
Comment thread lib/kernel/KernelOperationBackend.ts Resolved
Comment thread CHANGELOG.md
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog presents kernel Volume transfers as shipped, but they depend on a native build newer than the pinned 1.4.0. State the minimum native package version required and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased, so it describes the candidate rather than an already shipped version. The required compatible native version remains part of the open release/package gate: v1.4.0 is still the latest actual release, and no minimum compatible version can yet be stated accurately. The PR Dependencies section identifies databricks/databricks-sql-kernel#465 and the exact source pin, plus the required platform pin/lock and clean-install work. Keep this gate open until the actual release and dependency update, when the minimum version can be recorded without inventing one.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: CHANGELOG.md:9 should say a later native package release is required and cite databricks-sql-kernel#465. No version number is needed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The release entry will document the actual supported native version when the coordinated dependency update lands. It currently sits under Unreleased, while the PR Dependencies section already identifies databricks-sql-kernel#465, its exact source pin and the compatible-release/pin/lock/clean-install prerequisites. A source-PR dependency sentence in the eventual user release history would age poorly once those prerequisites are delivered; the actual package requirement belongs with that release update. The suggested wording is optional pre-release clarity, while the real compatible-package gate remains required and open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: CHANGELOG.md:9 still doesn't say a later native package release is needed or cite databricks-sql-kernel#465. Needs human triage.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium

Needs changes: 1 high and 1 medium concern. The staging dispatch, cancel routing and close serialization look correct and are well tested, but the native call has no capability guard against the 1.4.0 binding this package currently pins.

This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • 🟡 Medium — The new tests stub native completion, while existing staging integration tests use the Thrift backend, leaving kernel transfers and the symlink boundary untested in the repository. Add kernel-backed integration coverage for PUT, GET, REMOVE and allowed versus escaped symlinks. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
🔀 2 findings considered and dismissed by the judge
  • 🟡 Medium lib/kernel/KernelSessionBackend.ts:206 — Setting an allowlist forces even ordinary runAsync: true queries to wait for SQL completion, preventing callers from obtaining a pending handle to cancel long
    Judge's evidence: const runAsync = (options.runAsync ?? false) && options.stagingAllowedLocalPath === undefined;
  • 🟡 Medium lib/kernel/KernelErrorMapping.ts:330 — Safe integers outside the signed 32-bit errno range still reach getSystemErrorName, which throws instead of preserving the file error and path. Restrict the l
    Judge's evidence: typeof envelope.localOsError === 'number' && Number.isSafeInteger(envelope.localOsError)

failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The method is called unconditionally for every kernel statement with stagingAllowedLocalPath, but the pinned 1.4.0 native packages lack it, so even a plain SELECT fails with a TypeError. Guard with a typeof check and throw a clear error, or bump the platform package pins in this PR.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the same unresolved compatible-package gate. Native v1.4.0 is still the latest actual release and lacks completion APIs. Before merge/release this consumer needs a real compatible native release, all eight platform pin/lock updates and clean installed SELECT/metadata/Volume proof. KERNEL_REV and source-built CI validate source integration only. The coordinated release contract permits the new API; an older-binding fallback or invented minimum version would not satisfy the gate. This finding remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: completeVolumeOperation (KernelOperationBackend.ts:324) needs a compatible native release and all eight platform pins/locks bumped.

@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The integration-coverage observation concerns placement; another connector storage matrix is optional. The shared owner in databricks/databricks-sql-kernel#465 has the actual allowed/escaped-symlink test (symlinks_require_lexical_authority_and_resolved_target_in_any_allowed_root), transfer/retry/permission tests and exported-handle regressions in tests/v0_volume_wiremock.rs. Node tests isolate delegation, typed errors and lifecycle ownership. The retained source-built Node/Thrift sync/async comparisons exercise initial/changed PUT/GET bytes and storage-origin removal: two matching workflows, zero differences. databricks/databricks-driver-test#2122 keeps these verbs in the existing Node comparator configuration.

The new close-order correction also has compiled Node plus actual-native delayed-cancel/held-disposal controls; it does not rely solely on stubs. These receipts retain their original native/source identities. A separate clean installed-package run and named non-skipped scheduled execution remain required and open, pending compatible released inputs. No second Node matrix or broader test framework is needed for an additional adapter defect established by this review.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 2 Low

Needs changes: 1 medium and 1 low concern. The close/cancel serialization and error mapping look correct and are well tested. The main risk is that the new native API is called without checking that the loaded binding provides it.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Comment thread CHANGELOG.md
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry announces kernel Volume support, but the pinned native packages are still 1.4.0, which lacks the feature. Say that a published native package bump is required and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased. The Dependencies section identifies the exact kernel source and explicitly blocks merge/release on the compatible native version, platform pins/locks and clean installed-package validation. The eventual release entry will name the actual supported version when that package update lands. A source-PR dependency sentence in lasting release history would age poorly after delivery; no minimum version is invented while v1.4.0 remains the latest release. Optional pre-release wording is acknowledged; the actual package/documentation gate remains open.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased. The PR Dependencies section names the exact kernel source and explicitly blocks merge/release on the real compatible native version, all eight platform pins/locks and clean installed-package checks. The release entry must document the actual supported version when that coordinated update lands; native 1.4.0 remains incompatible. A source-PR dependency in the eventual release history would become stale. This is optional pre-release wording, while the actual version/pin/documentation gate remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at f877311: CHANGELOG.md:9 doesn't mention the required native package bump or the kernel change. The PR description's merge gate doesn't put that in the changelog.

failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The package.json pins still resolve native 1.4.0, which has no completeVolumeOperation, so every kernel statement with stagingAllowedLocalPath fails with a TypeError, including plain SELECTs. Guard with typeof handle.completeVolumeOperation === 'function' and give a clear error or fall back. Also ship the native package bump and pins together.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the same unresolved compatible-package prerequisite. Native v1.4.0 remains the latest actual release and lacks completion APIs. This consumer must receive a real compatible native release, all eight platform pin/lock updates and clean installed SELECT/metadata/Volume validation before merge/release. KERNEL_REV and source-built checks prove source integration only. The agreed resolution is the coordinated package update; a method check cannot make 1.4.0 support the required behavior, and silently skipping transfer is not acceptable. This gate remains open.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the same open compatible-package merge gate. Native 1.4.0 lacks completeVolumeOperation. Before merge/release all eight platform pins and locks must select a real compatible release and clean installed SELECT/metadata/Volume checks must pass. The source selector and local source-built validation do not establish package compatibility. A capability guard cannot supply the missing behavior, and an older-native fallback is outside the coordinated-release contract. No invented version or silent Volume fallback is being used.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at f877311: native pins still resolve 1.4.0, which lacks completeVolumeOperation (KernelOperationBackend.ts:324). This blocks merge until a compatible native release is pinned.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at f877311: the pins still resolve native 1.4.0, which has no completeVolumeOperation (KernelOperationBackend.ts:324). This gate stays open until a compatible release is pinned.

Comment thread lib/kernel/KernelErrorMapping.ts Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium · 2 Low

Needs changes: 1 high and 1 medium concern. The native-completion wiring is consistent, but it depends on a native API that the pinned published binding (1.4.0) doesn't have, and nothing checks for it.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • 🟡 Medium — The new completion tests replace the native method with a stub, while the existing staging integration tests select Thrift, leaving the native transfer and symlink allowlist untested in the repository. Add kernel-backed integration coverage for PUT/GET/REMOVE and permitted versus escaped symlink targets. (Found by GPT-6.1 Terra · the judge couldn't confirm it)

failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeStagingOperation calls handle.completeVolumeOperation with no capability check, while package.json still pins the native packages at 1.4.0. If those builds lack the method, every kernel statement with stagingAllowedLocalPath fails with a raw TypeError. Check that the method is a function and throw a clear error, or bump the native pins in the same change.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing merge prerequisite. Published native 1.4.0 lacks the required API. Before merging, this connector requires an actual compatible native release, all eight platform pins/locks and clean installed SELECT, metadata and Volume checks. KERNEL_REV and the source-built native prove source integration only. Coordinated release is the selected contract; a capability check cannot supply the missing transfer implementation. This gate remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 38edc92: native pins remain at 1.4.0, which lacks completeVolumeOperation, so the coordinated native release gate is still unmet.

Comment thread lib/kernel/KernelErrorMapping.ts Resolved
Comment thread CHANGELOG.md
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog announces kernel Volume transfers as supported, but the code needs a native build newer than the pinned 1.4.0 packages. Note the required native package bump in the entry and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased. The Dependencies section already blocks merge/release on a compatible native release, all platform pins/locks and clean installed checks. The final changelog must name the actual supported version when that coordinated update lands. No minimum version is invented while published 1.4.0 is incompatible; this stays with the open package/documentation gate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 38edc92: no Dependencies note in the CHANGELOG.md Unreleased section, and line 9 still names no required native bump or kernel change.

@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The overview integration observation is accurate about test placement, not an additional demonstrated adapter defect. Shared kernel tests own file bytes, allowed/escaped symlinks and in-flight cancellation; Node tests protect delegation, error mapping and lifecycle finalization. Retained source-built Node/Thrift sync/async lifecycle comparisons exercise initial/changed bytes and storage-origin removal (two matches, zero differences), and compiled Node with an actual native build verifies one completion event/metric and single statement/session disposal under concurrent cleanup. Those receipts keep their original source/native identities. The companion comparator retains the positive workflow; compatible packages, clean installed and named scheduled execution remain open. No duplicate connector storage matrix is added.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 2 Low · 1 Nit

Mostly sound: 1 medium and 1 low concern. Staging delegation, error mapping and the forced inline path match their tests. The native APIs are still not shipped in the pinned 1.4.0 binaries.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • ⚪ Nit — The changelog entry doesn't say that Volume transfers need a newer published native package, and it doesn't reference the kernel change. Add both to the entry. (Found by Claude Opus 5.5 · confirmed by the judge)

Comment thread lib/DBSQLSession.ts
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
if (opBackend.completeStagingOperation) {
await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — If native completion throws, the already-registered operation is never returned or closed, so its server handle stays alive until session close. Best-effort close the operation on this failure path and rethrow the original error.

🔀 Found by GPT-6.1 Terra · confirmed by the judge


for (const outcome of ['completed', 'not_volume']) {
it(`${
outcome === 'completed' ? 'consumes completed instructions' : 'preserves ordinary result rows'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The new tests stub the native outcome, and the kernel e2e suite (tests/e2e/kernel/**) has no Volume PUT, GET or REMOVE test. Add kernel e2e coverage for transfers, raw instructions and failure paths.

🔀 Found by GPT-6.1 Terra · confirmed by the judge

failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeVolumeOperation is called without checking it exists, and the platform packages are still pinned to 1.4.0. With those binaries, any kernel statement that sets stagingAllowedLocalPath fails with a TypeError. Check the method exists and throw a clear HiveDriverError, or land the pin bump in this PR.

🔀 Found by both reviewers · confirmed by the judge

This branch was successfully deployed

1 active deployment
azure-prod — ca22ba19 Deployed Nov 19, 2026 by aakash-saravanan-db via e2e-test (24.22.1, 24_22_1) #2132
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant