Skip to content

Complete Volume file operations through the kernel backend - #1057

Open
aakash-saravanan-db wants to merge 36 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 36 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Kernel-backed cursors need to complete server-marked Volume transfers through the Python staging API.

What this PR addresses

Complete file PUT, GET, REMOVE and BinaryIO PUT through the native core. Keep the mutable statement owner and detached canceller available during synchronous transfer, consume instruction rows on completion, and preserve Python filesystem and HTTP exception mapping. Async execution completes through its retained result stream.

Dependencies

Requires the kernel Volume APIs in databricks/databricks-sql-kernel#465 and a compatible coordinated native release. The package floor and lock must select that release, with clean installed-driver validation before merge.

Tests and validation

The focused client and result-set suites pass: 292 tests. Local validation uses source-built native inputs; package installation and scheduled activation remain separate gates.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Medium · 1 Low

Needs changes. The main blocker is that the code relies on new kernel APIs while the dependency floor still allows older wheels, which would break every kernel query, not only Volume operations. There is also a KeyError crash when the best-effort canceller is missing.

Other findings

  • 🔵 Low — This PR deletes the kernel e2e tests for staging and adds no replacements. All new coverage uses mocks that invent is_volume_operation, complete_volume_operation return strings ("needs_input_stream", "completed", "not_volume") and exception attributes (local_os_error, http_status, local_path). Nothing in CI checks that the real kernel binding actually has these names and values. Please add kernel e2e cases, such as PUT→GET→REMOVE round trips (file and input_stream) and a rejected symlink outside the allowed root, so a mismatch with the kernel contract fails the build.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — pyproject.toml:73 still allows databricks-sql-kernel ^1.4.0, but the code now calls three kernel APIs that don't exist yet: ExecutedStatement.is_volume_operation (read here on every sync execute), kernel_handle.is_volume_operation (read in KernelResultSet.__init__ for every result, including metadata ResultStreams), and fetch_result() in place of await_result() for async results.

If the resolver installs any wheel in the allowed range that lacks them, this isn't limited to Volume ops. Every execute(), every metadata call and every get_async_execution_result() raises AttributeError, so ordinary SELECT 1 breaks on use_kernel=True. The PR description says the floor "has not been selected". This should block merge: bump the floor in the same change. Until then, reading is_volume_operation with getattr(..., False) would at least keep non-volume queries working.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-canceller defect is corrected at ab1efce1: registration uses .get, the Statement remains retained, and cleanup skips absent cancellers. All 276 affected tests pass; final native live checks preserve lazy BinaryIO and one-read empty/nonempty streams.

The new Volume marker/completion API dependency is a valid merge prerequisite. The source build pins compatible 22f96fac; a compatible published native version/floor and installed-package proof remain explicitly open. This PR is not merge-ready, and no old-native fallback or invented release version was added.

The fetch_result subclaim is separate: released 1.4.0 exposes it, but the substitution is removed from this PR. Merged native #451 provides terminal-response reuse with the original await_result boundary. Actual public-cursor owner and reader/refetch tests show identical await/fetch requests, rows and metadata on the selected native; redundant #1056 is superseded without merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming the Volume-marker dependency is a merge prerequisite, and for dropping the fetch_result() substitution. That part of the finding is resolved. The core issue is still there, though. pyproject.toml:73 still says databricks-sql-kernel ^1.4.0. Line 641 still reads executed.is_volume_operation with no guard. KernelResultSet.__init__ still reads kernel_handle.is_volume_operation the same way. So if a published wheel in that range lacks the marker, ordinary sync executes and metadata calls fail with AttributeError before the code ever checks for a Volume op. A source build pinned to 22f96fac doesn't limit what the resolver installs. Leaving this open until the floor is raised to the first published wheel that ships the marker, or until both reads are guarded so non-volume queries keep working.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Medium · 2 Nit

Needs changes: 1 high, 1 medium and 2 low findings. The high one: the new is_volume_operation attribute is read unconditionally on every execute, but the kernel floor is still ^1.4.0, so released wheels would break all kernel queries. The medium one: any kernel error with an HTTP status is turned into a generic staging OperationalError. Follow-up already noted in the PR: Windows and installed released-package validation are still open.

Other findings

  • ⚪ Nit — The new ordering (backend close_command first, then _kernel_handle.close()) now applies to every kernel result set, not only Volume ones. For async results, close_command closes the retained owning handle and the parent Statement (client.py ~800-820) while the result stream taken from them is still open. Before this change the stream was closed first. Any error from closing the stream afterwards is logged and swallowed, but this changes the teardown order for all async results to work around a lock that only Volume transfers hold. Consider cancelling only the Volume canceller first and keeping the old order for everything else.
  • ⚪ Nit — Several new cancellation paths have no unit tests: close_session now cancels all in-flight sync cancellers and retained Volume cancellers, and close_command cancels a pending Volume canceller (client.py:784-790). The only Volume backend test covers cancel_command and finish_volume_operation. test_volume_close_signals_backend_before_native_lock uses a mocked backend, so it doesn't exercise the real close_command logic. Add tests that (a) close_command on a registered Volume id calls cancel() and removes the entry, and (b) close_session cancels and clears both maps, even when cancel() raises.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — executed.is_volume_operation is read on every sync execute, and kernel_handle.is_volume_operation is read in KernelResultSet.__init__ (result_set.py:88) for every result set, metadata streams included. Both attributes only exist in the unreleased kernel change (#465). pyproject.toml still pins databricks-sql-kernel = ^1.4.0. A released 1.4.x wheel will raise AttributeError on every use_kernel=True query, so this breaks plain SELECTs, not just Volume ops. Before merge, either bump the floor in the same change to the first release that has the attribute, or read it with getattr(..., "is_volume_operation", False) so older wheels keep working with staging unsupported.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed; this is an open merge prerequisite. A compatible native release must ship before the Python dependency floor/lock is updated and clean installed SELECT, metadata and Volume checks pass. Released 1.4.0 is incompatible; source-built validation does not close the gate. Coordinated release is the selected contract, so no older-native fallback is added. Leaving this thread open until the actual package update is verified.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is still a merge prerequisite. The current code still reads executed.is_volume_operation without a fallback, so it stays unsafe until the dependency floor (and lock) moves to a kernel release that actually ships the attribute. Coordinating the release, with no fallback for older native wheels, works as long as the floor bump lands in the same change. Then pip can't resolve 1.4.0 next to this code, and an old wheel fails at install time instead of on every SELECT. I'm leaving this thread open until the floor bump is pushed and the installed-wheel checks for SELECT, metadata and Volume pass.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

One high-severity issue: the code depends on kernel APIs that the allowed databricks-sql-kernel ^1.4.0 wheels don't provide, so every kernel query would break on install. The rest of the volume completion, cancellation and cleanup logic looks consistent and is well covered by unit tests.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation for every result set: SELECTs, metadata streams and async results. execute_command also reads executed.is_volume_operation (client.py:653) on every sync execute. Neither read is guarded. The PR description says published native 1.4.0 lacks these APIs, yet pyproject.toml:73 still allows databricks-sql-kernel ^1.4.0.

With any wheel the resolver can install today, every use_kernel=True query will raise AttributeError. That breaks all kernel queries, not just Volume operations. Only KERNEL_REV (used by source-build CI) was bumped, so CI built against kernel#465 won't catch this.

Either bump the dependency floor to the first release that ships is_volume_operation/complete_volume_operation in this PR, or block merging until that release exists. If the release ordering can't be made atomic, a getattr(handle, "is_volume_operation", False) fallback would keep ordinary queries working on older wheels.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid merge prerequisite. The connector must release with a native version that provides the Volume marker/completion APIs. Merge remains blocked on that actual release, an updated dependency floor and lock, and clean installed-package SELECT, metadata, async and Volume validation. The PR description records this gate. KERNEL_REV and source-built CI establish source integration only; they do not establish compatibility with published 1.4.0.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming this is a merge gate. The code at this line still reads kernel_handle.is_volume_operation with no guard, and nothing in this thread shows that pyproject.toml has moved off ^1.4.0. So I'm leaving this thread open as the blocker. It can be resolved once the dependency floor and lock point to the first published native release that ships is_volume_operation and complete_volume_operation. The other way to resolve it is a getattr(..., "is_volume_operation", False) fallback here and at the read in client.py that keeps installs on older wheels working. Either change should come with your planned installed-package validation of SELECT, metadata, async and Volume paths.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

The completion and cancellation wiring looks sound, with one high merge-gate concern: the PR calls new kernel attributes on every query, but the dependency floor still allows 1.4.0, which lacks them. Kernel e2e coverage for the new Volume path is also missing.

Other findings

  • 🔵 Low — This PR deletes the kernel e2e staging tests but adds no e2e test for the new behavior: PUT/GET/REMOVE round trip, the input_stream upload, and rejection of paths or symlink targets outside the allowed roots. The unit tests only exercise a mocked complete_volume_operation, so nothing in-tree shows that the real native API returns the needs_input_stream / completed / not_volume strings, or the local_os_error / VOLUME_STORAGE_HTTP_ERROR attributes, that the Python side keys on. Consider adding an e2e PUT→GET→REMOVE test (gated on the compatible kernel) to replace the deleted ones.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The new code reads executed.is_volume_operation with no guard on every sync execute. KernelResultSet.__init__ does the same with kernel_handle.is_volume_operation (result_set.py:87), and that runs for every result, metadata calls included. pyproject.toml:73 still allows databricks-sql-kernel = ^1.4.0, and the PR description says published 1.4.0 lacks these APIs. On an in-range 1.4.x wheel, a plain SELECT 1 therefore raises AttributeError (wrapped or not), so this breaks the kernel path for all queries, not just Volume operations.

Either bump the floor in this PR to the kernel release that ships is_volume_operation / complete_volume_operation, or read the attribute with getattr(..., 'is_volume_operation', False) so non-volume statements keep working on older wheels. The description calls the floor bump a merge gate, but nothing in the dependency graph enforces it today.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and retained as a merge prerequisite. Released native 1.4.0 lacks the marker/completion API used by ordinary results and metadata as well as Volume. The coordinated compatible native release must precede a real dependency floor/lock update and clean installed SELECT, metadata and Volume checks. The local source selector/build does not close this gate, and no older-native fallback is planned.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. With no older-native fallback planned, the dependency floor is the only thing that will enforce this gate. Until pyproject.toml (and the lock) require the kernel release that ships is_volume_operation / complete_volume_operation, the resolver can still install 1.4.0, and then every kernel-path query fails. That includes plain SELECTs and metadata calls, not just Volume operations. I'm leaving this thread open so it stays in the unresolved queue until the floor bump lands in this PR, or in a PR merged together with it. The merge gate is currently stated only in the PR description, so nothing enforces it yet.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

Mostly sound, with one high concern. With the current ^1.4.0 dependency floor, the new reads of is_volume_operation would break every kernel query and metadata call, not just Volume statements. The rest of the native completion, cancel and close bookkeeping looks consistent and is covered by unit tests.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation without a guard, and execute_command reads executed.is_volume_operation right after stmt.execute() (client.py:653). Every kernel result goes through this constructor: sync execute, async get_execution_result, and metadata calls (_make_result_set(stream, ...) at client.py:1063/1083/1112). pyproject.toml still allows databricks-sql-kernel ^1.4.0, and the PR description says published 1.4.0 lacks these APIs. So on any wheel the resolver currently allows, every use_kernel=True query and metadata call, not just Volume statements, would fail with AttributeError and be wrapped as a kernel error.

The description says the floor bump is a merge prerequisite, but nothing in the diff enforces it. Either bump the dependency floor (and poetry.lock) to the first kernel release that ships is_volume_operation/complete_volume_operation in this PR, or read the attribute with getattr(handle, "is_volume_operation", False) at both sites so older wheels keep the previous non-staging behavior.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed; this remains a merge/release prerequisite. The latest published kernel release is still v1.4.0 and lacks the required marker/completion APIs. A coordinated compatible native release must precede the actual Python dependency-floor and poetry.lock update, followed by clean installed SELECT, metadata and Volume checks. The source selector is updated to the corrected kernel but does not satisfy that package gate. Older-native fallback is outside the agreed release contract; no compatible version is invented here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that this is a release prerequisite. The current code still reads kernel_handle.is_volume_operation unguarded, and pyproject.toml still allows databricks-sql-kernel ^1.4.0, which you say doesn't have the API. So nothing in the dependency graph enforces that prerequisite yet. If this merges before the floor bump lands, a resolver that installs 1.4.0 will make every use_kernel=True query and metadata call fail. I'm leaving this thread open as the merge blocker until the dependency floor and poetry.lock point to the compatible kernel release. If you'd rather not depend on merge order, a getattr(..., "is_volume_operation", False) guard would also close it.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

The Volume completion flow looks sound, with good unit coverage of lazy stream reads, error mapping and cancel-before-close. The main blocker is that the kernel now requires is_volume_operation while the dependency floor still allows 1.4.0, which lacks it. Nit: the kernel's VOLUME_STORAGE_HTTP_ERROR message omits the -{error_text} suffix that the Thrift handlers include (client.py:1433).

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — Every KernelResultSet now reads kernel_handle.is_volume_operation without a fallback. That covers sync execute, async await_result() streams and all metadata ResultStreams. The same unguarded read happens in execute_command (client.py:653, executed.is_volume_operation). The PR description says the published kernel 1.4.0 does not have this API, yet pyproject.toml still allows ^1.4.0. If the resolver picks 1.4.x, every use_kernel=True query and metadata call fails with AttributeError. That breaks all kernel users, not just Volume operations.

The description treats this as a merge gate. The floor bump is still missing from this diff, so as written it regresses the currently resolvable dependency set. Either bump the floor to the release that ships is_volume_operation/complete_volume_operation, or read the attribute defensively (getattr(handle, "is_volume_operation", False)). The defensive read would restore the old behavior on older wheels.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing merge/release prerequisite. The latest actual native release is v1.4.0 and lacks the required Volume marker/completion APIs. Before merge, this connector needs a real compatible native release, the matching dependency floor/lock update, and clean installed SELECT, metadata and Volume validation. The source pin validates source integration only. The coordinated release contract intentionally requires the new API; treating an old binding as a non-Volume result would restore silent no-transfer behavior. This gate remains open.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the compatible-release merge gate. Published native 1.4.0 lacks the required APIs. Before merge this connector needs a real compatible release, its dependency floor/lock update, and clean installed SELECT, metadata and Volume validation. The exact source selector validates source integration only. The coordinated release contract requires the new marker/completion APIs; a getattr fallback would silently restore raw Volume instructions, so it is not the selected compatibility policy. This finding remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. Your reason for not using a getattr fallback makes sense: quietly treating an old binding as a non-Volume result would bring back the silent no-transfer bug, so failing on the missing API is a reasonable policy. The code at this line still reads kernel_handle.is_volume_operation with no fallback, and ^1.4.0 still lets the resolver install a wheel that lacks it. So this finding stays open until the floor and lock are bumped to a published native release that ships is_volume_operation/complete_volume_operation. If the gate gets bypassed before then, an explicit version check at session open with a clear NotSupportedError would fail more clearly than an AttributeError on every query.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing has changed since my last reply. pyproject.toml:73 still allows databricks-sql-kernel = "^1.4.0", and line 87 still reads kernel_handle.is_volume_operation directly. You've chosen to fail loudly on old wheels rather than add a getattr fallback, and I accept that. This stays open as the merge gate until the floor and lock are bumped to a published native release that ships is_volume_operation/complete_volume_operation.

@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

On the overview wording nit: the native staging path intentionally reports the storage HTTP status without copying the response body. result_set.py preserves OperationalError and the status, with the native exception as its cause; the common kernel error retains VOLUME_STORAGE_HTTP_ERROR and http_status. Thrift client.py includes its raw error_text suffix. Exact body-text equality is not the selected error contract, and the shared native path has no response body to forward here. That wording difference is an optional diagnostic enhancement; no adapter-only body extraction or additional error-string test is warranted. The compatible-release/floor/lock/install finding remains open.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

Mostly solid: Volume completion, cancellation and close ordering are wired consistently, with good unit coverage. One high concern: the new native attribute is read unconditionally while the dependency floor still allows a kernel version that lacks it.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ reads kernel_handle.is_volume_operation with no fallback, and execute_command reads executed.is_volume_operation unconditionally (client.py:653). Neither guards the access. The PR description says published kernel 1.4.0 lacks these APIs, yet pyproject.toml:73 still allows ^1.4.0. If the resolver installs 1.4.0, every use_kernel=True query hits AttributeError when its result set is built: SELECTs, metadata calls and async results, not just Volume operations. That would break kernel connections that work today.

The description notes that the floor must be bumped before release. Until it is, the merged code is broken for every in-range install. Either bump the floor in this PR to the first kernel version that ships is_volume_operation/complete_volume_operation, or read the attribute with getattr(handle, "is_volume_operation", False) so non-volume queries keep working on older wheels.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing merge prerequisite: native 1.4.0 lacks these marker/completion APIs. A real compatible native release, matching Python floor/lock update and clean installed SELECT, metadata and Volume checks are required before merge. The source selector proves source integration only. The coordinated release contract requires the API; silently treating an old binding as an unmarked result would restore the no-transfer behavior. This gate remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that a silent getattr(..., False) fallback is the wrong fix. It would hide a mismatched binding and bring back the no-transfer behavior for Volume operations, so bumping the floor is the right fix. The current head still has the unguarded kernel_handle.is_volume_operation read, and pyproject.toml still allows ^1.4.0. So the problem is still there: every use_kernel=True result set raises AttributeError on 1.4.0, including SELECTs and metadata calls, not just Volume operations. I'm leaving this thread open as the merge gate. It can be resolved once a push raises the floor and updates the lock to the first native release that ships is_volume_operation and complete_volume_operation.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

This needs changes before merge, mainly because of one dependency-floor problem. The kernel result set now requires is_volume_operation on every handle, but pyproject.toml still allows kernel 1.4.0, which lacks that API, so every kernel query would fail on 1.4.0. Separately, the PR removes the kernel e2e staging tests and adds no e2e PUT/GET/REMOVE coverage in their place; consider adding them once a compatible wheel is available.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation with no fallback, and execute_command (client.py:653) reads executed.is_volume_operation the same way. Every result set goes through this path: SELECTs, metadata ResultStreams and async await_result streams. The PR description says published kernel 1.4.0 does not have these APIs, but pyproject.toml still allows ^1.4.0. A resolver that installs 1.4.0 would get an AttributeError (wrapped as a DatabaseError) on every use_kernel=True query, not just Volume operations. That would break kernel connections that work today.

The description treats the floor bump as a release gate. This PR is the change that introduces the hard requirement, so either bump the floor to the first compatible kernel version here or read the attribute with getattr(handle, "is_volume_operation", False). Otherwise merging before the native release breaks ordinary queries.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing merge prerequisite. Native 1.4.0 lacks these APIs, including the ordinary-result marker reads. Before merge, this PR must select a real compatible native release, update its dependency floor/lock and pass clean installed SELECT, metadata and Volume checks. The Dependencies section blocks merge/release on that coordinated update. No version is invented and no old-native fallback is added; the source selector does not satisfy this gate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that this needs to be fixed before merge. I'm keeping the thread open because nothing has changed in the code yet: line 87 still reads kernel_handle.is_volume_operation with no fallback, and pyproject.toml still allows 1.4.0. The thread can stand as the visible merge blocker. It can be resolved once this PR bumps databricks-sql-kernel in pyproject.toml and the lock file to the first published native release that ships is_volume_operation, and the SELECT, metadata and Volume checks have passed against that installed wheel. Declining the getattr fallback is fine as long as the floor bump lands in this same change.

@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The overview E2E observation concerns test placement. The two removed cases asserted that kernel PUT always raises NotSupportedError, including a comment-prefixed SQL spelling; they were obsolete rejection expectations, not successful transfer tests. The shared marker/completion and raw-result boundaries remain tested, and the companion coverage PR retains Python synchronous/asynchronous file and empty/nonempty stream workflows that verify initial bytes, changed overwrite and storage-origin removal. Retained source-built Python/Thrift comparisons keep their exact kernel/build identity. This does not establish clean installed or scheduled execution: both remain required after a compatible wheel and floor/lock update. Restoring always-unsupported assertions would contradict the implemented feature; no duplicate connector storage matrix is added.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

The Volume completion wiring looks right. It holds the mutable owner and canceller until the transfer finishes, cancels before close, and keeps Python error mapping. One high concern: the new kernel APIs are read on every execute, but the package floor still allows kernel wheels that don't have them.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — executed.is_volume_operation is read on every sync execute. KernelResultSet.__init__ also reads kernel_handle.is_volume_operation for every result, including metadata and async streams, and result_set.py uses _kernel.VolumeCompletion. These APIs only exist in the kernel release from databricks-sql-kernel#465. The diff bumps KERNEL_REV, but pyproject.toml still allows databricks-sql-kernel ^1.4.0 and poetry.lock is unchanged.

On any installed 1.4.x wheel, every use_kernel=True execute() will raise AttributeError after the statement has already run on the server, and stmt.execute() sits inside the wrapping try. Before this PR those queries worked. That is a regression for all kernel queries, not just Volume operations. The PR description calls this a merge gate, but nothing in the dependency graph enforces it.

Fix: bump the floor and lock in this PR to the first wheel that ships these APIs. Or, if this must merge before that release, check for the attribute (e.g. getattr(executed, "is_volume_operation", False)) and fall back to the old fail-loud NotSupportedError for staging statements.

This branch was previously deployed

1 inactive deployment
azure-prod — 923b6675 Deployed Nov 19, 2026 by peco-review-bot[bot] via followup #2497
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant