Repository navigation
Complete configured Volume transfers through the native backend #613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
0a534de
47e379e
f482924
c1f0821
9d95ba0
f6de7d3
973bad3
66a4c61
81c00ae
e080abd
3ec62f5
240c70f
69cc142
8b6e7fd
8bfa89c
13b0e60
7f8736d
f2e3d68
5658cb5
a8029e0
baf8c7d
a35f7e1
d54e1af
c4b5c4c
ec92b5b
3be7f97
a7c7fd1
1f87603
0b4d434
c89456f
52c78ca
2870cfe
c7a9e46
887d7d8
f877311
38edc92
ca22ba1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,6 +6,7 @@ | |
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The changelog entry doesn't say which native package version is required or link the kernel change, even though published 1.4.0 lacks the API. Add the required follow-up package bump and a kernel reference. 🔀 Found by Claude Opus 5.5 · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Retained with the native release gate. A compatible published version has not been selected or released, so the changelog cannot accurately name one yet. Before merge, the native release, all eight platform pins/locks and installed checks must be completed together, and the final changelog must describe the actual required version. No invented version or older-native fallback is being used. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 52c78ca: CHANGELOG.md:9 doesn't name the required native package bump or link the kernel change; it should at least note that the follow-up bump is pending. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The changelog presents kernel Volume transfers as shipped, but they depend on a native build newer than the pinned 1.4.0. State the minimum native package version required and reference the kernel change. 🔀 Found by Claude Opus 5.5 · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The entry is under Unreleased, so it describes the candidate rather than an already shipped version. The required compatible native version remains part of the open release/package gate: v1.4.0 is still the latest actual release, and no minimum compatible version can yet be stated accurately. The PR Dependencies section identifies databricks/databricks-sql-kernel#465 and the exact source pin, plus the required platform pin/lock and clean-install work. Keep this gate open until the actual release and dependency update, when the minimum version can be recorded without inventing one. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 887d7d8: CHANGELOG.md:9 should say a later native package release is required and cite databricks-sql-kernel#465. No version number is needed.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The release entry will document the actual supported native version when the coordinated dependency update lands. It currently sits under Unreleased, while the PR Dependencies section already identifies databricks-sql-kernel#465, its exact source pin and the compatible-release/pin/lock/clean-install prerequisites. A source-PR dependency sentence in the eventual user release history would age poorly once those prerequisites are delivered; the actual package requirement belongs with that release update. The suggested wording is optional pre-release clarity, while the real compatible-package gate remains required and open. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 887d7d8: CHANGELOG.md:9 still doesn't say a later native package release is needed or cite databricks-sql-kernel#465. Needs human triage. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The changelog entry announces kernel Volume support, but the pinned native packages are still 1.4.0, which lacks the feature. Say that a published native package bump is required and reference the kernel change. 🔀 Found by Claude Opus 5.5 · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The entry is under Unreleased. The Dependencies section identifies the exact kernel source and explicitly blocks merge/release on the compatible native version, platform pins/locks and clean installed-package validation. The eventual release entry will name the actual supported version when that package update lands. A source-PR dependency sentence in lasting release history would age poorly after delivery; no minimum version is invented while v1.4.0 remains the latest release. Optional pre-release wording is acknowledged; the actual package/documentation gate remains open.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The entry is under Unreleased. The PR Dependencies section names the exact kernel source and explicitly blocks merge/release on the real compatible native version, all eight platform pins/locks and clean installed-package checks. The release entry must document the actual supported version when that coordinated update lands; native 1.4.0 remains incompatible. A source-PR dependency in the eventual release history would become stale. This is optional pre-release wording, while the actual version/pin/documentation gate remains open. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at f877311: CHANGELOG.md:9 doesn't mention the required native package bump or the kernel change. The PR description's merge gate doesn't put that in the changelog. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The changelog announces kernel Volume transfers as supported, but the code needs a native build newer than the pinned 1.4.0 packages. Note the required native package bump in the entry and reference the kernel change. 🔀 Found by Claude Opus 5.5 · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The entry is under Unreleased. The Dependencies section already blocks merge/release on a compatible native release, all platform pins/locks and clean installed checks. The final changelog must name the actual supported version when that coordinated update lands. No minimum version is invented while published 1.4.0 is incompatible; this stays with the open package/documentation gate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 38edc92: no Dependencies note in the CHANGELOG.md Unreleased section, and line 9 still names no required native bump or kernel change. |
||
| - Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520) | ||
| - Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485) | ||
| - Honor the client `useCloudFetch: false` default on the kernel backend so complete results can be fetched without cloud-storage downloads. (#574) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| dac32e2bf44468bdab06747dad4f289d1c0a18bd | ||
| 2fe28af460d0db6eeb8c653e52eede954c90374c |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -109,11 +109,15 @@ export default class DBSQLSession implements IDBSQLSession { | |
| // operation, everything will work as usual. In a case of staging operation, it will be processed like any | ||
| // other query - it will be possible to get data from it as usual, or use other operation methods. | ||
| if (options.stagingAllowedLocalPath !== undefined) { | ||
| const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath) | ||
| ? options.stagingAllowedLocalPath | ||
| : [options.stagingAllowedLocalPath]; | ||
| if (opBackend.completeStagingOperation) { | ||
| await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath)); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — If native completion throws, the already-registered operation is never returned or closed, so its server handle stays alive until session close. Best-effort close the operation on this failure path and rethrow the original error. 🔀 Found by GPT-6.1 Terra · confirmed by the judge |
||
| return operation; | ||
| } | ||
| const metadata = await operation.getResultMetadata(); | ||
| if (metadata.isStagingOperation) { | ||
| const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath) | ||
| ? options.stagingAllowedLocalPath | ||
| : [options.stagingAllowedLocalPath]; | ||
| return this.handleStagingOperation(operation, allowedLocalPath, metadata); | ||
| } | ||
| } | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,12 +46,14 @@ import { LogLevel } from '../contracts/IDBSQLLogger'; | |
| import { RowMode } from '../contracts/IDBSQLSession'; | ||
| import Status from '../dto/Status'; | ||
| import HiveDriverError from '../errors/HiveDriverError'; | ||
| import StagingError from '../errors/StagingError'; | ||
| import ParameterError from '../errors/ParameterError'; | ||
| import OperationStateError, { OperationStateErrorCode } from '../errors/OperationStateError'; | ||
| import ArrowResultConverter from '../result/ArrowResultConverter'; | ||
| import ResultSlicer from '../result/ResultSlicer'; | ||
| import KernelResultsProvider from './KernelResultsProvider'; | ||
| import { arrowSchemaToThriftSchema, decodeIpcSchema, patchIpcBytes } from './KernelArrowIpc'; | ||
| import { decodeNapiKernelError } from './KernelErrorMapping'; | ||
| import { decodeNapiKernelError, ErrorWithSqlState } from './KernelErrorMapping'; | ||
| import { | ||
| KernelStatement, | ||
| KernelNativeAsyncStatement, | ||
|
|
@@ -84,7 +86,10 @@ export type KernelOperationStatement = KernelStatementHandle & Partial<KernelSta | |
| * (`KernelResultsProvider` → `ArrowResultConverter` → `ResultSlicer`) consumes | ||
| * either interchangeably. | ||
| */ | ||
| type KernelFetchHandle = Pick<KernelStatement, 'fetchNextBatch' | 'schema'>; | ||
| type KernelFetchHandle = Pick< | ||
| KernelStatement, | ||
| 'fetchNextBatch' | 'schema' | 'isVolumeOperation' | 'completeVolumeOperation' | ||
| >; | ||
|
|
||
| /** | ||
| * The rich operation-status surface the kernel exposes on a terminal sync | ||
|
|
@@ -220,6 +225,8 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // already-terminal statement. Drives both fetch and result-metadata. | ||
| private fetchHandlePromise?: Promise<KernelFetchHandle>; | ||
|
|
||
| private volumeCompleted = false; | ||
|
|
||
| // Async path only: set once the kernel reported the statement Succeeded (a | ||
| // status poll, or `awaitResult()` resolving). The kernel exposes no | ||
| // synchronous terminal-state accessor, so the observed success is retained. | ||
|
|
@@ -308,6 +315,28 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // Fetch / metadata (owned by the kernel-results pipeline). | ||
| // --------------------------------------------------------------------------- | ||
|
|
||
| public async completeStagingOperation(allowedPaths: string[]): Promise<void> { | ||
| failIfNotActive(this.lifecycle); | ||
| const handle = await this.getFetchHandle(); | ||
| try { | ||
| const outcome = await handle.completeVolumeOperation(allowedPaths, true); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — Nothing checks that the loaded native binding actually has If the binding can't be pinned to a compatible release when this merges, add a feature-detect: when
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed: compatible published native packages are a merge prerequisite, including ordinary statements with stagingAllowedLocalPath. Source KERNEL_REV does not update installed platform dependencies. The matching release must supply completion, all platform pins/locks must advance, and clean installed queries plus Volume workflows must pass before merge. The accessible proxies currently expose 1.4.0, whose API is incompatible. Keeping the gate open; no old-native fallback or unavailable version is being introduced. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming. I agree that pinning a compatible published native release is the right fix, and that it has to land before merge. Until then, the code at this line still calls
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The release plan is coordinated native/connector version updates. Published 1.4.0 remains unsupported for these calls: before connector release, compatible platform pins and locks plus clean installed ordinary-query and Volume validation are required. Source validation selects kernel 855847bd with completeVolumeOperation. We are continuing review of that source pair while keeping package integration as a release prerequisite; no fallback, invented version or completed package proof is claimed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for laying out the release plan. Your reply doesn't challenge the finding. The code at line 318 still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — If the pins can't be bumped in this PR, check
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed; this remains a merge prerequisite and also affects ordinary queries when the staging option is set. Published native 1.4.0 does not supply this completion API. The coordinated release must provide the compatible API, then all platform pins/locks must select it and clean installed-package ordinary-query/Volume workflows must pass. Source-built validation does not close that gate. We are requiring the new native API rather than silently returning raw rows with an older binary or selecting an invented version. The PR description keeps this dependency unresolved; this comment is acknowledged, not claimed fixed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming this. Requiring the new native API and treating it as a merge gate makes sense, and so does not quietly falling back to raw rows. The current code still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The installed native packages are pinned to 1.4.0, which has no The merge plan relies on the pins being bumped later, but nothing in the code enforces it. A small guard would avoid the regression. For example, if
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that published 1.4.0 is incompatible. This remains an explicit merge/release gate: select the real compatible native release, update all platform pins and locks, and validate a clean installed package on ordinary queries and Volume workflows before merging. The source pin and source-built tests do not complete that gate. We require the new native APIs and a coordinated release; this change will not silently fall back to ignored staging or raw instructions. No fallback or invented version is being used to close the finding. It remains open until the compatible packages and installation proof exist. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming the 1.4.0 incompatibility. Requiring a coordinated native release instead of a silent fallback is a fair design choice. Right now, though, the gate exists only in the merge plan. At head, line 318 still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The PR description says published native 1.4.0 has no completion API. With that binding (or any stale cached
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Published native 1.4.0 is incompatible; the compatible native versions, all eight platform pins/locks and clean installed-package ordinary-query/Volume checks remain required before merge. This is a coordinated release dependency, with no older-native fallback in the chosen contract. The source pin selects the current kernel, and source-built checks do not complete the package gate. Checking an old binding at load time could improve its diagnostic, but cannot make it compatible or replace that gate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that the version pins are the real compatibility gate, and that a shape check can't make an old binding work. My concern is narrower: what happens when the pin gate is bypassed by a stale cached There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — Before this PR the option was logged and ignored, so callers that set it globally and run ordinary queries break. The facade check
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Published native 1.4.0 is incompatible. The actual compatible versions, all eight platform dependency pins/locks, and clean installed ordinary-query/metadata/Volume proof remain required before merge; a source SHA pin does not satisfy those requirements. This is a coordinated-release dependency, with no older-native fallback in the selected contract. A stale-binding method check could improve the load/runtime diagnostic, but cannot make the missing API compatible or replace the package gate. The PR remains blocked on that gate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, a method check on the handle can't make native 1.4.0 compatible, and bumping the pins is the real fix. That's still outstanding, though: line 318 calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low —
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The incompatible-package concern is valid and remains a required merge gate: compatible native platform versions, all dependency pins/locks, and clean installed SELECT/metadata/Volume proof. Native 1.4.0 lacks these APIs. The source build selects kernel30c8d8e7; both the rebuilt sync/async lifecycle comparisons and ordinary-query controls pass. Coordinated native uptake is required, so we are retaining direct calls rather than adding an older-native compatibility path. This PR is not ready to merge until that release gate passes. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming. Requiring a coordinated native release instead of adding a compatibility path is a reasonable choice. The code at this line still calls
aakash-saravanan-db marked this conversation as resolved.
Resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The PR does say merge depends on updated pins. Until that happens, though, every environment that installs from npm gets this regression. Either guard with
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed: compatible native packages, all eight platform pins/locks and clean installed SELECT, metadata and Volume proof are required before this PR can merge. The declared There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, blocking the merge on the native package bump is one of the resolutions I suggested, and choosing it over a silent no-op makes sense. Line 317 still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The PR description already lists the pin bump as a merge prerequisite, so this may be intentional. If the code can land before compatible native packages ship, check
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed as an unresolved package compatibility gate: published native 1.4.0 lacks this API. Merge/release requires the actual compatible versions across all platform packages, updated pins/locks and clean installed SELECT, metadata and staging checks. The source build uses kernel 4e90b0d0; it does not prove installed-package compatibility. The PR description records the gate and the PR remains open. The description’s source selector now matches KERNEL_REV. The retained comparisons at kernel029c59cc keep their original build identity. Completion declarations and the C/JS binding API are unchanged between the previous d9fcf834 source selector and 4e90b0d0; these kernel corrections are private. The optional needs_input_stream stub case would repeat the already-covered native stream-handshake and adapter error-pass-through contracts; the existing catch preserves an ordinary StagingError. It is not evidence of an adapter behavior defect. The changelog’s extra blank line is a cosmetic nit, left outside this focused source-pin update. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at c89456f: lib/kernel/KernelOperationBackend.ts:322 has no capability guard, and the 1.4.0 native pins remain until compatible packages ship. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — The optional native packages are pinned to 1.4.0, which lacks completeVolumeOperation. A kernel statement with stagingAllowedLocalPath then throws a raw TypeError, even for ordinary SQL. Guard for the missing method with a clear error or ignore fallback, and bump the native packages. 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Valid required merge gate: all eight platform packages and lock entries must select a native release with the completion APIs. Clean installed ordinary SELECT, metadata and Volume checks are required before merge. The PR description explicitly records this prerequisite; source-built CI is not installed-package proof. The test-coverage observation is also bounded correctly: the repository staging suite currently selects Thrift. Retained source-built native/driver comparisons verify transfer bytes and removal; the shared core verifies symlink authorization. Named non-skipped scheduled execution with compatible released inputs remains open. Additional duplicate connector coverage is optional test placement, and is not claimed present. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at c89456f: package.json:95-102 pins 1.4.0, and KernelOperationBackend.ts:322 has no missing-method guard. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — package.json pins the kernel platform packages at 1.4.0, which has no completeVolumeOperation. Any kernel statement run with stagingAllowedLocalPath would fail with a TypeError instead of the old no-op warning. Bump the pins together with this change, or feature-detect the method and raise a clear error. 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed; this remains an open merge gate. The compatible native release must be published and all eight platform pins/locks updated before merging this consumer, followed by clean installed SELECT/metadata/Volume checks. The current source-built native proof does not make released 1.4.0 compatible. Coordinated release is the selected contract, so no silent older-native fallback is added. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 52c78ca: package.json:95-102 pins all eight kernel platform packages at 1.4.0, which has no completeVolumeOperation. Needs the published release. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed the compatible-package prerequisite. Published platform packages remain 1.4.0; all supported platform pins and locks must move to an actual compatible release before merge/release, with clean installed SELECT/metadata/Volume validation. The source selector alone does not close that gate. The coordinated release contract does not support older native packages, so feature detection is not a substitute for the required package bump. This finding stays open. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 2870cfe: package.json:95-102 still pin every native platform package at 1.4.0, and KernelOperationBackend.ts:322 calls completeVolumeOperation without a guard. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — The pinned 1.4.0 native packages appear to predate 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed as the existing required package gate. The latest native release remains v1.4.0, which lacks the completion API. Before merge/release, all eight platform pins/locks must select an actual compatible release and clean installed SELECT, metadata and Volume checks must pass. KERNEL_REV validates source integration only. Coordinated release is the selected contract, so feature detection or silently retaining the old ignore behavior would not satisfy it. This finding remains open. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 887d7d8: pins remain 1.4.0, which lacks completeVolumeOperation (KernelOperationBackend.ts:324). Merge is blocked until the release is coordinated. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — The method is called unconditionally for every kernel statement with stagingAllowedLocalPath, but the pinned 1.4.0 native packages lack it, so even a plain SELECT fails with a TypeError. Guard with a typeof check and throw a clear error, or bump the platform package pins in this PR. 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed as the same unresolved compatible-package gate. Native v1.4.0 is still the latest actual release and lacks completion APIs. Before merge/release this consumer needs a real compatible native release, all eight platform pin/lock updates and clean installed SELECT/metadata/Volume proof. KERNEL_REV and source-built CI validate source integration only. The coordinated release contract permits the new API; an older-binding fallback or invented minimum version would not satisfy the gate. This finding remains open. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 887d7d8: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — The package.json pins still resolve native 1.4.0, which has no completeVolumeOperation, so every kernel statement with stagingAllowedLocalPath fails with a TypeError, including plain SELECTs. Guard with typeof handle.completeVolumeOperation === 'function' and give a clear error or fall back. Also ship the native package bump and pins together. 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed as the same unresolved compatible-package prerequisite. Native v1.4.0 remains the latest actual release and lacks completion APIs. This consumer must receive a real compatible native release, all eight platform pin/lock updates and clean installed SELECT/metadata/Volume validation before merge/release. KERNEL_REV and source-built checks prove source integration only. The agreed resolution is the coordinated package update; a method check cannot make 1.4.0 support the required behavior, and silently skipping transfer is not acceptable. This gate remains open.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed as the same open compatible-package merge gate. Native 1.4.0 lacks completeVolumeOperation. Before merge/release all eight platform pins and locks must select a real compatible release and clean installed SELECT/metadata/Volume checks must pass. The source selector and local source-built validation do not establish package compatibility. A capability guard cannot supply the missing behavior, and an older-native fallback is outside the coordinated-release contract. No invented version or silent Volume fallback is being used. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at f877311: native pins still resolve 1.4.0, which lacks completeVolumeOperation (KernelOperationBackend.ts:324). This blocks merge until a compatible native release is pinned. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at f877311: the pins still resolve native 1.4.0, which has no completeVolumeOperation (KernelOperationBackend.ts:324). This gate stays open until a compatible release is pinned. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — completeStagingOperation calls handle.completeVolumeOperation with no capability check, while package.json still pins the native packages at 1.4.0. If those builds lack the method, every kernel statement with stagingAllowedLocalPath fails with a raw TypeError. Check that the method is a function and throw a clear error, or bump the native pins in the same change. 🔀 Found by both reviewers · confirmed by the judge
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed as the existing merge prerequisite. Published native 1.4.0 lacks the required API. Before merging, this connector requires an actual compatible native release, all eight platform pins/locks and clean installed SELECT, metadata and Volume checks. KERNEL_REV and the source-built native prove source integration only. Coordinated release is the selected contract; a capability check cannot supply the missing transfer implementation. This gate remains open. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Still open at 38edc92: native pins remain at 1.4.0, which lacks completeVolumeOperation, so the coordinated native release gate is still unmet. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — completeVolumeOperation is called without checking it exists, and the platform packages are still pinned to 1.4.0. With those binaries, any kernel statement that sets stagingAllowedLocalPath fails with a TypeError. Check the method exists and throw a clear HiveDriverError, or land the pin bump in this PR. 🔀 Found by both reviewers · confirmed by the judge |
||
| if (outcome === 'needs_input_stream') { | ||
|
aakash-saravanan-db marked this conversation as resolved.
Resolved
peco-review-bot[bot] marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
|
||
| throw new StagingError('Volume PUT requires a local file path'); | ||
|
aakash-saravanan-db marked this conversation as resolved.
Resolved
|
||
| } | ||
| this.volumeCompleted = outcome === 'completed'; | ||
|
aakash-saravanan-db marked this conversation as resolved.
Resolved
peco-review-bot[bot] marked this conversation as resolved.
Resolved
|
||
| } catch (err) { | ||
| const mapped = decodeNapiKernelError(err); | ||
| const metadata = (mapped as ErrorWithSqlState).kernelMetadata; | ||
| if ( | ||
|
peco-review-bot[bot] marked this conversation as resolved.
Resolved
|
||
| (mapped instanceof ParameterError && !metadata?.httpStatus) || | ||
| metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR' | ||
| ) { | ||
| throw new StagingError(mapped.message); | ||
|
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
aakash-saravanan-db marked this conversation as resolved.
Resolved
|
||
| } | ||
|
aakash-saravanan-db marked this conversation as resolved.
Resolved
|
||
| throw mapped; | ||
| } | ||
| } | ||
|
|
||
| public async fetchChunk({ | ||
| limit, | ||
| disableBuffering, | ||
|
|
@@ -320,6 +349,7 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // Cancel-mid-fetch propagation: if cancel() has flipped the | ||
| // lifecycle flag, fail locally without a wire round-trip. | ||
| failIfNotActive(this.lifecycle); | ||
| if (this.volumeCompleted) return []; | ||
| // Cooperative cancel (parity with ThriftOperationBackend): the facade | ||
| // supplies `isClosed` and re-checks `failIfClosed()` after we return, so | ||
| // bailing with `[]` at a yield point is the contract-correct way for a | ||
|
|
@@ -340,6 +370,7 @@ export default class KernelOperationBackend implements IOperationBackend { | |
|
|
||
| public async hasMore(): Promise<boolean> { | ||
| failIfNotActive(this.lifecycle); | ||
| if (this.volumeCompleted) return false; | ||
| try { | ||
| // `hasMore()` prefetches the next batch, so it fails like `fetchChunk()`. | ||
| const slicer = await this.getResultSlicer(); | ||
|
|
@@ -383,7 +414,7 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // `arrowSchema` doesn't hit apache-arrow@13's `Unrecognized type | ||
| // "Duration" (18)`. Matches what the per-batch fetch path already does. | ||
| arrowSchema: patchIpcBytes(arrowSchemaIpc.ipcBytes), | ||
| isStagingOperation: false, | ||
| isStagingOperation: Boolean(handle.isVolumeOperation), | ||
| }; | ||
| this.metadata = meta; | ||
| return meta; | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 Low — The changelog entry claims kernel Volume support, but the pinned native packages (1.4.0) don't include it. Note the required native package bump and the kernel change.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Unreleased entry describes the implemented source behavior. The actual release still requires a compatible native package set; the PR description records the kernel source selection and the package/pin/lock/install merge gate. Keep this release-note/package dependency open until the real versions exist. No currently published 1.4.0 compatibility is claimed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still open at c89456f: CHANGELOG.md:9 still doesn't say a native package bump is required or reference the kernel change.