Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
0a534de
Complete configured Volume operations in the native backend
aakash-saravanan-db Nov 13, 2026
47e379e
Retain inline results for configured Volume transfers
aakash-saravanan-db Nov 13, 2026
f482924
Correct Volume backend support documentation
aakash-saravanan-db Nov 13, 2026
c1f0821
Pin the native source for Volume completion
aakash-saravanan-db Nov 14, 2026
9d95ba0
Select the Volume kernel and format its path option
aakash-saravanan-db Nov 14, 2026
f6de7d3
Regenerate Volume completion declarations from the pinned kernel
aakash-saravanan-db Nov 14, 2026
973bad3
Preserve SQL execution errors during Volume completion
aakash-saravanan-db Nov 14, 2026
66a4c61
Pin native Volume storage retry and terminal response fixes
aakash-saravanan-db Nov 14, 2026
81c00ae
Cover portable native local file error decoding
aakash-saravanan-db Nov 14, 2026
e080abd
Preserve deferred Volume instruction HTTP errors
aakash-saravanan-db Nov 14, 2026
3ec62f5
Preserve portable file error fallback for unknown errno values
aakash-saravanan-db Nov 14, 2026
240c70f
build(kernel): select corrected Volume completion
aakash-saravanan-db Nov 14, 2026
69cc142
build: select native Volume cancellation and error handling
aakash-saravanan-db Nov 15, 2026
8b6e7fd
build: use native root authorization fix
aakash-saravanan-db Nov 15, 2026
8bfa89c
build: update kernel source for replayable uploads
aakash-saravanan-db Nov 15, 2026
13b0e60
build: update kernel for local stream close
aakash-saravanan-db Nov 15, 2026
7f8736d
fix(volume): select replayable file uploads and tighten validation
aakash-saravanan-db Nov 15, 2026
f2e3d68
fix(kernel): use permission-preserving Volume downloads
aakash-saravanan-db Nov 16, 2026
5658cb5
build(kernel): select sanitized storage redirects
aakash-saravanan-db Nov 16, 2026
a8029e0
Clarify Volume cancellation and consolidate routing coverage
aakash-saravanan-db Nov 16, 2026
baf8c7d
Validate native file error numbers and select shared completion
aakash-saravanan-db Nov 16, 2026
a35f7e1
Pin kernel with statement lifecycle cleanup
aakash-saravanan-db Nov 16, 2026
d54e1af
build: select kernel with async result cleanup
aakash-saravanan-db Nov 16, 2026
c4b5c4c
build: select kernel with Volume instruction error telemetry
aakash-saravanan-db Nov 16, 2026
ec92b5b
build: pin kernel with complete Volume instruction handling
aakash-saravanan-db Nov 16, 2026
3be7f97
fix(kernel): preserve ordinary post-SQL cancellation routing
aakash-saravanan-db Nov 17, 2026
a7c7fd1
build: use corrected kernel volume completion
aakash-saravanan-db Nov 17, 2026
1f87603
build: select kernel with origin-safe Volume redirects
aakash-saravanan-db Nov 17, 2026
0b4d434
build: select kernel with normalized upload lengths
aakash-saravanan-db Nov 17, 2026
c89456f
docs: keep unreleased changelog entries in one list
aakash-saravanan-db Nov 17, 2026
52c78ca
build: pin kernel completion telemetry correction
aakash-saravanan-db Nov 17, 2026
2870cfe
build: select kernel with restricted-directory GET support
aakash-saravanan-db Nov 18, 2026
c7a9e46
fix(kernel): cancel pending result waits before close
aakash-saravanan-db Nov 18, 2026
887d7d8
fix(kernel): await shared statement disposal on close
aakash-saravanan-db Nov 18, 2026
f877311
Finalize concurrent operation cleanup once
aakash-saravanan-db Nov 18, 2026
38edc92
build: pin kernel with bounded symlink resolution
aakash-saravanan-db Nov 18, 2026
ca22ba1
refactor(volume): reduce staging completion plumbing
aakash-saravanan-db Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
- Reduce result chunking and `fetchAll()` overhead for large query results.
- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry claims kernel Volume support, but the pinned native packages (1.4.0) don't include it. Note the required native package bump and the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Unreleased entry describes the implemented source behavior. The actual release still requires a compatible native package set; the PR description records the kernel source selection and the package/pin/lock/install merge gate. Keep this release-note/package dependency open until the real versions exist. No currently published 1.4.0 compatibility is claimed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at c89456f: CHANGELOG.md:9 still doesn't say a native package bump is required or reference the kernel change.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry doesn't say which native package version is required or link the kernel change, even though published 1.4.0 lacks the API. Add the required follow-up package bump and a kernel reference.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retained with the native release gate. A compatible published version has not been selected or released, so the changelog cannot accurately name one yet. Before merge, the native release, all eight platform pins/locks and installed checks must be completed together, and the final changelog must describe the actual required version. No invented version or older-native fallback is being used.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 52c78ca: CHANGELOG.md:9 doesn't name the required native package bump or link the kernel change; it should at least note that the follow-up bump is pending.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog presents kernel Volume transfers as shipped, but they depend on a native build newer than the pinned 1.4.0. State the minimum native package version required and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased, so it describes the candidate rather than an already shipped version. The required compatible native version remains part of the open release/package gate: v1.4.0 is still the latest actual release, and no minimum compatible version can yet be stated accurately. The PR Dependencies section identifies databricks/databricks-sql-kernel#465 and the exact source pin, plus the required platform pin/lock and clean-install work. Keep this gate open until the actual release and dependency update, when the minimum version can be recorded without inventing one.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: CHANGELOG.md:9 should say a later native package release is required and cite databricks-sql-kernel#465. No version number is needed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The release entry will document the actual supported native version when the coordinated dependency update lands. It currently sits under Unreleased, while the PR Dependencies section already identifies databricks-sql-kernel#465, its exact source pin and the compatible-release/pin/lock/clean-install prerequisites. A source-PR dependency sentence in the eventual user release history would age poorly once those prerequisites are delivered; the actual package requirement belongs with that release update. The suggested wording is optional pre-release clarity, while the real compatible-package gate remains required and open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: CHANGELOG.md:9 still doesn't say a later native package release is needed or cite databricks-sql-kernel#465. Needs human triage.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog entry announces kernel Volume support, but the pinned native packages are still 1.4.0, which lacks the feature. Say that a published native package bump is required and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased. The Dependencies section identifies the exact kernel source and explicitly blocks merge/release on the compatible native version, platform pins/locks and clean installed-package validation. The eventual release entry will name the actual supported version when that package update lands. A source-PR dependency sentence in lasting release history would age poorly after delivery; no minimum version is invented while v1.4.0 remains the latest release. Optional pre-release wording is acknowledged; the actual package/documentation gate remains open.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased. The PR Dependencies section names the exact kernel source and explicitly blocks merge/release on the real compatible native version, all eight platform pins/locks and clean installed-package checks. The release entry must document the actual supported version when that coordinated update lands; native 1.4.0 remains incompatible. A source-PR dependency in the eventual release history would become stale. This is optional pre-release wording, while the actual version/pin/documentation gate remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at f877311: CHANGELOG.md:9 doesn't mention the required native package bump or the kernel change. The PR description's merge gate doesn't put that in the changelog.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The changelog announces kernel Volume transfers as supported, but the code needs a native build newer than the pinned 1.4.0 packages. Note the required native package bump in the entry and reference the kernel change.

🔀 Found by Claude Opus 5.5 · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry is under Unreleased. The Dependencies section already blocks merge/release on a compatible native release, all platform pins/locks and clean installed checks. The final changelog must name the actual supported version when that coordinated update lands. No minimum version is invented while published 1.4.0 is incompatible; this stays with the open package/documentation gate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 38edc92: no Dependencies note in the CHANGELOG.md Unreleased section, and line 9 still names no required native bump or kernel change.

- Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520)
- Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485)
- Honor the client `useCloudFetch: false` default on the kernel backend so complete results can be fetched without cloud-storage downloads. (#574)
Expand Down
9 changes: 6 additions & 3 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ telemetry implementation, so only the options explicitly forwarded by
| `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ❌ | `100000` / — | Thrift uses it for the initial Direct Results row count; `null` disables Direct Results. Kernel does not retain or forward the supplied value. |
| `useCloudFetch` | `boolean` | ✅ | ❌ | `true` / — | Kernel logs and ignores this per-statement hint; Kernel result fetching owns its CloudFetch behavior. |
| `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` / — | Thrift uses it when supported and when the result is not CloudFetch. Kernel owns and auto-detects result compression. |
| `stagingAllowedLocalPath` | `string \| string[]` | ✅ | ❌ | — | Local allowlist for Thrift volume/staging operations. Kernel volume operations are unsupported. |
| `stagingAllowedLocalPath` | `string \| string[]` | ✅ | ✅ | — | Local allowlist for Volume operations. Kernel also requires resolved symlink targets within allowed roots; links within those roots are permitted. |
| `namedParameters` | `Record<string, DBSQLParameter \| DBSQLParameterValue>` | ✅ | ✅ | — | Named SQL parameters. Cannot be combined with non-empty `ordinalParameters`. Thrift requires a protocol that supports parameterized queries. |
| `ordinalParameters` | `Array<DBSQLParameter \| DBSQLParameterValue>` | ✅ | ✅ | — | Positional SQL parameters. Cannot be combined with non-empty `namedParameters`. |
| `queryTags` | `Record<string, string \| null \| undefined>` | ✅ | ✅ | — | Serialized into the per-statement `query_tags` conf overlay on both. |
Expand Down Expand Up @@ -244,11 +244,14 @@ session parameter surface.

Thrift-only capabilities include custom/token-provider authentication, custom
OAuth persistence, SOCKS proxies, authenticated/driver-buffer telemetry knobs,
query timeout, Direct Results `maxRows`, per-statement CloudFetch/LZ4 hints, and
volume operations.
query timeout, Direct Results `maxRows`, and per-statement CloudFetch/LZ4 hints.

Kernel-only capabilities include connection-pool sizing, proxy bypass hosts,
independent TLS hostname verification, optional U2M disk caching, and
per-statement `rowLimit`/`statementConf`.

The Kernel path remains internal and unstable; its behavior may change.

With the kernel backend, supplying `stagingAllowedLocalPath` completes Volume
transfers before `executeStatement` resolves and consumes instruction rows.
Omitting the option returns raw instructions without file effects.
2 changes: 1 addition & 1 deletion KERNEL_REV
Original file line number Diff line number Diff line change
@@ -1 +1 @@
dac32e2bf44468bdab06747dad4f289d1c0a18bd
2fe28af460d0db6eeb8c653e52eede954c90374c
10 changes: 7 additions & 3 deletions lib/DBSQLSession.ts
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,15 @@ export default class DBSQLSession implements IDBSQLSession {
// operation, everything will work as usual. In a case of staging operation, it will be processed like any
// other query - it will be possible to get data from it as usual, or use other operation methods.
if (options.stagingAllowedLocalPath !== undefined) {
const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath)
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
if (opBackend.completeStagingOperation) {
await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath));

ghost Nov 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — If native completion throws, the already-registered operation is never returned or closed, so its server handle stays alive until session close. Best-effort close the operation on this failure path and rethrow the original error.

🔀 Found by GPT-6.1 Terra · confirmed by the judge

return operation;
}
const metadata = await operation.getResultMetadata();
if (metadata.isStagingOperation) {
const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath)
? options.stagingAllowedLocalPath
: [options.stagingAllowedLocalPath];
return this.handleStagingOperation(operation, allowedLocalPath, metadata);
}
}
Expand Down
3 changes: 2 additions & 1 deletion lib/contracts/IDBSQLSession.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ export type ExecuteStatementOptions = {
* `false`/unset (default) runs the blocking direct-results path (faster,
* cancellable mid-compute); `true` submits and polls (returns a pending
* handle before completion). Default is sync, matching the python
* connector's `cursor.execute()`.
* connector's `cursor.execute()`. With `stagingAllowedLocalPath` configured,
* execution waits for staging completion regardless of this option.
*/
runAsync?: boolean;
maxRows?: number | bigint | Int64 | null;
Expand Down
2 changes: 2 additions & 0 deletions lib/contracts/IOperationBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@ export default interface IOperationBackend {
*/
getCachedResultMetadata?(): ResultMetadata | undefined;

completeStagingOperation?(allowedPaths: string[]): Promise<void>;

/** Cancel the operation. */
cancel(): Promise<Status>;

Expand Down
26 changes: 25 additions & 1 deletion lib/kernel/KernelErrorMapping.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { constants } from 'node:os';
import { getSystemErrorName } from 'node:util';
import HiveDriverError from '../errors/HiveDriverError';
import AuthenticationError from '../errors/AuthenticationError';
import OperationStateError, { OperationStateErrorCode } from '../errors/OperationStateError';
Expand Down Expand Up @@ -311,7 +313,29 @@ export function decodeNapiKernelError(err: unknown): Error {
const msg = envelope.message as string;
const sqlState = typeof envelope.sqlState === 'string' ? envelope.sqlState : undefined;

const jsErr = mapKernelErrorToJsError({ code, message: msg, sqlstate: sqlState });
let jsErr = mapKernelErrorToJsError({ code, message: msg, sqlstate: sqlState });
if (typeof envelope.localPath === 'string' && typeof envelope.localErrorKind === 'string') {
const kindCodes: Record<string, string> = {
NotFound: 'ENOENT',
PermissionDenied: 'EACCES',
AlreadyExists: 'EEXIST',
NotADirectory: 'ENOTDIR',
IsADirectory: 'EISDIR',
DirectoryNotEmpty: 'ENOTEMPTY',
};
const raw =
typeof envelope.localOsError === 'number' && Number.isSafeInteger(envelope.localOsError)
? envelope.localOsError
: undefined;
const systemCode = process.platform !== 'win32' && raw ? getSystemErrorName(-Math.abs(raw)) : undefined;
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
const localCode =
systemCode && /^E[A-Z0-9]+$/.test(systemCode) ? systemCode : kindCodes[envelope.localErrorKind] ?? 'EIO';
jsErr = Object.assign(new Error(msg), {
code: localCode,
errno: -((constants.errno as Record<string, number>)[localCode] ?? Math.abs(raw ?? constants.errno.EIO)),
path: envelope.localPath,
});
}

const meta = buildKernelMetadata(envelope);
// Skip the namespace attachment entirely when no fields validated
Expand Down
37 changes: 34 additions & 3 deletions lib/kernel/KernelOperationBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,14 @@ import { LogLevel } from '../contracts/IDBSQLLogger';
import { RowMode } from '../contracts/IDBSQLSession';
import Status from '../dto/Status';
import HiveDriverError from '../errors/HiveDriverError';
import StagingError from '../errors/StagingError';
import ParameterError from '../errors/ParameterError';
import OperationStateError, { OperationStateErrorCode } from '../errors/OperationStateError';
import ArrowResultConverter from '../result/ArrowResultConverter';
import ResultSlicer from '../result/ResultSlicer';
import KernelResultsProvider from './KernelResultsProvider';
import { arrowSchemaToThriftSchema, decodeIpcSchema, patchIpcBytes } from './KernelArrowIpc';
import { decodeNapiKernelError } from './KernelErrorMapping';
import { decodeNapiKernelError, ErrorWithSqlState } from './KernelErrorMapping';
import {
KernelStatement,
KernelNativeAsyncStatement,
Expand Down Expand Up @@ -84,7 +86,10 @@ export type KernelOperationStatement = KernelStatementHandle & Partial<KernelSta
* (`KernelResultsProvider` → `ArrowResultConverter` → `ResultSlicer`) consumes
* either interchangeably.
*/
type KernelFetchHandle = Pick<KernelStatement, 'fetchNextBatch' | 'schema'>;
type KernelFetchHandle = Pick<
KernelStatement,
'fetchNextBatch' | 'schema' | 'isVolumeOperation' | 'completeVolumeOperation'
>;

/**
* The rich operation-status surface the kernel exposes on a terminal sync
Expand Down Expand Up @@ -220,6 +225,8 @@ export default class KernelOperationBackend implements IOperationBackend {
// already-terminal statement. Drives both fetch and result-metadata.
private fetchHandlePromise?: Promise<KernelFetchHandle>;

private volumeCompleted = false;

// Async path only: set once the kernel reported the statement Succeeded (a
// status poll, or `awaitResult()` resolving). The kernel exposes no
// synchronous terminal-state accessor, so the observed success is retained.
Expand Down Expand Up @@ -308,6 +315,28 @@ export default class KernelOperationBackend implements IOperationBackend {
// Fetch / metadata (owned by the kernel-results pipeline).
// ---------------------------------------------------------------------------

public async completeStagingOperation(allowedPaths: string[]): Promise<void> {
failIfNotActive(this.lifecycle);
const handle = await this.getFetchHandle();
try {
const outcome = await handle.completeVolumeOperation(allowedPaths, true);

ghost Nov 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — Nothing checks that the loaded native binding actually has completeVolumeOperation. The PR description says published native 1.4.0 doesn't have it. On that binding, any kernel executeStatement that passes stagingAllowedLocalPath will reject with TypeError: handle.completeVolumeOperation is not a function. That includes plain SELECTs, because DBSQLSession now calls completeStagingOperation for every statement with the option set. decodeNapiKernelError passes the TypeError through unchanged. Before this PR the same call logged a warning and continued.

If the binding can't be pinned to a compatible release when this merges, add a feature-detect: when typeof handle.completeVolumeOperation !== 'function', throw a clear StagingError/HiveDriverError stating that the native binding doesn't support Volume operations. That would replace the opaque TypeError.

ghost Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: compatible published native packages are a merge prerequisite, including ordinary statements with stagingAllowedLocalPath. Source KERNEL_REV does not update installed platform dependencies. The matching release must supply completion, all platform pins/locks must advance, and clean installed queries plus Volume workflows must pass before merge. The accessible proxies currently expose 1.4.0, whose API is incompatible. Keeping the gate open; no old-native fallback or unavailable version is being introduced.

ghost Nov 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. I agree that pinning a compatible published native release is the right fix, and that it has to land before merge. Until then, the code at this line still calls handle.completeVolumeOperation without checking it exists, so on 1.4.0 it would still fail with the opaque TypeError. I'm leaving this thread open as the merge gate. It can be resolved once the platform pins and lockfiles move to a release that includes completeVolumeOperation. If that release won't be ready by merge time, the typeof check plus a clear StagingError is still a cheap safeguard.

ghost Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The release plan is coordinated native/connector version updates. Published 1.4.0 remains unsupported for these calls: before connector release, compatible platform pins and locks plus clean installed ordinary-query and Volume validation are required. Source validation selects kernel 855847bd with completeVolumeOperation. We are continuing review of that source pair while keeping package integration as a release prerequisite; no fallback, invented version or completed package proof is claimed.

ghost Nov 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for laying out the release plan. Your reply doesn't challenge the finding. The code at line 318 still calls handle.completeVolumeOperation without checking that it exists, and the installed pins still resolve to 1.4.0. My last reply already said this thread stays open as the merge gate. It can be resolved once the platform pins and lockfiles move to a native release that has completeVolumeOperation, or once a typeof guard that throws a clear StagingError lands. Until one of those is pushed, I'm keeping it open.

ghost Nov 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — completeStagingOperation calls handle.completeVolumeOperation(...) without checking that it exists. The repo still pins the published native packages at 1.4.0, which the PR description says lack this API. On those binaries, every kernel statement run with stagingAllowedLocalPath would fail. That includes plain SELECTs, because DBSQLSession sends every such statement through completion whenever the backend has the method. The failure is TypeError: handle.completeVolumeOperation is not a function, it is rethrown unchanged by decodeNapiKernelError, and the operation is closed. Before this PR the option was a no-op with a warning, so this is a regression for anyone running a published build, while the CHANGELOG says the feature is supported.

If the pins can't be bumped in this PR, check typeof handle.completeVolumeOperation === 'function' first. When it is missing, either return the raw rows (the omitted-option behaviour) or throw a clear StagingError saying the native binary is too old.

ghost Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed; this remains a merge prerequisite and also affects ordinary queries when the staging option is set. Published native 1.4.0 does not supply this completion API. The coordinated release must provide the compatible API, then all platform pins/locks must select it and clean installed-package ordinary-query/Volume workflows must pass. Source-built validation does not close that gate.

We are requiring the new native API rather than silently returning raw rows with an older binary or selecting an invented version. The PR description keeps this dependency unresolved; this comment is acknowledged, not claimed fixed.

ghost Nov 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming this. Requiring the new native API and treating it as a merge gate makes sense, and so does not quietly falling back to raw rows. The current code still calls handle.completeVolumeOperation without checking that it exists, and the pins are still at the published 1.4.0, so I'm leaving this thread open until the coordinated release ships and every platform pin and lock moves to it. Consider adding a typeof check that throws a clear StagingError saying the native binary is too old. That stays within your no-fallback policy and gives users a readable error in place of a TypeError if a package mismatch slips through.

ghost Nov 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — KernelOperationBackend always defines completeStagingOperation, so the if (opBackend.completeStagingOperation) check in DBSQLSession passes for every kernel operation. That call then runs handle.completeVolumeOperation(...) unconditionally.

The installed native packages are pinned to 1.4.0, which has no completeVolumeOperation. With that binding, any kernel executeStatement that sets stagingAllowedLocalPath hits TypeError: handle.completeVolumeOperation is not a function. That includes a plain SELECT. decodeNapiKernelError passes the error through unchanged, and the operation is closed. Before this PR the option was a logged no-op, so code written for the Thrift backend that sets it on every call would start failing.

The merge plan relies on the pins being bumped later, but nothing in the code enforces it. A small guard would avoid the regression. For example, if typeof handle.completeVolumeOperation !== 'function', fall back to the old warn-and-ignore behaviour or throw a clear HiveDriverError saying the native binding is too old. A unit test with a handle that lacks the method would cover it.

ghost Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that published 1.4.0 is incompatible. This remains an explicit merge/release gate: select the real compatible native release, update all platform pins and locks, and validate a clean installed package on ordinary queries and Volume workflows before merging. The source pin and source-built tests do not complete that gate.

We require the new native APIs and a coordinated release; this change will not silently fall back to ignored staging or raw instructions. No fallback or invented version is being used to close the finding. It remains open until the compatible packages and installation proof exist.

ghost Nov 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming the 1.4.0 incompatibility. Requiring a coordinated native release instead of a silent fallback is a fair design choice. Right now, though, the gate exists only in the merge plan. At head, line 318 still calls handle.completeVolumeOperation with no check, and the native pins haven't changed, so this thread stays open until the compatible packages and pins land. If you'd like a cheap safety net that keeps your no-fallback stance, a typeof check that throws a clear HiveDriverError saying the native binding is too old would turn the raw TypeError into a useful message. It would also guard against a mismatched install after release.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — completeStagingOperation calls handle.completeVolumeOperation(...) with no check that the method exists. DBSQLSession sends every kernel executeStatement that has stagingAllowedLocalPath set through this path, including ordinary SELECTs, because KernelOperationBackend always defines completeStagingOperation.

The PR description says published native 1.4.0 has no completion API. With that binding (or any stale cached .node), these calls now fail with TypeError: handle.completeVolumeOperation is not a function, and the operation gets closed. Before this change, the option was logged and ignored, and the statement worked normally.

assertBindingShape was written to catch exactly this kind of drift, but it doesn't check the new surface. Two possible fixes:

  • check typeof handle.completeVolumeOperation !== 'function' here and throw a clear error, or skip completion;
  • or add the method to the binding shape check so a mismatched binding is rejected when it loads.

ghost Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Published native 1.4.0 is incompatible; the compatible native versions, all eight platform pins/locks and clean installed-package ordinary-query/Volume checks remain required before merge. This is a coordinated release dependency, with no older-native fallback in the chosen contract. The source pin selects the current kernel, and source-built checks do not complete the package gate. Checking an old binding at load time could improve its diagnostic, but cannot make it compatible or replace that gate.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that the version pins are the real compatibility gate, and that a shape check can't make an old binding work. My concern is narrower: what happens when the pin gate is bypassed by a stale cached .node, a source build, or a hand-installed platform package. In that case every kernel executeStatement with stagingAllowedLocalPath set fails, including ordinary SELECTs, and the error is an opaque TypeError. Adding completeVolumeOperation to assertBindingShape is a one-line change. It would turn that into a clear error when the binding loads, which is the kind of drift that check exists to catch. The code at line 318 is unchanged, so I'm leaving this open for the maintainers to decide on. It's still low severity and doesn't block the coordinated release.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — completeStagingOperation calls handle.completeVolumeOperation(...) without checking that the loaded binding has it. The optional dependencies are still pinned to native 1.4.0, which has no completion API, so on the kernel backend every executeStatement with stagingAllowedLocalPath now fails, including a plain SELECT. The error is TypeError: handle.completeVolumeOperation is not a function. decodeNapiKernelError passes it through unchanged, and the facade closes the operation.

Before this PR the option was logged and ignored, so callers that set it globally and run ordinary queries break.

The facade check if (opBackend.completeStagingOperation) doesn't prevent this. That method exists on every KernelOperationBackend instance, so the check is always true. The check needs to be on the native handle. If typeof handle.completeVolumeOperation !== 'function', either keep the earlier behaviour (do nothing and return the raw instruction rows) or throw a clear HiveDriverError that names the minimum native version. Alternatively, bump the dependency pins in this PR.

ghost Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Published native 1.4.0 is incompatible. The actual compatible versions, all eight platform dependency pins/locks, and clean installed ordinary-query/metadata/Volume proof remain required before merge; a source SHA pin does not satisfy those requirements. This is a coordinated-release dependency, with no older-native fallback in the selected contract. A stale-binding method check could improve the load/runtime diagnostic, but cannot make the missing API compatible or replace the package gate. The PR remains blocked on that gate.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, a method check on the handle can't make native 1.4.0 compatible, and bumping the pins is the real fix. That's still outstanding, though: line 318 calls handle.completeVolumeOperation with no guard, and the optional dependencies still point to 1.4.0. I'm leaving this thread open until the new versions are published and all eight platform pins and locks are bumped. That way it stays in the unresolved queue as the visible merge gate. If the release slips and this branch gets installed against 1.4.0, a typeof guard that throws a HiveDriverError naming the minimum native version would still be cheap. It would replace an unclear TypeError on a plain SELECT.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — completeVolumeOperation is called without checking that it exists. The PR says published native 1.4.0 lacks the completion APIs, and the backend's own statement type is Partial<KernelStatement>. On a binding without the method, every kernel executeStatement that sets stagingAllowedLocalPath fails with TypeError: handle.completeVolumeOperation is not a function, and DBSQLSession then closes the operation. That includes ordinary SELECT/DML statements that aren't Volume operations; before this PR they ran with a warning. A typeof handle.completeVolumeOperation !== 'function' guard would fix it. It could throw a clear HiveDriverError/StagingError saying the native binding is too old, or fall back to the previous behaviour for non-volume statements (isVolumeOperation falsy).

ghost Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The incompatible-package concern is valid and remains a required merge gate: compatible native platform versions, all dependency pins/locks, and clean installed SELECT/metadata/Volume proof. Native 1.4.0 lacks these APIs. The source build selects kernel30c8d8e7; both the rebuilt sync/async lifecycle comparisons and ordinary-query controls pass. Coordinated native uptake is required, so we are retaining direct calls rather than adding an older-native compatibility path. This PR is not ready to merge until that release gate passes.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. Requiring a coordinated native release instead of adding a compatibility path is a reasonable choice. The code at this line still calls completeVolumeOperation directly, though. Until the published native dependency is pinned to a version that has the completion APIs, an install that pulls native 1.4.0 will still fail ordinary statements that set stagingAllowedLocalPath. I'll leave this thread open so the release gate stays in the unresolved queue. It can be closed once the pin and lockfile point to a compatible native release, or once a guard is added.

Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — handle.completeVolumeOperation(...) is called without checking that the loaded binding has it. The packages declared in package.json are still 1.4.0, and per the PR description that version has no completion API. With that binary installed, any kernel executeStatement that passes stagingAllowedLocalPath fails with TypeError: handle.completeVolumeOperation is not a function. That includes a plain SELECT. Before this change the option was logged and ignored. decodeNapiKernelError returns non-envelope errors unchanged, and DBSQLSession then closes the operation and rethrows, so callers see the raw TypeError.

The PR does say merge depends on updated pins. Until that happens, though, every environment that installs from npm gets this regression. Either guard with typeof handle.completeVolumeOperation === 'function' and throw a clear StagingError or HiveDriverError (or keep the old no-op), as readStatusFieldsFrom does for older bindings, or block the merge on the pin bump.

ghost Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: compatible native packages, all eight platform pins/locks and clean installed SELECT, metadata and Volume proof are required before this PR can merge. The declared 1.4.0 inputs do not satisfy the new API contract, and the source SHA pin is not released-package proof. This gate remains open in the PR description. Coordinated native uptake is required; an older-binding no-op/fallback would conceal unsupported Volume behavior and is not the selected contract. No merge-ready or complete-parity claim is made.

ghost Nov 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, blocking the merge on the native package bump is one of the resolutions I suggested, and choosing it over a silent no-op makes sense. Line 317 still calls handle.completeVolumeOperation with no guard, though, and package.json still pins 1.4.0. So the regression is real at current head. I'll leave this thread open as the visible merge gate. It can be resolved once the eight platform pins and lockfile point at a released binding that ships the completion API.

ghost Nov 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — completeStagingOperation calls handle.completeVolumeOperation(...) with no check that the method exists. The repo still pins the native packages to 1.4.0, and that release has no completion API. With the native module that actually gets installed, any kernel executeStatement that sets stagingAllowedLocalPath now rejects with TypeError: handle.completeVolumeOperation is not a function. That includes ordinary SELECTs, which used to log a warning and run normally. DBSQLSession only checks for the backend's completeStagingOperation, which always exists on KernelOperationBackend, so this check never falls back.

The PR description already lists the pin bump as a merge prerequisite, so this may be intentional. If the code can land before compatible native packages ship, check typeof handle.completeVolumeOperation === 'function' and throw a clear StagingError or HiveDriverError (or keep the old warn-and-ignore path) instead of surfacing a raw TypeError.

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as an unresolved package compatibility gate: published native 1.4.0 lacks this API. Merge/release requires the actual compatible versions across all platform packages, updated pins/locks and clean installed SELECT, metadata and staging checks. The source build uses kernel 4e90b0d0; it does not prove installed-package compatibility. The PR description records the gate and the PR remains open.

The description’s source selector now matches KERNEL_REV. The retained comparisons at kernel029c59cc keep their original build identity. Completion declarations and the C/JS binding API are unchanged between the previous d9fcf834 source selector and 4e90b0d0; these kernel corrections are private. The optional needs_input_stream stub case would repeat the already-covered native stream-handshake and adapter error-pass-through contracts; the existing catch preserves an ordinary StagingError. It is not evidence of an adapter behavior defect. The changelog’s extra blank line is a cosmetic nit, left outside this focused source-pin update.

ghost Nov 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at c89456f: lib/kernel/KernelOperationBackend.ts:322 has no capability guard, and the 1.4.0 native pins remain until compatible packages ship.

ghost Nov 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The optional native packages are pinned to 1.4.0, which lacks completeVolumeOperation. A kernel statement with stagingAllowedLocalPath then throws a raw TypeError, even for ordinary SQL. Guard for the missing method with a clear error or ignore fallback, and bump the native packages.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid required merge gate: all eight platform packages and lock entries must select a native release with the completion APIs. Clean installed ordinary SELECT, metadata and Volume checks are required before merge. The PR description explicitly records this prerequisite; source-built CI is not installed-package proof.

The test-coverage observation is also bounded correctly: the repository staging suite currently selects Thrift. Retained source-built native/driver comparisons verify transfer bytes and removal; the shared core verifies symlink authorization. Named non-skipped scheduled execution with compatible released inputs remains open. Additional duplicate connector coverage is optional test placement, and is not claimed present.

ghost Nov 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at c89456f: package.json:95-102 pins 1.4.0, and KernelOperationBackend.ts:322 has no missing-method guard.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — package.json pins the kernel platform packages at 1.4.0, which has no completeVolumeOperation. Any kernel statement run with stagingAllowedLocalPath would fail with a TypeError instead of the old no-op warning. Bump the pins together with this change, or feature-detect the method and raise a clear error.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed; this remains an open merge gate. The compatible native release must be published and all eight platform pins/locks updated before merging this consumer, followed by clean installed SELECT/metadata/Volume checks. The current source-built native proof does not make released 1.4.0 compatible. Coordinated release is the selected contract, so no silent older-native fallback is added.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 52c78ca: package.json:95-102 pins all eight kernel platform packages at 1.4.0, which has no completeVolumeOperation. Needs the published release.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeVolumeOperation is called with no feature-detection, and package.json still pins the native kernel packages at 1.4.0. A binding without the method fails every statement run with stagingAllowedLocalPath with a raw TypeError, where it used to be ignored with a warning. Check typeof handle.completeVolumeOperation === 'function' and throw a clear error naming the required native version, and bump the platform pins together with this change.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the compatible-package prerequisite. Published platform packages remain 1.4.0; all supported platform pins and locks must move to an actual compatible release before merge/release, with clean installed SELECT/metadata/Volume validation. The source selector alone does not close that gate. The coordinated release contract does not support older native packages, so feature detection is not a substitute for the required package bump. This finding stays open.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 2870cfe: package.json:95-102 still pin every native platform package at 1.4.0, and KernelOperationBackend.ts:322 calls completeVolumeOperation without a guard.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The pinned 1.4.0 native packages appear to predate completeVolumeOperation, so every kernel statement with stagingAllowedLocalPath would fail with a TypeError. Check typeof handle.completeVolumeOperation === 'function' and throw a clear unsupported-binding error, or fall back to the old ignore path. Land the package pin bump with this change.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing required package gate. The latest native release remains v1.4.0, which lacks the completion API. Before merge/release, all eight platform pins/locks must select an actual compatible release and clean installed SELECT, metadata and Volume checks must pass. KERNEL_REV validates source integration only. Coordinated release is the selected contract, so feature detection or silently retaining the old ignore behavior would not satisfy it. This finding remains open.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: pins remain 1.4.0, which lacks completeVolumeOperation (KernelOperationBackend.ts:324). Merge is blocked until the release is coordinated.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The method is called unconditionally for every kernel statement with stagingAllowedLocalPath, but the pinned 1.4.0 native packages lack it, so even a plain SELECT fails with a TypeError. Guard with a typeof check and throw a clear error, or bump the platform package pins in this PR.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the same unresolved compatible-package gate. Native v1.4.0 is still the latest actual release and lacks completion APIs. Before merge/release this consumer needs a real compatible native release, all eight platform pin/lock updates and clean installed SELECT/metadata/Volume proof. KERNEL_REV and source-built CI validate source integration only. The coordinated release contract permits the new API; an older-binding fallback or invented minimum version would not satisfy the gate. This finding remains open.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 887d7d8: completeVolumeOperation (KernelOperationBackend.ts:324) needs a compatible native release and all eight platform pins/locks bumped.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The package.json pins still resolve native 1.4.0, which has no completeVolumeOperation, so every kernel statement with stagingAllowedLocalPath fails with a TypeError, including plain SELECTs. Guard with typeof handle.completeVolumeOperation === 'function' and give a clear error or fall back. Also ship the native package bump and pins together.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the same unresolved compatible-package prerequisite. Native v1.4.0 remains the latest actual release and lacks completion APIs. This consumer must receive a real compatible native release, all eight platform pin/lock updates and clean installed SELECT/metadata/Volume validation before merge/release. KERNEL_REV and source-built checks prove source integration only. The agreed resolution is the coordinated package update; a method check cannot make 1.4.0 support the required behavior, and silently skipping transfer is not acceptable. This gate remains open.

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the same open compatible-package merge gate. Native 1.4.0 lacks completeVolumeOperation. Before merge/release all eight platform pins and locks must select a real compatible release and clean installed SELECT/metadata/Volume checks must pass. The source selector and local source-built validation do not establish package compatibility. A capability guard cannot supply the missing behavior, and an older-native fallback is outside the coordinated-release contract. No invented version or silent Volume fallback is being used.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at f877311: native pins still resolve 1.4.0, which lacks completeVolumeOperation (KernelOperationBackend.ts:324). This blocks merge until a compatible native release is pinned.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at f877311: the pins still resolve native 1.4.0, which has no completeVolumeOperation (KernelOperationBackend.ts:324). This gate stays open until a compatible release is pinned.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeStagingOperation calls handle.completeVolumeOperation with no capability check, while package.json still pins the native packages at 1.4.0. If those builds lack the method, every kernel statement with stagingAllowedLocalPath fails with a raw TypeError. Check that the method is a function and throw a clear error, or bump the native pins in the same change.

🔀 Found by both reviewers · confirmed by the judge

ghost Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed as the existing merge prerequisite. Published native 1.4.0 lacks the required API. Before merging, this connector requires an actual compatible native release, all eight platform pins/locks and clean installed SELECT, metadata and Volume checks. KERNEL_REV and the source-built native prove source integration only. Coordinated release is the selected contract; a capability check cannot supply the missing transfer implementation. This gate remains open.

ghost Nov 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 38edc92: native pins remain at 1.4.0, which lacks completeVolumeOperation, so the coordinated native release gate is still unmet.

ghost Nov 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — completeVolumeOperation is called without checking it exists, and the platform packages are still pinned to 1.4.0. With those binaries, any kernel statement that sets stagingAllowedLocalPath fails with a TypeError. Check the method exists and throw a clear HiveDriverError, or land the pin bump in this PR.

🔀 Found by both reviewers · confirmed by the judge

if (outcome === 'needs_input_stream') {
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
throw new StagingError('Volume PUT requires a local file path');
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
}
this.volumeCompleted = outcome === 'completed';
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
} catch (err) {
const mapped = decodeNapiKernelError(err);
const metadata = (mapped as ErrorWithSqlState).kernelMetadata;
if (
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
(mapped instanceof ParameterError && !metadata?.httpStatus) ||
metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR'
) {
throw new StagingError(mapped.message);
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
}
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
throw mapped;
}
}

public async fetchChunk({
limit,
disableBuffering,
Expand All @@ -320,6 +349,7 @@ export default class KernelOperationBackend implements IOperationBackend {
// Cancel-mid-fetch propagation: if cancel() has flipped the
// lifecycle flag, fail locally without a wire round-trip.
failIfNotActive(this.lifecycle);
if (this.volumeCompleted) return [];
// Cooperative cancel (parity with ThriftOperationBackend): the facade
// supplies `isClosed` and re-checks `failIfClosed()` after we return, so
// bailing with `[]` at a yield point is the contract-correct way for a
Expand All @@ -340,6 +370,7 @@ export default class KernelOperationBackend implements IOperationBackend {

public async hasMore(): Promise<boolean> {
failIfNotActive(this.lifecycle);
if (this.volumeCompleted) return false;
try {
// `hasMore()` prefetches the next batch, so it fails like `fetchChunk()`.
const slicer = await this.getResultSlicer();
Expand Down Expand Up @@ -383,7 +414,7 @@ export default class KernelOperationBackend implements IOperationBackend {
// `arrowSchema` doesn't hit apache-arrow@13's `Unrecognized type
// "Duration" (18)`. Matches what the per-batch fetch path already does.
arrowSchema: patchIpcBytes(arrowSchemaIpc.ipcBytes),
isStagingOperation: false,
isStagingOperation: Boolean(handle.isVolumeOperation),
};
this.metadata = meta;
return meta;
Expand Down
23 changes: 7 additions & 16 deletions lib/kernel/KernelSessionBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,9 @@ export default class KernelSessionBackend implements ISessionBackend {
* `query_tags` key (the same wire shape Thrift's `serializeQueryTags`
* produces), merged with any explicit `statementConf`.
*
* Still rejected (genuinely unsupported on kernel, rather than silently
* dropped): `useCloudFetch` (governed by the kernel `ResultConfig`, not a
* per-statement knob), `useLZ4Compression` (kernel owns result compression),
* and `stagingAllowedLocalPath` (volume operations). `maxRows` is applied by
* `useCloudFetch` and `useLZ4Compression` are logged and ignored because the
* kernel owns result fetching and compression. `stagingAllowedLocalPath` is
* applied by the facade. `maxRows` is applied by
* the facade at fetch time, so it is intentionally not handled here.
*/
public async executeStatement(statement: string, options: ExecuteStatementOptions): Promise<IOperationBackend> {
Expand All @@ -160,8 +159,7 @@ export default class KernelSessionBackend implements ISessionBackend {
// Per-statement options the kernel backend doesn't honour are treated as
// NO-OPs (logged at warn), not errors — so call sites written for the Thrift
// backend are drop-in on the kernel path. These are perf/format hints the
// kernel governs internally (useCloudFetch / useLZ4Compression) or a feature
// it doesn't expose yet (staging); ignoring them cannot change query results.
// kernel governs internally (useCloudFetch / useLZ4Compression).
// NB: parameter binding (compound/BINARY) is deliberately NOT no-op'd — a
// dropped param would silently change results, so it still throws.
if (options.useCloudFetch !== undefined) {
Expand All @@ -180,15 +178,6 @@ export default class KernelSessionBackend implements ISessionBackend {
'kernel executeStatement: ignoring per-statement `useLZ4Compression` — result compression is governed by the kernel (no-op on kernel).',
);
}
if (options.stagingAllowedLocalPath !== undefined) {
this.context
.getLogger()
.log(
LogLevel.warn,
'kernel executeStatement: ignoring `stagingAllowedLocalPath` — volume/staging operations are not yet supported on the kernel backend (no-op).',
);
}

// `runAsync` selects the kernel execution path. NOTE: this is a kernel-
// specific use of the option — the Thrift backend hardcodes `runAsync: true`
// on the wire and never reads `options.runAsync`, so the field is a no-op
Expand All @@ -212,7 +201,9 @@ export default class KernelSessionBackend implements ISessionBackend {
// the backend polls `status()` to terminal in `waitUntilReady()` and
// materialises results via `awaitResult()`. `queryTimeout` is a no-op
// here too.
const runAsync = options.runAsync ?? false;
// Configured staging completes before the facade returns, so retain inline
// results even when the caller requests asynchronous execution.
const runAsync = (options.runAsync ?? false) && options.stagingAllowedLocalPath === undefined;
// `queryTimeout` is a NO-OP on the kernel (kernel) backend. It is the JDBC
// `setQueryTimeout` knob which — per the option's JSDoc — is effective only
// on Compute clusters; SQL Warehouses (what kernel targets) use the
Expand Down
25 changes: 18 additions & 7 deletions native/kernel/index.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions native/kernel/index.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading