diff --git a/CHANGELOG.md b/CHANGELOG.md index 938983a3..8fd334f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. - Reduce result chunking and `fetchAll()` overhead for large query results. +- Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. - Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520) - Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485) - Honor the client `useCloudFetch: false` default on the kernel backend so complete results can be fetched without cloud-storage downloads. (#574) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index edba6002..e7c19dbd 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -213,7 +213,7 @@ telemetry implementation, so only the options explicitly forwarded by | `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ❌ | `100000` / — | Thrift uses it for the initial Direct Results row count; `null` disables Direct Results. Kernel does not retain or forward the supplied value. | | `useCloudFetch` | `boolean` | ✅ | ❌ | `true` / — | Kernel logs and ignores this per-statement hint; Kernel result fetching owns its CloudFetch behavior. | | `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` / — | Thrift uses it when supported and when the result is not CloudFetch. Kernel owns and auto-detects result compression. | -| `stagingAllowedLocalPath` | `string \| string[]` | ✅ | ❌ | — | Local allowlist for Thrift volume/staging operations. Kernel volume operations are unsupported. | +| `stagingAllowedLocalPath` | `string \| string[]` | ✅ | ✅ | — | Local allowlist for Volume operations. Kernel also requires resolved symlink targets within allowed roots; links within those roots are permitted. | | `namedParameters` | `Record` | ✅ | ✅ | — | Named SQL parameters. Cannot be combined with non-empty `ordinalParameters`. Thrift requires a protocol that supports parameterized queries. | | `ordinalParameters` | `Array` | ✅ | ✅ | — | Positional SQL parameters. Cannot be combined with non-empty `namedParameters`. | | `queryTags` | `Record` | ✅ | ✅ | — | Serialized into the per-statement `query_tags` conf overlay on both. | @@ -244,11 +244,14 @@ session parameter surface. Thrift-only capabilities include custom/token-provider authentication, custom OAuth persistence, SOCKS proxies, authenticated/driver-buffer telemetry knobs, -query timeout, Direct Results `maxRows`, per-statement CloudFetch/LZ4 hints, and -volume operations. +query timeout, Direct Results `maxRows`, and per-statement CloudFetch/LZ4 hints. Kernel-only capabilities include connection-pool sizing, proxy bypass hosts, independent TLS hostname verification, optional U2M disk caching, and per-statement `rowLimit`/`statementConf`. The Kernel path remains internal and unstable; its behavior may change. + +With the kernel backend, supplying `stagingAllowedLocalPath` completes Volume +transfers before `executeStatement` resolves and consumes instruction rows. +Omitting the option returns raw instructions without file effects. diff --git a/KERNEL_REV b/KERNEL_REV index 18337b65..bc49f880 100644 --- a/KERNEL_REV +++ b/KERNEL_REV @@ -1 +1 @@ -dac32e2bf44468bdab06747dad4f289d1c0a18bd +2fe28af460d0db6eeb8c653e52eede954c90374c diff --git a/lib/DBSQLSession.ts b/lib/DBSQLSession.ts index 5bca61d6..95024946 100644 --- a/lib/DBSQLSession.ts +++ b/lib/DBSQLSession.ts @@ -109,11 +109,15 @@ export default class DBSQLSession implements IDBSQLSession { // operation, everything will work as usual. In a case of staging operation, it will be processed like any // other query - it will be possible to get data from it as usual, or use other operation methods. if (options.stagingAllowedLocalPath !== undefined) { + const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath) + ? options.stagingAllowedLocalPath + : [options.stagingAllowedLocalPath]; + if (opBackend.completeStagingOperation) { + await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath)); + return operation; + } const metadata = await operation.getResultMetadata(); if (metadata.isStagingOperation) { - const allowedLocalPath = Array.isArray(options.stagingAllowedLocalPath) - ? options.stagingAllowedLocalPath - : [options.stagingAllowedLocalPath]; return this.handleStagingOperation(operation, allowedLocalPath, metadata); } } diff --git a/lib/contracts/IDBSQLSession.ts b/lib/contracts/IDBSQLSession.ts index 12d7d6c2..da5a01b1 100644 --- a/lib/contracts/IDBSQLSession.ts +++ b/lib/contracts/IDBSQLSession.ts @@ -32,7 +32,8 @@ export type ExecuteStatementOptions = { * `false`/unset (default) runs the blocking direct-results path (faster, * cancellable mid-compute); `true` submits and polls (returns a pending * handle before completion). Default is sync, matching the python - * connector's `cursor.execute()`. + * connector's `cursor.execute()`. With `stagingAllowedLocalPath` configured, + * execution waits for staging completion regardless of this option. */ runAsync?: boolean; maxRows?: number | bigint | Int64 | null; diff --git a/lib/contracts/IOperationBackend.ts b/lib/contracts/IOperationBackend.ts index 109de87e..791008d3 100644 --- a/lib/contracts/IOperationBackend.ts +++ b/lib/contracts/IOperationBackend.ts @@ -113,6 +113,8 @@ export default interface IOperationBackend { */ getCachedResultMetadata?(): ResultMetadata | undefined; + completeStagingOperation?(allowedPaths: string[]): Promise; + /** Cancel the operation. */ cancel(): Promise; diff --git a/lib/kernel/KernelErrorMapping.ts b/lib/kernel/KernelErrorMapping.ts index d83d7e59..2239444b 100644 --- a/lib/kernel/KernelErrorMapping.ts +++ b/lib/kernel/KernelErrorMapping.ts @@ -1,3 +1,5 @@ +import { constants } from 'node:os'; +import { getSystemErrorName } from 'node:util'; import HiveDriverError from '../errors/HiveDriverError'; import AuthenticationError from '../errors/AuthenticationError'; import OperationStateError, { OperationStateErrorCode } from '../errors/OperationStateError'; @@ -311,7 +313,29 @@ export function decodeNapiKernelError(err: unknown): Error { const msg = envelope.message as string; const sqlState = typeof envelope.sqlState === 'string' ? envelope.sqlState : undefined; - const jsErr = mapKernelErrorToJsError({ code, message: msg, sqlstate: sqlState }); + let jsErr = mapKernelErrorToJsError({ code, message: msg, sqlstate: sqlState }); + if (typeof envelope.localPath === 'string' && typeof envelope.localErrorKind === 'string') { + const kindCodes: Record = { + NotFound: 'ENOENT', + PermissionDenied: 'EACCES', + AlreadyExists: 'EEXIST', + NotADirectory: 'ENOTDIR', + IsADirectory: 'EISDIR', + DirectoryNotEmpty: 'ENOTEMPTY', + }; + const raw = + typeof envelope.localOsError === 'number' && Number.isSafeInteger(envelope.localOsError) + ? envelope.localOsError + : undefined; + const systemCode = process.platform !== 'win32' && raw ? getSystemErrorName(-Math.abs(raw)) : undefined; + const localCode = + systemCode && /^E[A-Z0-9]+$/.test(systemCode) ? systemCode : kindCodes[envelope.localErrorKind] ?? 'EIO'; + jsErr = Object.assign(new Error(msg), { + code: localCode, + errno: -((constants.errno as Record)[localCode] ?? Math.abs(raw ?? constants.errno.EIO)), + path: envelope.localPath, + }); + } const meta = buildKernelMetadata(envelope); // Skip the namespace attachment entirely when no fields validated diff --git a/lib/kernel/KernelOperationBackend.ts b/lib/kernel/KernelOperationBackend.ts index 81647ece..4dffdf2b 100644 --- a/lib/kernel/KernelOperationBackend.ts +++ b/lib/kernel/KernelOperationBackend.ts @@ -46,12 +46,14 @@ import { LogLevel } from '../contracts/IDBSQLLogger'; import { RowMode } from '../contracts/IDBSQLSession'; import Status from '../dto/Status'; import HiveDriverError from '../errors/HiveDriverError'; +import StagingError from '../errors/StagingError'; +import ParameterError from '../errors/ParameterError'; import OperationStateError, { OperationStateErrorCode } from '../errors/OperationStateError'; import ArrowResultConverter from '../result/ArrowResultConverter'; import ResultSlicer from '../result/ResultSlicer'; import KernelResultsProvider from './KernelResultsProvider'; import { arrowSchemaToThriftSchema, decodeIpcSchema, patchIpcBytes } from './KernelArrowIpc'; -import { decodeNapiKernelError } from './KernelErrorMapping'; +import { decodeNapiKernelError, ErrorWithSqlState } from './KernelErrorMapping'; import { KernelStatement, KernelNativeAsyncStatement, @@ -84,7 +86,10 @@ export type KernelOperationStatement = KernelStatementHandle & Partial; +type KernelFetchHandle = Pick< + KernelStatement, + 'fetchNextBatch' | 'schema' | 'isVolumeOperation' | 'completeVolumeOperation' +>; /** * The rich operation-status surface the kernel exposes on a terminal sync @@ -220,6 +225,8 @@ export default class KernelOperationBackend implements IOperationBackend { // already-terminal statement. Drives both fetch and result-metadata. private fetchHandlePromise?: Promise; + private volumeCompleted = false; + // Async path only: set once the kernel reported the statement Succeeded (a // status poll, or `awaitResult()` resolving). The kernel exposes no // synchronous terminal-state accessor, so the observed success is retained. @@ -308,6 +315,28 @@ export default class KernelOperationBackend implements IOperationBackend { // Fetch / metadata (owned by the kernel-results pipeline). // --------------------------------------------------------------------------- + public async completeStagingOperation(allowedPaths: string[]): Promise { + failIfNotActive(this.lifecycle); + const handle = await this.getFetchHandle(); + try { + const outcome = await handle.completeVolumeOperation(allowedPaths, true); + if (outcome === 'needs_input_stream') { + throw new StagingError('Volume PUT requires a local file path'); + } + this.volumeCompleted = outcome === 'completed'; + } catch (err) { + const mapped = decodeNapiKernelError(err); + const metadata = (mapped as ErrorWithSqlState).kernelMetadata; + if ( + (mapped instanceof ParameterError && !metadata?.httpStatus) || + metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR' + ) { + throw new StagingError(mapped.message); + } + throw mapped; + } + } + public async fetchChunk({ limit, disableBuffering, @@ -320,6 +349,7 @@ export default class KernelOperationBackend implements IOperationBackend { // Cancel-mid-fetch propagation: if cancel() has flipped the // lifecycle flag, fail locally without a wire round-trip. failIfNotActive(this.lifecycle); + if (this.volumeCompleted) return []; // Cooperative cancel (parity with ThriftOperationBackend): the facade // supplies `isClosed` and re-checks `failIfClosed()` after we return, so // bailing with `[]` at a yield point is the contract-correct way for a @@ -340,6 +370,7 @@ export default class KernelOperationBackend implements IOperationBackend { public async hasMore(): Promise { failIfNotActive(this.lifecycle); + if (this.volumeCompleted) return false; try { // `hasMore()` prefetches the next batch, so it fails like `fetchChunk()`. const slicer = await this.getResultSlicer(); @@ -383,7 +414,7 @@ export default class KernelOperationBackend implements IOperationBackend { // `arrowSchema` doesn't hit apache-arrow@13's `Unrecognized type // "Duration" (18)`. Matches what the per-batch fetch path already does. arrowSchema: patchIpcBytes(arrowSchemaIpc.ipcBytes), - isStagingOperation: false, + isStagingOperation: Boolean(handle.isVolumeOperation), }; this.metadata = meta; return meta; diff --git a/lib/kernel/KernelSessionBackend.ts b/lib/kernel/KernelSessionBackend.ts index dd55dd4c..3fe5919f 100644 --- a/lib/kernel/KernelSessionBackend.ts +++ b/lib/kernel/KernelSessionBackend.ts @@ -148,10 +148,9 @@ export default class KernelSessionBackend implements ISessionBackend { * `query_tags` key (the same wire shape Thrift's `serializeQueryTags` * produces), merged with any explicit `statementConf`. * - * Still rejected (genuinely unsupported on kernel, rather than silently - * dropped): `useCloudFetch` (governed by the kernel `ResultConfig`, not a - * per-statement knob), `useLZ4Compression` (kernel owns result compression), - * and `stagingAllowedLocalPath` (volume operations). `maxRows` is applied by + * `useCloudFetch` and `useLZ4Compression` are logged and ignored because the + * kernel owns result fetching and compression. `stagingAllowedLocalPath` is + * applied by the facade. `maxRows` is applied by * the facade at fetch time, so it is intentionally not handled here. */ public async executeStatement(statement: string, options: ExecuteStatementOptions): Promise { @@ -160,8 +159,7 @@ export default class KernelSessionBackend implements ISessionBackend { // Per-statement options the kernel backend doesn't honour are treated as // NO-OPs (logged at warn), not errors — so call sites written for the Thrift // backend are drop-in on the kernel path. These are perf/format hints the - // kernel governs internally (useCloudFetch / useLZ4Compression) or a feature - // it doesn't expose yet (staging); ignoring them cannot change query results. + // kernel governs internally (useCloudFetch / useLZ4Compression). // NB: parameter binding (compound/BINARY) is deliberately NOT no-op'd — a // dropped param would silently change results, so it still throws. if (options.useCloudFetch !== undefined) { @@ -180,15 +178,6 @@ export default class KernelSessionBackend implements ISessionBackend { 'kernel executeStatement: ignoring per-statement `useLZ4Compression` — result compression is governed by the kernel (no-op on kernel).', ); } - if (options.stagingAllowedLocalPath !== undefined) { - this.context - .getLogger() - .log( - LogLevel.warn, - 'kernel executeStatement: ignoring `stagingAllowedLocalPath` — volume/staging operations are not yet supported on the kernel backend (no-op).', - ); - } - // `runAsync` selects the kernel execution path. NOTE: this is a kernel- // specific use of the option — the Thrift backend hardcodes `runAsync: true` // on the wire and never reads `options.runAsync`, so the field is a no-op @@ -212,7 +201,9 @@ export default class KernelSessionBackend implements ISessionBackend { // the backend polls `status()` to terminal in `waitUntilReady()` and // materialises results via `awaitResult()`. `queryTimeout` is a no-op // here too. - const runAsync = options.runAsync ?? false; + // Configured staging completes before the facade returns, so retain inline + // results even when the caller requests asynchronous execution. + const runAsync = (options.runAsync ?? false) && options.stagingAllowedLocalPath === undefined; // `queryTimeout` is a NO-OP on the kernel (kernel) backend. It is the JDBC // `setQueryTimeout` knob which — per the option's JSDoc — is effective only // on Compute clusters; SQL Warehouses (what kernel targets) use the diff --git a/native/kernel/index.d.ts b/native/kernel/index.d.ts index 87640ca0..b5a315dc 100644 --- a/native/kernel/index.d.ts +++ b/native/kernel/index.d.ts @@ -18,6 +18,9 @@ * post-close contract. */ export declare class AsyncResultHandle { + get isVolumeOperation(): boolean + /** Complete a server-marked file transfer while retaining its instruction rows. */ + completeVolumeOperation(allowedPaths: Array, removeNotFoundIsSuccess?: boolean | undefined | null): Promise /** * Server-issued statement id. Cached at construction; readable * for log correlation. Matches the parent `AsyncStatement`'s @@ -149,9 +152,8 @@ export declare class AsyncStatement { * re-attempting the wire call. The kernel's own `Drop` * fire-and-forget retry runs once in the background. * - * Also invalidates the handles `awaitResult()` returned: their - * `fetchNextBatch()` then rejects with `InvalidStatementHandle`, - * unless a fetch was in progress when `close()` ran. + * Also invalidates the handles `awaitResult()` returned. An operation + * already in flight may finish before its handle is cleared. */ close(): Promise } @@ -438,6 +440,9 @@ export declare class Connection { * disposed statement can still read them. */ export declare class Statement { + get isVolumeOperation(): boolean + /** Complete a server-marked file transfer while retaining its instruction rows. */ + completeVolumeOperation(allowedPaths: Array, removeNotFoundIsSuccess?: boolean | undefined | null): Promise /** * Server-issued statement id. Cached at construction; readable * even after `close()` so JS-side log lines can correlate against @@ -1054,11 +1059,11 @@ export interface ConnectionOptions { */ proxy?: ProxyInput /** - * Per-connection socket read timeout, in milliseconds. Caps how - * long a single HTTP round-trip may block waiting on the server - * before the request errors out. Maps onto the kernel + * Per-connection HTTP request deadline, in milliseconds. Caps a single + * request through response-body completion. Maps onto the kernel * [`HttpConfig::request_timeout`] (the internal reqwest - * `Client::timeout`). + * `Client::timeout`). Does not bound Volume storage requests or configure + * their inactivity limit. Storage connection setup has a separate timeout. * * Omitted ⇒ kernel default (120 000 ms / 120 s). Napi-rs * serialises `u32` as JS `number`; the largest representable value @@ -1325,3 +1330,9 @@ export interface TypedValueInput { * loaded?" probe for the JS-side loader's structured diagnostics. */ export declare function version(): string + +export declare const enum VolumeCompletion { + NotVolume = 'not_volume', + NeedsInputStream = 'needs_input_stream', + Completed = 'completed' +} diff --git a/native/kernel/index.js b/native/kernel/index.js index 6902ddee..511f20a1 100644 --- a/native/kernel/index.js +++ b/native/kernel/index.js @@ -711,3 +711,4 @@ module.exports.kernelLoggingStats = nativeBinding.kernelLoggingStats module.exports.openSession = nativeBinding.openSession module.exports.setKernelLogLevel = nativeBinding.setKernelLogLevel module.exports.version = nativeBinding.version +module.exports.VolumeCompletion = nativeBinding.VolumeCompletion diff --git a/tests/unit/DBSQLSession.test.ts b/tests/unit/DBSQLSession.test.ts index 00498c54..7c73c1c0 100644 --- a/tests/unit/DBSQLSession.test.ts +++ b/tests/unit/DBSQLSession.test.ts @@ -7,6 +7,7 @@ import Status from '../../lib/dto/Status'; import DBSQLOperation from '../../lib/DBSQLOperation'; import { PrimaryKeysRequest } from '../../lib/contracts/IDBSQLSession'; import ISessionBackend from '../../lib/contracts/ISessionBackend'; +import IOperationBackend from '../../lib/contracts/IOperationBackend'; import ParameterError from '../../lib/errors/ParameterError'; import { DBSQLParameter } from '../../lib/DBSQLParameter'; import { TSessionHandle, TProtocolVersion, TOperationState, TStatusCode } from '../../thrift/TCLIService_types'; @@ -883,3 +884,18 @@ describe('DBSQLSession', () => { }); }); }); + +describe('DBSQLSession native staging dispatch', () => { + for (const configured of [false, true]) { + it(`calls native completion only when local paths are ${configured ? 'configured' : 'omitted'}`, async () => { + const completeStagingOperation = sinon.stub().resolves(); + const op = { id: 'volume-operation', completeStagingOperation } as unknown as IOperationBackend; + const session = new DBSQLSession({ + backend: { id: 'volume-session', executeStatement: async () => op } as unknown as ISessionBackend, + context: new ClientContextStub(), + }); + await session.executeStatement('REMOVE remote', configured ? { stagingAllowedLocalPath: [] } : {}); + expect(completeStagingOperation.callCount).to.equal(configured ? 1 : 0); + }); + } +}); diff --git a/tests/unit/kernel/KernelOperationBackend.test.ts b/tests/unit/kernel/KernelOperationBackend.test.ts index 972de4fc..1a0da3b3 100644 --- a/tests/unit/kernel/KernelOperationBackend.test.ts +++ b/tests/unit/kernel/KernelOperationBackend.test.ts @@ -40,6 +40,8 @@ import { import KernelOperationBackend from '../../../lib/kernel/KernelOperationBackend'; import { TTypeId } from '../../../thrift/TCLIService_types'; import ClientContextStub from '../.stubs/ClientContextStub'; +import ParameterError from '../../../lib/errors/ParameterError'; +import StagingError from '../../../lib/errors/StagingError'; // Minimal stub of the napi `Statement` surface that emits a precomputed // Arrow IPC payload per `fetchNextBatch()` call. Used to feed @@ -389,3 +391,85 @@ describe('KernelOperationBackend — M0 datatype round-trip via napi → ArrowRe expect(stub.closed).to.equal(true); }); }); + +describe('KernelOperationBackend Volume completion', () => { + it('preserves SQL execution errors with HTTP metadata', async () => { + const error = Object.assign(new ParameterError('invalid SQL parameter'), { + sqlState: '22023', + kernelMetadata: { httpStatus: 400 }, + }); + const backend = new KernelOperationBackend({ + asyncStatement: { + awaitResult: async () => { + throw error; + }, + } as any, + context: new ClientContextStub(), + }); + const actual = await backend.completeStagingOperation(['/allowed']).then( + () => expect.fail('execution should fail'), + (failure) => failure, + ); + expect(actual).to.equal(error); + expect(actual.sqlState).to.equal('22023'); + }); + + for (const { error, staging } of [ + { error: new ParameterError('path is not allowed'), staging: true }, + { + error: Object.assign(new Error('storage HTTP 403'), { + kernelMetadata: { httpStatus: 403, errorCode: 'VOLUME_STORAGE_HTTP_ERROR' }, + }), + staging: true, + }, + { + error: Object.assign(new ParameterError('invalid instruction request'), { + sqlState: '22023', + kernelMetadata: { httpStatus: 400 }, + }), + staging: false, + }, + ]) { + it(`maps completion failure: ${error.message}`, async () => { + const schema = new Schema([withTypeName(new Field('x', new Int32(), true), 'INT')]); + const statement = new StatementStub(ipcSchemaOnly(schema), []); + Object.assign(statement, { + completeVolumeOperation: async () => { + throw error; + }, + }); + const backend = new KernelOperationBackend({ statement, context: new ClientContextStub() }); + const actual = await backend.completeStagingOperation(['/allowed']).then( + () => expect.fail('completion should fail'), + (failure) => failure, + ); + if (staging) { + expect(actual).to.be.instanceOf(StagingError).with.property('message', error.message); + } else { + expect(actual).to.equal(error); + if ('sqlState' in error) expect(actual.sqlState).to.equal(error.sqlState); + if ('kernelMetadata' in error) expect(actual.kernelMetadata).to.equal(error.kernelMetadata); + } + }); + } + + for (const outcome of ['completed', 'not_volume']) { + it(`${ + outcome === 'completed' ? 'consumes completed instructions' : 'preserves ordinary result rows' + }`, async () => { + const schema = new Schema([withTypeName(new Field('x', new Int32(), true), 'INT')]); + const stub = new StatementStub(ipcSchemaOnly(schema), [ipcFromColumns(schema, { x: [1] })]); + const completeVolumeOperation = async (paths: string[], remove404: boolean) => { + expect(paths).to.deep.equal(['/allowed']); + expect(remove404).to.equal(true); + return outcome; + }; + Object.assign(stub, { isVolumeOperation: outcome === 'completed', completeVolumeOperation }); + const backend = new KernelOperationBackend({ statement: stub, context: new ClientContextStub() }); + expect((await backend.getResultMetadata()).isStagingOperation).to.equal(outcome === 'completed'); + await backend.completeStagingOperation(['/allowed']); + expect((await backend.fetchChunk({ limit: 100 })).length).to.equal(outcome === 'completed' ? 0 : 1); + expect(await backend.hasMore()).to.equal(false); + }); + } +}); diff --git a/tests/unit/kernel/error-mapping.test.ts b/tests/unit/kernel/error-mapping.test.ts index fefa8f2a..6a1506c8 100644 --- a/tests/unit/kernel/error-mapping.test.ts +++ b/tests/unit/kernel/error-mapping.test.ts @@ -372,3 +372,32 @@ describe('KernelErrorMapping.decodeNapiKernelError — cause chain', () => { expect(err.cause).to.equal(undefined); }); }); + +for (const [raw, kind, expected] of [ + [2, 'NotFound', 'ENOENT'], + [undefined, 'PermissionDenied', 'EACCES'], + [undefined, 'Unrecognized', 'EIO'], + [1000000, 'NotFound', 'ENOENT'], +] as const) { + it(`preserves local file error code and path for ${kind}/${raw}`, () => { + const error = decodeNapiKernelError( + new Error( + `__databricks_error__:${JSON.stringify({ + code: 'NotFound', + message: 'Local file does not exist', + localOsError: raw, + localErrorKind: kind, + localPath: '/allowed/missing', + sqlState: 'HY000', + queryId: 'query-id', + })}`, + ), + ) as NodeJS.ErrnoException; + expect(error.code).to.equal(expected); + expect(error.path).to.equal('/allowed/missing'); + expect(error.errno).to.be.lessThan(0); + expect(error).to.have.property('kernelMetadata').with.property('queryId', 'query-id'); + expect(error).to.have.property('cause').that.is.instanceOf(Error); + expect(error).not.to.have.property('sqlState'); + }); +} diff --git a/tests/unit/kernel/execution.test.ts b/tests/unit/kernel/execution.test.ts index 9aae43f5..8dd40ad9 100644 --- a/tests/unit/kernel/execution.test.ts +++ b/tests/unit/kernel/execution.test.ts @@ -24,6 +24,7 @@ import DBSQLSession from '../../../lib/DBSQLSession'; import DBSQLOperation from '../../../lib/DBSQLOperation'; import asyncDisposable from '../../../lib/asyncDisposable'; import { KernelNativeBinding, KernelConnection, KernelStatement } from '../../../lib/kernel/KernelNativeLoader'; +import { VolumeCompletion } from '../../../native/kernel'; import IClientContext, { ClientConfig } from '../../../lib/contracts/IClientContext'; import { CrossReferenceRequest, PrimaryKeysRequest } from '../../../lib/contracts/IDBSQLSession'; import IDBSQLLogger, { LogLevel } from '../../../lib/contracts/IDBSQLLogger'; @@ -41,6 +42,12 @@ import { DBSQLParameter, DBSQLParameterType } from '../../../lib/DBSQLParameter' // ----------------------------------------------------------------------------- class FakeNativeStatement implements KernelStatement { + public readonly isVolumeOperation = false; + + public async completeVolumeOperation(): Promise { + return VolumeCompletion.NotVolume; + } + public closed = false; public cancelled = false; @@ -964,6 +971,16 @@ describe('KernelSessionBackend', () => { expect(op.id).to.be.a('string').and.have.length.greaterThan(0); }); + it('retains inline results for configured async staging paths', async () => { + const connection = new FakeNativeConnection(); + const direct = sinon.spy(connection, 'executeStatementDirect'); + const submit = sinon.spy(connection, 'submitStatement'); + const session = makeSession(connection); + await session.executeStatement('REMOVE remote', { runAsync: true, stagingAllowedLocalPath: [] }); + expect(direct.calledOnce).to.equal(true); + expect(submit.called).to.equal(false); + }); + it('executeStatement (sync default) routes a still-running query through the AsyncStatement arm', async () => { // The directResults Running arm — a query that did NOT finish within the // server inline wait comes back as an AsyncStatement (poll/cancel handle). @@ -1152,13 +1169,11 @@ describe('KernelSessionBackend', () => { // Unsupported-on-kernel per-statement HINTS are NO-OPs (not errors) so Thrift- // targeted call sites are drop-in on the kernel path. They are kernel-governed - // perf/format hints (useCloudFetch / useLZ4Compression) or an unexposed feature - // (staging); ignoring them can't change results. (Parameter binding is NOT here + // perf/format hints (useCloudFetch / useLZ4Compression). (Parameter binding is NOT here // — a dropped param would change results, so it still throws; asserted above.) for (const { name, options } of [ { name: 'useCloudFetch', options: { useCloudFetch: true } }, { name: 'useLZ4Compression', options: { useLZ4Compression: true } }, - { name: 'stagingAllowedLocalPath', options: { stagingAllowedLocalPath: '/tmp' } }, ] as const) { it(`executeStatement ignores unsupported ${name} (no-op, does not throw)`, async () => { const connection = new FakeNativeConnection();