Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

## Unreleased

- Thrift now honors the previously ignored per-statement `rowLimit`, sends the server cap, and enforces the total locally across fetches and all result formats (including legacy columnar overflow). `0` returns no rows, unlike JDBC's unlimited `setMaxRows(0)`; `maxRows` remains fetch-page sizing. (#623)
- Capped Thrift Arrow/CloudFetch results stop prefetching once the cap is satisfied, without fetching or downloading beyond it. With `stagingAllowedLocalPath`, Thrift sends at least one for an explicitly set, normalized `rowLimit` so the server preserves the staging instruction row, including on Arrow. The client still caps ordinary results at the requested limit and exempts internal staging instructions; an unset limit remains omitted. (#623)
- Thrift converts `rowLimit` with `Number` (`null`/`NaN` become zero) and truncates toward zero, then rejects values outside `0..2147483646`, including infinities, with `ParameterError` before sending a request. Kernel behavior is unchanged. (#623)
- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
Expand Down
16 changes: 14 additions & 2 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,21 @@ telemetry implementation, so only the options explicitly forwarded by
| `namedParameters` | `Record<string, DBSQLParameter \| DBSQLParameterValue>` | ✅ | ✅ | — | Named SQL parameters. Cannot be combined with non-empty `ordinalParameters`. Thrift requires a protocol that supports parameterized queries. |
| `ordinalParameters` | `Array<DBSQLParameter \| DBSQLParameterValue>` | ✅ | ✅ | — | Positional SQL parameters. Cannot be combined with non-empty `namedParameters`. |
| `queryTags` | `Record<string, string \| null \| undefined>` | ✅ | ✅ | — | Serialized into the per-statement `query_tags` conf overlay on both. |
| `rowLimit` | `number` | ❌ | ✅ | — | Kernel-only server-side row cap. Thrift logs and ignores it. |
| `rowLimit` | `number` | ✅ | ✅ | — | Per-statement server-side row cap, separate from `maxRows` fetch sizing. Omitted by default (no driver-imposed cap). |
| `statementConf` | `Record<string, string>` | ❌ | ✅ | — | Kernel-only per-statement Spark conf overlay. Structured `queryTags` overwrite its `query_tags` key when non-empty. |

Thrift sends `resultRowLimit` without a protocol-version gate, as JDBC and Python do.
Thrift also enforces the total cap client-side across fetches and all result formats,
including on servers that ignore this field. It applies `Number` coercion (`null`/`NaN`
become zero), truncates toward zero, then requires `0..2147483646`; out-of-range
values and infinities throw `ParameterError` before any statement request. Zero
returns no rows. Arrow/CloudFetch prefetch stops at the cap. When
`stagingAllowedLocalPath` is set, Thrift sends `Math.max(normalizedRowLimit, 1)`
for an explicitly set limit so the server preserves the staging instruction row;
an unset limit remains omitted. The Thrift client still caps ordinary results at the
requested limit and exempts internal staging instructions from that cap. Kernel
validation happens on the server; its behavior is unchanged.

## Metadata-operation request parameters

The public session metadata methods also accept request objects. These are not
Expand Down Expand Up @@ -249,6 +261,6 @@ volume operations.

Kernel-only capabilities include connection-pool sizing, proxy bypass hosts,
independent TLS hostname verification, optional U2M disk caching, and
per-statement `rowLimit`/`statementConf`.
per-statement `statementConf`.

The Kernel path remains internal and unstable; its behavior may change.
16 changes: 13 additions & 3 deletions lib/contracts/IDBSQLSession.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,19 @@ export type ExecuteStatementOptions = {
*/
queryTags?: Record<string, string | null | undefined>;
/**
* kernel-only: server-side row cap for this statement (kernel `row_limit`). The
* Thrift backend has no execute-time server cap, so this is a no-op there;
* use `maxRows` for the cross-backend client-side fetch limit.
* Per-statement server-side row cap on both backends. Omitted by default (no
* driver-imposed cap). Distinct from `maxRows`, which controls fetch-page size.
* Thrift also enforces the total cap locally across fetches and result formats.
* On Thrift, Number coercion applies (numeric strings become numbers, null/NaN
* become zero), followed by truncation toward zero. On Thrift, the resulting
* integer must be in 0..2147483646; otherwise executeStatement throws
* ParameterError before sending the request. The kernel validates on the server.
* Zero returns no rows (unlike JDBC's unlimited setMaxRows(0)). When
* stagingAllowedLocalPath is set, Thrift sends at least one for a set rowLimit
* to preserve the staging instruction row; an unset limit remains omitted.
* The Thrift client still caps ordinary results at the requested limit and exempts
* internal staging instructions from that cap.
* Thrift previously ignored this option.
*/
rowLimit?: number;
/**
Expand Down
2 changes: 1 addition & 1 deletion lib/kernel/KernelSessionBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ export default class KernelSessionBackend implements ISessionBackend {
* - `queryTimeout` → NO-OP on kernel (SQL Warehouses use `STATEMENT_TIMEOUT`);
* never forwarded to the kernel and never applied as a client-side
* deadline — see the note in `executeStatement`;
* - `rowLimit` → `rowLimit` (kernel-only server-side row cap);
* - `rowLimit` → `rowLimit` (server-side row cap);
* - `queryTags` → serialised into the conf overlay's reserved
* `query_tags` key (the same wire shape Thrift's `serializeQueryTags`
* produces), merged with any explicit `statementConf`.
Expand Down
28 changes: 22 additions & 6 deletions lib/result/ArrowResultConverter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,8 @@ export default class ArrowResultConverter implements IResultsProvider<Array<any>
// Remaining rows in current Arrow batch (not the record batch!)
private remainingRows: number = 0;

private remainingRowLimit?: number;

// This is the next (!!) record batch to be read. It is unset only in two cases:
// - prior to the first call to `fetchNext`
// - when no more data available
Expand All @@ -236,21 +238,28 @@ export default class ArrowResultConverter implements IResultsProvider<Array<any>
preserveBigNumericPrecision = false,
disableRowMaterialization = false,
rowMode = 'object',
}: { preserveBigNumericPrecision?: boolean; disableRowMaterialization?: boolean; rowMode?: RowMode } = {},
rowLimit,
}: {
preserveBigNumericPrecision?: boolean;
disableRowMaterialization?: boolean;
rowMode?: RowMode;
rowLimit?: number;
} = {},
) {
this.context = context;
this.source = source;
this.schema = getSchemaColumns(schema);
this.preserveBigNumericPrecision = preserveBigNumericPrecision;
this.disableRowMaterialization = disableRowMaterialization;
this.rowMode = rowMode;
this.remainingRowLimit = rowLimit;
if (rowMode === 'object' && !disableRowMaterialization) {
warnOnDuplicateColumnNames(context, this.schema);
}
}

public async hasMore() {
if (this.schema.length === 0) {
if (this.schema.length === 0 || this.remainingRowLimit === 0) {
return false;
}
if (this.prefetchedRecordBatch) {
Expand All @@ -260,7 +269,7 @@ export default class ArrowResultConverter implements IResultsProvider<Array<any>
}

public async fetchNext(options: ResultsProviderFetchNextOptions) {
if (this.schema.length === 0) {
if (this.schema.length === 0 || this.remainingRowLimit === 0) {
return [];
}

Expand All @@ -280,22 +289,25 @@ export default class ArrowResultConverter implements IResultsProvider<Array<any>
this.prefetchedRecordBatch = undefined;
// Get table rows, but not more than remaining count. When materialization
// is disabled, skip `toArray()`/`getRows()` and return dense `null` placeholders.
const batchRows = Math.min(batch.numRows, this.remainingRows);
const batchRows = Math.min(batch.numRows, this.remainingRows, this.remainingRowLimit ?? Infinity);
let result: Array<any>;
if (this.disableRowMaterialization) {
result = new Array(batchRows).fill(null);
} else if (this.rowMode === 'array') {
result = this.getRowArrays(batch, batchRows);
} else {
const table = new Table(batch);
result = this.getRows(table.schema, table.toArray().slice(0, this.remainingRows));
result = this.getRows(table.schema, table.toArray().slice(0, batchRows));
}

// Reduce remaining rows count by a count of rows we just processed.
// If the remaining count reached zero - we're done with current arrow
// batch, so discard the batch reader
this.remainingRows -= result.length;
if (this.remainingRows === 0) {
if (this.remainingRowLimit !== undefined) {
this.remainingRowLimit -= result.length;
}
if (this.remainingRows === 0 || this.remainingRowLimit === 0) {
this.recordBatchReader = undefined;
}

Expand All @@ -313,6 +325,10 @@ export default class ArrowResultConverter implements IResultsProvider<Array<any>
// This method pulls the next item from source if needed, initializes a record batch reader and
// gets the next item from it - until either reaches end of data or finds a non-empty record batch
private async prefetch(options: ResultsProviderFetchNextOptions) {
if (this.remainingRowLimit === 0) {
return;
}

// This loop will be executed until a next non-empty record batch is retrieved
// Another implicit loop condition (end of data) is checked in the loop body
while (!this.prefetchedRecordBatch) {
Expand Down
22 changes: 17 additions & 5 deletions lib/result/CloudFetchResultHandler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,20 @@ export default class CloudFetchResultHandler implements IResultsProvider<ArrowBa

private chunkIndex: number = 0;

private remainingRows?: number;

constructor(
context: IClientContext,
source: IResultsProvider<TRowSet | undefined>,
metadata: TGetResultSetMetadataResp,
statementId?: string,
rowLimit?: number,
) {
this.context = context;
this.source = source;
this.isLZ4Compressed = metadata.lz4Compressed ?? false;
this.statementId = statementId;
this.remainingRows = rowLimit;

if (this.isLZ4Compressed && !LZ4()) {
throw new HiveDriverError('Cannot handle LZ4 compressed result: module `lz4-napi` not installed');
Expand All @@ -55,15 +59,23 @@ export default class CloudFetchResultHandler implements IResultsProvider<ArrowBa
if (this.pendingLinks.length > 0 || this.downloadTasks.length > 0) {
return true;
}
return this.source.hasMore();
return this.remainingRows !== 0 && this.source.hasMore();
}

public async fetchNext(options: ResultsProviderFetchNextOptions) {
const data = await this.source.fetchNext(options);
if (this.remainingRows !== 0) {
const data = await this.source.fetchNext(options);

data?.resultLinks?.forEach((link) => {
this.pendingLinks.push(link);
});
for (const link of data?.resultLinks ?? []) {
if (this.remainingRows === 0) {
break;
}
this.pendingLinks.push(link);
if (this.remainingRows !== undefined) {
this.remainingRows = Math.max(0, this.remainingRows - link.rowCount.toNumber(true));
}
}
}

const clientConfig = this.context.getConfig();
const freeTaskSlotsCount = clientConfig.cloudFetchConcurrentDownloads - this.downloadTasks.length;
Expand Down
4 changes: 4 additions & 0 deletions lib/result/RowSetProvider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ export default class RowSetProvider implements IResultsProvider<TRowSet | undefi
this.returnOnlyPrefetchedResults = returnOnlyPrefetchedResults;
}

public validatePrefetchedResults(): void {
this.prefetchedResults.forEach((response) => Status.assert(response.status));
}

private processFetchResponse(response: TFetchResultsResp): TRowSet | undefined {
Status.assert(response.status);
this.fetchOrientation = TFetchOrientation.FETCH_NEXT;
Expand Down
41 changes: 38 additions & 3 deletions lib/thrift-backend/ThriftOperationBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ interface ThriftOperationBackendOptions {
functionCatalog?: string | null;
emptyResultOnNotFound?: TTableSchema;
rowMode?: RowMode;
rowLimit?: number;
stagingAllowed?: boolean;
}

async function delay(ms?: number): Promise<void> {
Expand Down Expand Up @@ -105,19 +107,27 @@ export default class ThriftOperationBackend implements IOperationBackend {

private resultHandler?: ResultSlicer<any>;

private remainingRows?: number;

private readonly stagingAllowed?: boolean;

constructor({
handle,
directResults,
context,
functionCatalog,
emptyResultOnNotFound,
rowMode,
rowLimit,
stagingAllowed,
}: ThriftOperationBackendOptions) {
this.operationHandle = handle;
this.context = context;
this.functionCatalog = functionCatalog;
this.emptyResultOnNotFound = emptyResultOnNotFound;
this.rowMode = rowMode;
this.remainingRows = rowLimit;
this.stagingAllowed = stagingAllowed;
this.metadata = directResults?.resultSetMetadata;

const useOnlyPrefetchedResults = Boolean(directResults?.closeOperation);
Expand Down Expand Up @@ -168,6 +178,11 @@ export default class ThriftOperationBackend implements IOperationBackend {
disableBuffering?: boolean;
isClosed?: () => boolean;
}): Promise<Array<object>> {
if (this.remainingRows === 0) {
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
this._data.validatePrefetchedResults();
return [];
}

const resultHandler = await this.getResultHandler();

// All the library code is Promise-based, however, since Promises are microtasks,
Expand All @@ -194,7 +209,16 @@ export default class ThriftOperationBackend implements IOperationBackend {
return [];
}

const rows = await resultHandler.fetchNext({ limit, disableBuffering });
let rows = await resultHandler.fetchNext({
limit: this.remainingRows === undefined ? limit : Math.min(Number(limit), this.remainingRows),
disableBuffering,
});
if (this.remainingRows !== undefined) {
if (rows.length > this.remainingRows) {
rows = rows.slice(0, this.remainingRows);
}
this.remainingRows -= rows.length;
}
if (this.functionCatalog === undefined) {
return rows;
}
Expand All @@ -204,6 +228,11 @@ export default class ThriftOperationBackend implements IOperationBackend {
}

public async hasMore(): Promise<boolean> {
if (this.remainingRows === 0) {
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
this._data.validatePrefetchedResults();
return false;
}

const resultHandler = await this.getResultHandler();
return resultHandler.hasMore();
}
Expand Down Expand Up @@ -292,7 +321,11 @@ export default class ThriftOperationBackend implements IOperationBackend {
}

public async getResultMetadata(): Promise<ResultMetadata> {
return this.adaptResultMetadata(await this.thriftResultMetadataResponse());
const metadata = await this.thriftResultMetadataResponse();
if (this.stagingAllowed && metadata.isStagingOperation) {
this.remainingRows = undefined;
}
return this.adaptResultMetadata(metadata);
}

public getCachedResultMetadata(): ResultMetadata | undefined {
Expand Down Expand Up @@ -386,18 +419,20 @@ export default class ThriftOperationBackend implements IOperationBackend {
preserveBigNumericPrecision: this.context.getConfig().preserveBigNumericPrecision ?? false,
disableRowMaterialization: this.context.getConfig().disableRowMaterialization ?? false,
rowMode: this.rowMode,
rowLimit: this.remainingRows,
},
);
break;
case TSparkRowSetType.URL_BASED_SET:
resultSource = new ArrowResultConverter(
this.context,
new CloudFetchResultHandler(this.context, this._data, metadata, this.id),
new CloudFetchResultHandler(this.context, this._data, metadata, this.id, this.remainingRows),
metadata,
{
preserveBigNumericPrecision: this.context.getConfig().preserveBigNumericPrecision ?? false,
disableRowMaterialization: this.context.getConfig().disableRowMaterialization ?? false,
rowMode: this.rowMode,
rowLimit: this.remainingRows,
},
);
break;
Expand Down
Loading
Loading