Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased

Add opt-in `client.close({ waitForTelemetry: false })` for background Thrift telemetry export with built-in access-token/PAT auth; OAuth, token-provider/federation and custom auth retain the unchanged awaited path. Background mode abandons exports and retries at `telemetryCloseTimeoutMs` (2s by default) even in a live process, exit or suspension can also lose batches, and no public hook drains detached exports.

- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
Expand Down
22 changes: 15 additions & 7 deletions lib/DBSQLClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,12 @@ import tls from 'tls';

import { EventEmitter } from 'events';
import TCLIService from '../thrift/TCLIService';
import IDBSQLClient, { ClientOptions, ConnectionOptions, OpenSessionRequest } from './contracts/IDBSQLClient';
import IDBSQLClient, {
ClientOptions,
ClientCloseOptions,
ConnectionOptions,
OpenSessionRequest,
} from './contracts/IDBSQLClient';
import IDriver from './contracts/IDriver';
import IClientContext, { ClientConfig } from './contracts/IClientContext';
import IThriftClient from './contracts/IThriftClient';
Expand Down Expand Up @@ -946,8 +951,15 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I
* telemetry buffered between flush ticks is lost if the process exits
* without calling `close()`. Long-lived applications should `await` this
* method on shutdown so the aggregator drains its remaining metrics.
* `waitForTelemetry: false` enables background Thrift export only for built-in
* access-token/PAT auth. OAuth, token-provider/federation and custom auth retain
* the awaited path, without detaching exchange or refresh work.
* Background exports and retries still active at the close deadline are abandoned
* even in a live process. Explicit exit or runtime suspension can also lose batches.
* No public hook drains previously detached batches.
*/
public async close(): Promise<void> {
public async close(options: ClientCloseOptions = {}): Promise<void> {
const waitForTelemetry = options?.waitForTelemetry !== false;
try {
await this.sessions.closeAll();
await this.backend?.close();
Expand All @@ -957,13 +969,9 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I

this.backend = undefined;

// Cleanup telemetry. Releasing our refcount on the shared TelemetryClient
// is awaited because the underlying close() drains the final HTTP POST —
// a caller doing `await client.close(); process.exit(0)` would otherwise
// truncate the in-flight request when this is the last refcount holder.
if (this.host && this.telemetryClient) {
try {
await TelemetryClientProvider.getInstance().releaseClient(this, this.host);
await TelemetryClientProvider.getInstance().releaseClient(this, this.host, waitForTelemetry);
} catch (error: any) {
this.logger.log(LogLevel.debug, `Telemetry cleanup error: ${error.message}`);
}
Expand Down
17 changes: 16 additions & 1 deletion lib/contracts/IDBSQLClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,21 @@ export interface ClientOptions {
logger?: IDBSQLLogger;
}

export interface ClientCloseOptions {
/**
* Wait for the final Thrift telemetry export (default true). With false,
* built-in access-token/PAT auth exports in the background up to
* telemetryCloseTimeoutMs (2s by default). All other auth providers (OAuth,
* token adapters/federation, custom) and a full queue retain the awaited path; no
* exchange or refresh work is detached. Background exports and retries
* still active at the deadline are abandoned even in a live process. Explicit
* process.exit() or runtime suspension can also lose batches. No public hook
* drains previously detached batches. Native kernel session cleanup, including
* its telemetry flush, is always awaited.
*/
waitForTelemetry?: boolean;
}

type AuthOptions =
| {
authType?: 'access-token';
Expand Down Expand Up @@ -319,5 +334,5 @@ export default interface IDBSQLClient {

openSession(request?: OpenSessionRequest): Promise<IDBSQLSession>;

close(): Promise<void>;
close(options?: ClientCloseOptions): Promise<void>;
}
1 change: 1 addition & 0 deletions lib/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import { formatProgress } from './utils';
import { LogLevel } from './contracts/IDBSQLLogger';

// Re-export types for TypeScript users
export type { ClientCloseOptions } from './contracts/IDBSQLClient';
export type { default as ITokenProvider } from './connection/auth/tokenProvider/ITokenProvider';
export type { RowMode } from './contracts/IDBSQLSession';

Expand Down
41 changes: 39 additions & 2 deletions lib/telemetry/DatabricksTelemetryExporter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ import fetch, { RequestInit, Response, Request } from 'node-fetch';
import IClientContext from '../contracts/IClientContext';
import { LogLevel } from '../contracts/IDBSQLLogger';
import IAuthentication from '../connection/contracts/IAuthentication';
import IConnectionProvider from '../connection/contracts/IConnectionProvider';
import PlainHttpAuthentication from '../connection/auth/PlainHttpAuthentication';
import AuthenticationError from '../errors/AuthenticationError';
import HiveDriverError from '../errors/HiveDriverError';
import { TelemetryMetric, DEFAULT_TELEMETRY_CONFIG } from './types';
Expand Down Expand Up @@ -125,6 +127,10 @@ export default class DatabricksTelemetryExporter {

private authMissingWarned = false;

private abortController?: AbortController;

private connectionProvider?: Promise<IConnectionProvider>;

constructor(
private context: IClientContext,
private host: string,
Expand Down Expand Up @@ -157,9 +163,24 @@ export default class DatabricksTelemetryExporter {
* registry will land in the consumer-wiring PR.
*/
dispose(): void {
this.abortController?.abort();
this.circuitBreakerRegistry.removeCircuitBreaker(this.host);
}

detachContext(): boolean {
const authProvider = this.authProvider ?? this.context.getAuthProvider?.();
if (
!(authProvider instanceof PlainHttpAuthentication) ||
authProvider.authenticate !== PlainHttpAuthentication.prototype.authenticate
)
return false;
this.authProvider = authProvider;
this.abortController = new AbortController();
this.connectionProvider = this.context.getConnectionProvider();
this.connectionProvider.catch(() => {});
return true;
}

async export(metrics: TelemetryMetric[]): Promise<void> {
if (!metrics || metrics.length === 0) {
return;
Expand Down Expand Up @@ -207,10 +228,12 @@ export default class DatabricksTelemetryExporter {

/* eslint-disable no-await-in-loop */
for (let attempt = 0; attempt < totalAttempts; attempt += 1) {
if (this.abortController?.signal.aborted) return;
try {
await this.exportInternal(metrics);
return;
} catch (error: any) {
if (this.abortController?.signal.aborted) return;
lastError = error;

if (
Expand Down Expand Up @@ -299,6 +322,7 @@ export default class DatabricksTelemetryExporter {
headers,
body,
timeout: 10000,
...(this.abortController ? { signal: this.abortController.signal as unknown as RequestInit['signal'] } : {}),
});

if (!response.ok) {
Expand Down Expand Up @@ -333,7 +357,7 @@ export default class DatabricksTelemetryExporter {
}

private async sendRequest(url: string, init: RequestInit): Promise<Response> {
const connectionProvider = await this.context.getConnectionProvider();
const connectionProvider = await (this.connectionProvider ?? this.context.getConnectionProvider());
const agent = await connectionProvider.getAgent(url);
// Do NOT route through `connectionProvider.getRetryPolicy()` here. That
// policy is bounded by the customer's `retriesTimeout` (15 min default)
Expand Down Expand Up @@ -471,8 +495,21 @@ export default class DatabricksTelemetryExporter {
}

private sleep(ms: number): Promise<void> {
if (!this.abortController)
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
const { signal } = this.abortController;
return new Promise((resolve) => {
setTimeout(resolve, ms);
let timer: NodeJS.Timeout;
const finish = (): void => {
clearTimeout(timer);
signal.removeEventListener('abort', finish);
resolve();
};
timer = setTimeout(finish, ms);
if (signal.aborted) finish();
else signal.addEventListener('abort', finish, { once: true });
});
}
}
8 changes: 7 additions & 1 deletion lib/telemetry/TelemetryClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ class TelemetryClient implements IClientContext {

private readonly logger: IDBSQLLogger;

private readonly config: ClientConfig;
private config: ClientConfig;

private readonly circuitBreakerRegistry: CircuitBreakerRegistry;

Expand Down Expand Up @@ -302,6 +302,12 @@ class TelemetryClient implements IClientContext {
return this.closed;
}

detachContext(): boolean {
if (!this.exporter.detachContext()) return false;
this.config = { ...this.config, customHeaders: this.config.customHeaders && { ...this.config.customHeaders } };
return true;
}

/**
* Drain pending metrics and tear down owned resources. Called by
* `TelemetryClientProvider.releaseClient` when refCount hits zero.
Expand Down
28 changes: 21 additions & 7 deletions lib/telemetry/TelemetryClientProvider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ interface TelemetryClientHolder {
// in logs rather than silently growing the map.
const MAX_CLIENTS_SOFT_LIMIT = 128;

const MAX_PENDING_CLOSES = 32;

/**
* Process-wide registry of `TelemetryClient`s, one per host. Multiple
* `DBSQLClient` instances connecting to the same host share the same
Expand All @@ -47,6 +49,8 @@ class TelemetryClientProvider {

private clients = new Map<string, TelemetryClientHolder>();

private pendingCloses = new Set<Promise<void>>();

private softLimitWarned = false;

// Production code should use `TelemetryClientProvider.getInstance()` for
Expand Down Expand Up @@ -118,7 +122,7 @@ class TelemetryClientProvider {
* Release `context`'s registration. When the last `DBSQLClient` releases,
* the underlying `TelemetryClient.close()` runs and the entry is removed.
*/
async releaseClient(context: IClientContext, host: string): Promise<void> {
async releaseClient(context: IClientContext, host: string, waitForTelemetry: boolean = true): Promise<void> {
const logger = context.getLogger();
const key = TelemetryClientProvider.normalizeHostKey(host);
const holder = this.clients.get(key);
Expand Down Expand Up @@ -146,12 +150,22 @@ class TelemetryClientProvider {
// getOrCreateClient creates a fresh instance rather than receiving
// this closing one.
this.clients.delete(key);
try {
await holder.client.close();
logger.log(LogLevel.debug, `Closed and removed TelemetryClient for host: ${host}`);
} catch (error) {
const msg = error instanceof Error ? error.message : String(error);
logger.log(LogLevel.debug, `Error releasing TelemetryClient: ${msg}`);
const background =
waitForTelemetry === false && this.pendingCloses.size < MAX_PENDING_CLOSES && holder.client.detachContext();
const closing = (async () => {
try {
await holder.client.close();
logger.log(LogLevel.debug, `Closed and removed TelemetryClient for host: ${host}`);
} catch (error) {
const msg = error instanceof Error ? error.message : String(error);
logger.log(LogLevel.debug, `Error releasing TelemetryClient: ${msg}`);
}
})();
if (background) {
this.pendingCloses.add(closing);
closing.finally(() => this.pendingCloses.delete(closing)).catch(() => {});
} else {
await closing;
}
}
}
Expand Down
56 changes: 56 additions & 0 deletions tests/e2e/telemetry/close-e2e.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
import { expect } from 'chai';
import { DBSQLClient } from '../../../lib';
import { LogLevel } from '../../../lib/contracts/IDBSQLLogger';
import config from '../utils/config';

describe('client close telemetry lifecycle', function closeSuite() {
this.timeout(60000);

for (const useKernel of [false, true]) {
for (const waitForTelemetry of [true, false]) {
it(`${
useKernel ? 'kernel' : 'Thrift'
} closes and reopens with waitForTelemetry=${waitForTelemetry}`, async () => {
let errors = 0;
const client = new DBSQLClient({
logger: {
log(level) {
if (level === LogLevel.error) errors += 1;
},
},
});
try {
const cycle = async (): Promise<void> => {
await client.connect({ host: config.host, path: config.path, token: config.token, ...{ useKernel } });
if (!useKernel) expect(client.getTelemetryStats()).not.to.equal(undefined);
const session = await client.openSession();
const operation = await session.executeStatement(
'SELECT id, CAST(NULL AS STRING) AS missing, id % 2 = 0 AS even FROM range(12) ORDER BY id LIMIT 3',
{ maxRows: 2 },
);
const rows = (await operation.fetchAll({ maxRows: 2 })) as Array<Record<string, unknown>>;
expect(rows.map((row) => ({ ...row, id: Number(row.id) }))).to.deep.equal([
{ id: 0, missing: null, even: true },
{ id: 1, missing: null, even: false },
{ id: 2, missing: null, even: true },
]);
await operation.close();
const early = await session.executeStatement('SELECT id FROM range(100) ORDER BY id', { maxRows: 1 });
await early.fetchChunk({ maxRows: 1 });
await early.close();
const cancelled = await session.executeStatement('SELECT id FROM range(100)', { runAsync: true });
await cancelled.cancel();
await cancelled.close();
await client.close({ waitForTelemetry });
};
await cycle();
await cycle();
await cycle();
expect(errors).to.equal(0);
} finally {
await client.close();
}
});
}
}
});
Loading
Loading