Skip to content

Allow bounded background telemetry export during client close - #629

Draft
vuanhphung wants to merge 3 commits into
mainfrom
perf/close-latency
Draft

vuanhphung wants to merge 3 commits into
mainfrom
perf/close-latency

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

The last Thrift client awaits its final telemetry POST during close(): the initial live pecotesting probe measured 47.9 ms close, with 46.8 ms in /telemetry-ext. A successful local HTTP mock reproduces that wait. Kernel session close also awaits native telemetry and remains unchanged.

Add opt-in client.close({ waitForTelemetry: false }) only for unmodified built-in access-token/PAT auth. At detach, copy all config values and customHeaders, capture auth/transport, and bound HTTP/retry work by the existing deadline. OAuth, all token adapters (including static/external tokens and federation), custom auth and a full 32-export queue retain main's awaited path and existing timeout behavior. Auth factories, connect ordering, OAuthManager and telemetry close are byte-identical to main; no authentication work is newly detached or pinned.

Verification: 1,826 unit tests, 28 local live checks (both backends, no skips), lint/prettier/type-check/build, wire/value parity, nine targeted mutations and 1,003-cycle 64 MiB cleanup pass. Four real reconnect tests cover same/cross-host retries and a second drain; federation/U2M/M2M/custom providers demonstrably stay awaited. Production lib/ diff: +99/−18; overall +816/−18, mostly tests. Executed CI checks pass; Node 22's unchanged fetch-count assertion (6 versus 5) passed on one job rerun with no code change, and both main and branch pass that file locally.

Five rotated fresh processes/arm, three discarded warmups, 30 measured cycles; entries are medians of process quantiles. Thrift numbers below use PAT auth.

Measurement main opt-in background
Mock 50 ms RTT close p50 / p95 51.21 / 51.76 ms 0.165 / 0.419 ms
Mock burst close p99 (32-export cap) 51.76 ms 72.07 ms
Live Thrift close p50, retries disabled 40.67 ms 0.223 ms
Live Thrift full-cycle p50, retries disabled 463.36 ms 424.11 ms
Live kernel full-cycle p50, retries disabled 412.58 ms 403.46 ms

Live PAT single-attempt runs still saw 429s (main 54 HTTP 200/111 HTTP 429; branch 47/118): these isolate response RTT, not successful ingestion. Awaited-default mock medians are 51.34 / 51.45 ms; kernel differences are noise, not a claimed gain.

The cap trades burst tail latency and 33 in-flight POSTs versus one on main (32 detached plus an awaited fallback) for lower PAT close latency, potentially increasing 429 pressure. Earlier independent review also reproduced the burst p99 cost (51.6 → 79.7 ms).

Decision: opt-in PAT only; default unchanged. Background mode abandons HTTP/retries still active at telemetryCloseTimeoutMs (2 s default), even in a live process; main only stops waiting. Exit/freeze can lose batches, and no public hook drains detached exports. No async-close flag exists yet or is added here. Any default rollout remains the user's call and a separate follow-up: a sub-flag under the databricks.partnerplatform.clientConfigsFeatureFlags SAFE batch flag, with the client option as override.

PAT mock: five fresh processes/arm/case; complete bodies received / HTTP responses emitted, not production-ingestion guarantees.

Exit / mock RTT main bodies / responses (of 5) background bodies / responses (of 5)
Immediate process.exit(), 50 ms or 1 s (each) 5 / 5 0 / 0
Immediate process.exit(), 5 s 5 / 0 0 / 0
Natural return, 50 ms or 1 s (each) 5 / 5 5 / 5
Natural return, 5 s 5 / 5 5 / 0

At 5 s RTT, natural process lifetime is ~5.2 s on main versus ~2.2 s in PAT background mode. Both transmitted bodies; the missing response is unacknowledged completion, not proven ingestion loss. A failing federation endpoint (200 ms/503), five fresh processes/arm, gives ~3.61 s natural exit for main, branch default and opt-in fallback, two exchanges before close and none after: the original wait-only auth timeout behavior is preserved, not newly bounded or extended.

The in-process delivery risk also appears under throttling: the independent review's live default-retry run saw 94 HTTP 200 / 397 HTTP 429 on main versus 57 / 518 in background mode, over 165 cycles/arm. Disabling retries made successful-response counts similar (30 versus 26). Faster close also increases request cadence, so this is not a controlled production-ingestion loss rate; it is fewer acknowledged exports, and deadline cancellation can lose slow/retrying batches without any process exit.

Kernel follow-up: native Session::close() awaits DeleteSession then telemetry.finish() (session-close medians 93.6 / 92.6 ms; JS close ~0.03 ms). Potential one-RTT saving of 35–50 ms is an estimate, not isolated; no kernel change here. Overlap: #606 edits Azure OAuthManager; this PR now has no auth-file/connect diff, and OAuth stays awaited.

Adversarial review: round-2 scoped remedies submitted as e4502c4; awaiting the orchestrator's final assessment. Keep draft.

This pull request and its description were written by Isaac.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

This branch was successfully deployed

1 active deployment
azure-prod — e4502c42 Deployed Nov 19, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #2146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant