Repository navigation
Allow bounded background telemetry export during client close - #629
Draft
vuanhphung wants to merge 3 commits into
Draft
vuanhphung wants to merge 3 commits into
vuanhphung wants to merge 3 commits into
Conversation
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
vuanhphung
had a problem deploying
to
azure-prod
November 19, 2026 09:39 — with
GitHub Actions
Failure
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last Thrift client awaits its final telemetry POST during
close(): the initial live pecotesting probe measured 47.9 ms close, with 46.8 ms in/telemetry-ext. A successful local HTTP mock reproduces that wait. Kernel session close also awaits native telemetry and remains unchanged.Add opt-in
client.close({ waitForTelemetry: false })only for unmodified built-in access-token/PAT auth. At detach, copy all config values andcustomHeaders, capture auth/transport, and bound HTTP/retry work by the existing deadline. OAuth, all token adapters (including static/external tokens and federation), custom auth and a full 32-export queue retain main's awaited path and existing timeout behavior. Auth factories, connect ordering, OAuthManager and telemetry close are byte-identical to main; no authentication work is newly detached or pinned.Verification: 1,826 unit tests, 28 local live checks (both backends, no skips), lint/prettier/type-check/build, wire/value parity, nine targeted mutations and 1,003-cycle 64 MiB cleanup pass. Four real reconnect tests cover same/cross-host retries and a second drain; federation/U2M/M2M/custom providers demonstrably stay awaited. Production
lib/diff: +99/−18; overall +816/−18, mostly tests. Executed CI checks pass; Node 22's unchanged fetch-count assertion (6 versus 5) passed on one job rerun with no code change, and both main and branch pass that file locally.Five rotated fresh processes/arm, three discarded warmups, 30 measured cycles; entries are medians of process quantiles. Thrift numbers below use PAT auth.
Live PAT single-attempt runs still saw 429s (main 54 HTTP 200/111 HTTP 429; branch 47/118): these isolate response RTT, not successful ingestion. Awaited-default mock medians are 51.34 / 51.45 ms; kernel differences are noise, not a claimed gain.
The cap trades burst tail latency and 33 in-flight POSTs versus one on main (32 detached plus an awaited fallback) for lower PAT close latency, potentially increasing 429 pressure. Earlier independent review also reproduced the burst p99 cost (51.6 → 79.7 ms).
Decision: opt-in PAT only; default unchanged. Background mode abandons HTTP/retries still active at
telemetryCloseTimeoutMs(2 s default), even in a live process; main only stops waiting. Exit/freeze can lose batches, and no public hook drains detached exports. No async-close flag exists yet or is added here. Any default rollout remains the user's call and a separate follow-up: a sub-flag under thedatabricks.partnerplatform.clientConfigsFeatureFlagsSAFE batch flag, with the client option as override.PAT mock: five fresh processes/arm/case; complete bodies received / HTTP responses emitted, not production-ingestion guarantees.
process.exit(), 50 ms or 1 s (each)process.exit(), 5 sAt 5 s RTT, natural process lifetime is ~5.2 s on main versus ~2.2 s in PAT background mode. Both transmitted bodies; the missing response is unacknowledged completion, not proven ingestion loss. A failing federation endpoint (200 ms/503), five fresh processes/arm, gives ~3.61 s natural exit for main, branch default and opt-in fallback, two exchanges before close and none after: the original wait-only auth timeout behavior is preserved, not newly bounded or extended.
The in-process delivery risk also appears under throttling: the independent review's live default-retry run saw 94 HTTP 200 / 397 HTTP 429 on main versus 57 / 518 in background mode, over 165 cycles/arm. Disabling retries made successful-response counts similar (30 versus 26). Faster close also increases request cadence, so this is not a controlled production-ingestion loss rate; it is fewer acknowledged exports, and deadline cancellation can lose slow/retrying batches without any process exit.
Kernel follow-up: native
Session::close()awaits DeleteSession thentelemetry.finish()(session-close medians 93.6 / 92.6 ms; JS close ~0.03 ms). Potential one-RTT saving of 35–50 ms is an estimate, not isolated; no kernel change here. Overlap: #606 edits Azure OAuthManager; this PR now has no auth-file/connect diff, and OAuth stays awaited.Adversarial review: round-2 scoped remedies submitted as e4502c4; awaiting the orchestrator's final assessment. Keep draft.
This pull request and its description were written by Isaac.