Repository navigation
Retry recoverable telemetry relay transport failures - #519
Merged
Merged
Conversation
Sayan-
marked this pull request as ready for review
November 14, 2026 03:18
hiroTamada
approved these changes
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Retry telemetry relay batches after recoverable connection failures, including wrapped connection refusal, reset/abort, broken pipe, EOF, and incomplete response headers.
Why
These failures can report Temporary() as false, causing the relay to drop a batch immediately instead of retrying it.
How
Add a narrow error allowlist. Keep retry-go, full jitter, Retry-After, the 30-second total batch budget, HTTP status policy, redirect protection, and transport replay unchanged. Permanent URL and certificate failures remain final.
Context
Follow-up to #508. Delivery remains best effort: retries can duplicate records, and persistent failures or queue pressure can still lose them.
Testing & Visuals
Note
Medium Risk
Changes batch delivery semantics for telemetry export—fewer dropped batches but possible duplicates on retry—while keeping permanent misconfiguration failures final.
Overview
Telemetry relay batches now retry when the HTTP client hits connection-level failures that
net/httpoften does not classify as temporary (EOF, truncated responses, refused/reset/aborted connections, broken pipe), instead of dropping the batch immediately.relayRetryablekeeps the existing 429/502/503/504 andurl.Errortemporary policy; the new allowlist is intentionally narrow so bad URLs, TLS, and other permanent config errors still fail fast. Retry timing, jitter, Retry-After, and the batch export budget are unchanged.Tests cover successful recovery after failed dials and broken connections, no retry on permanent transport errors, shutdown during retry/backoff, client timeouts within the batch window, and dropping only after retries are exhausted.
Reviewed by Cursor Bugbot for commit 57faaec. Bugbot is set up for automated code reviews on this repo. Configure here.