Skip to content

Retry recoverable telemetry relay transport failures - #519

Merged
Sayan- merged 2 commits into
mainfrom
hypeship/relay-transport-retries
Nov 16, 2026
Merged

Sayan- merged 2 commits into
mainfrom
hypeship/relay-transport-retries

Conversation

@Sayan-

@Sayan- Sayan- commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What

Retry telemetry relay batches after recoverable connection failures, including wrapped connection refusal, reset/abort, broken pipe, EOF, and incomplete response headers.

Why

These failures can report Temporary() as false, causing the relay to drop a batch immediately instead of retrying it.

How

Add a narrow error allowlist. Keep retry-go, full jitter, Retry-After, the 30-second total batch budget, HTTP status policy, redirect protection, and transport replay unchanged. Permanent URL and certificate failures remain final.

Context

Follow-up to #508. Delivery remains best effort: retries can duplicate records, and persistent failures or queue pressure can still lose them.

Testing & Visuals

  • Regression tests failed before the fix for connection refusal, reset, EOF, and truncated headers, then passed. Local HTTP/TLS tests also cover permanent failures, per-request timeout recovery, and shutdown during requests or retry backoff.
  • Passed: go test -race ./lib/events -count=1.
  • Passed the full Go server suite excluding container e2e tests, with the race detector: go test -race $(go list ./... | rg -v '/e2e$').
  • Independent review found no actionable issues and passed: go test -race ./lib/events -run 'TestRelayStorage_' -count=5 -timeout=120s.
  • Container/image e2e and a full 30-second batch-expiration test were not run. ECONNABORTED and EPIPE have no dedicated behavioral regression cases. No screenshots needed.

Note

Medium Risk
Changes batch delivery semantics for telemetry export—fewer dropped batches but possible duplicates on retry—while keeping permanent misconfiguration failures final.

Overview
Telemetry relay batches now retry when the HTTP client hits connection-level failures that net/http often does not classify as temporary (EOF, truncated responses, refused/reset/aborted connections, broken pipe), instead of dropping the batch immediately.

relayRetryable keeps the existing 429/502/503/504 and url.Error temporary policy; the new allowlist is intentionally narrow so bad URLs, TLS, and other permanent config errors still fail fast. Retry timing, jitter, Retry-After, and the batch export budget are unchanged.

Tests cover successful recovery after failed dials and broken connections, no retry on permanent transport errors, shutdown during retry/backoff, client timeouts within the batch window, and dropping only after retries are exhausted.

Reviewed by Cursor Bugbot for commit 57faaec. Bugbot is set up for automated code reviews on this repo. Configure here.

@Sayan-
Sayan- marked this pull request as ready for review November 14, 2026 03:18
@Sayan-
Sayan- merged commit d295da9 into main Nov 16, 2026
12 checks passed
@Sayan-
Sayan- deleted the hypeship/relay-transport-retries branch November 16, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants