Skip to content

fix: flush feature flag telemetry at invocation end - #872

Open
vjfridge wants to merge 3 commits into
DataDog:mainfrom
vjfridge:vickie/FFL-3412/lambda-telemetry-flush
Open

vjfridge wants to merge 3 commits into
DataDog:mainfrom
vjfridge:vickie/FFL-3412/lambda-telemetry-flush

Conversation

@vjfridge

@vjfridge vjfridge commented Oct 20, 2026 •

Copy link
Copy Markdown

Motivation

A customer asked about flushing feature-flag telemetry from Python Lambda functions (FFL-3412). Short invocations can return before background writers send evaluation and exposure events, leaving telemetry delayed by Lambda freezing or lost on termination.

This change automatically attempts a flush before the invocation finishes, without customers adding handler code. Providers remain available for warm invocations.

Changes

  • Flush feature-flag telemetry during invocation cleanup, including handler errors and failures in other cleanup steps.
  • Cap the wait at DD_FEATURE_FLAG_FLUSH_TIMEOUT_MS (default 500 ms) and remaining invocation time minus a 100 ms safety margin.
  • Preserve the handler's response or exception; safely skip older tracers and unused OpenFeature.
  • Add compatibility tests, configuration docs, and a smoke handler/runbook.

Testing

  • After merging main: 361 tests plus 15 subtests passed with the rebuilt candidate tracer; 360 plus 15 subtests passed with ddtrace 4.1.1 (candidate-only test skipped). Flake8 passed.
  • HTTP patch tests passed separately; running them after the wrapper tests exposed shared instrumentation state.
  • Earlier local Lambda emulator smoke tests covered Agent/direct delivery, handler errors, timeouts, and warm recovery using local receivers and pre-merge builds.
  • Still outstanding: deployed AWS freeze/thaw and Datadog ingestion validation; I don't have access to the sandbox.

Release coordination

Companion: dd-trace-py #21135. Either PR can merge first; ship the tracer capability before the Lambda library/layer containing both changes. Older tracers safely skip the flush. Delivery remains best effort; timeout stops waiting, not the underlying send.

PR checklist

Types of Changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)

Use the tracer flush contract within the remaining Lambda budget while preserving handler outcomes and warm providers.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
vjfridge and others added 2 commits October 22, 2026 23:42
Keep invalid timeout defaults numeric, initialize the smoke provider before invocation, and assert older tracer compatibility.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
Incorporate upstream Lambda v130 and instrumentation updates for FFL-3412.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
@vjfridge
vjfridge marked this pull request as ready for review October 23, 2026 02:50
@vjfridge
vjfridge requested review from a team as code owners October 23, 2026 02:50

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Static review identified no actionable defect in cleanup placement, timeout budgeting, or older-tracer compatibility. Runtime behavior and deployed AWS freeze/thaw delivery remain unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit f6f08ba

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant