Skip to content

feat: report unresolved local flags on evaluate_flags snapshots - #1133

Merged
gustavohstrassburger merged 5 commits into
mainfrom
posthog/unresolved-local-flags-v2
Oct 28, 2026
Merged

gustavohstrassburger merged 5 commits into
mainfrom
posthog/unresolved-local-flags-v2

Conversation

@gustavohstrassburger

@gustavohstrassburger gustavohstrassburger commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Why: sdk-specs #85 defines how a server SDK reports flags that local evaluation could not resolve. Today such a flag is absent from the evaluate_flags() snapshot, so it looks the same as a flag that does not exist, and $feature_flag_called reports flag_missing for both. With only_evaluate_locally=True this is permanent. posthog-node already implements this (PostHog/posthog-js#5234).

This replaces #1033, which was closed as stale. It is rebased on main and adds the spec parts that #1033 did not have.

Changes:

  • FeatureFlagEvaluations.unresolved_flags maps each unresolved flag to an UnresolvedFlagReason (experience_continuity, unsupported_definition, unresolved_dependency, missing_context). When several causes apply, the spec's fixed precedence picks one. Snapshots from only() / only_accessed() have no entries, and reading the map is not an access.
  • A snapshot read of an unresolved flag reports local_evaluation_inconclusive instead of flag_missing. Values, keys and capture enrichment do not change.
  • Every raise site carries an explicit reason, including the regex fallback sites. A bad person value is missing_context; a bad filter value is unsupported_definition.
  • When definitions load, the SDK logs a warning (WARNING level) for flags local evaluation can never resolve. Each flag warns once until its definition changes; 304 and unchanged cache loads do not repeat it.
  • AsyncClient evaluates only remotely, so its snapshots have no unresolved flags. This is documented.

Behavior changes (in the changeset):

  • A condition no longer stops at the first inconclusive filter. If another filter does not match, the condition resolves locally as no match instead of falling back to /flags. A static cohort still forces the fallback only when it is the first filter in its condition that cannot be checked, as in posthog-node.
  • An inactive flag with experience continuity now resolves to false locally.
  • A flag that depends on a flag with experience continuity no longer resolves locally.

Divergences from the spec or posthog-node:

  • A missing group key still resolves to false, as before, so it is not reported as missing_context. posthog-node does the same.
  • Regex deferrals that depend on the person value report unsupported_definition but are not in the load-time warning, because the definition alone does not show them.

💚 How did you test it?

Unit tests mirror every scenario tagged @unresolved_flags_capable and @unresolvable_flag_warning_capable in acceptance/public/evaluate-flags.feature, including the device-id row, plus precedence, the condition loop, dependencies, warning pruning and unexpected errors. Run locally:

  • pytest --timeout=30. Live AI-provider tests were run with the provider API keys unset.
  • ruff format --check ., ruff check ., mypy with the baseline
  • make public_api_check

No manual testing in an app.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Written with Claude Code in PostHog Desktop, from a cross-SDK plan to apply sdk-specs chore: Remove upper bound #85 to all server SDKs.
  • Started from the feat: report unresolved local flags on evaluate_flags snapshots #1033 commit and resolved its conflicts with main. Then added reasons for the newer regex raise sites, the spec's reason precedence (so a later server error cannot overwrite a higher reason), the condition loop change, dependency checks, the unsupported_definition warning and per-flag pruning of the warning history.

Created with PostHog Desktop

🤖 Generated with Claude Code

Implements the "Snapshot unresolved flag reporting" and "Warning for flags local evaluation never resolves" requirements of the evaluate-flags spec (PostHog/sdk-specs#85).

- Add `FeatureFlagEvaluations.unresolved_flags`, with an `UnresolvedFlagReason` for each flag and the spec's fixed precedence.
- Report `local_evaluation_inconclusive` instead of `flag_missing` when a snapshot read hits an unresolved flag.
- Do not stop a condition at the first inconclusive filter, so a filter that does not match resolves the condition locally.
- Check `active` before experience continuity, also for dependencies.
- Log one warning per flag that local evaluation can never resolve, repeated only when its definition changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 39793b31-809e-4689-a5d8-b4359cbb22c1
@gustavohstrassburger gustavohstrassburger self-assigned this Oct 27, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 5572e1c: 3 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread posthog/feature_flags.py
Comment thread posthog/client.py
Comment thread posthog/client.py Outdated
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

posthog-python Compliance Report

Date: 2026-10-28T20:00:33.406868+00:00
Duration: 259408ms

✅ All Tests Passed!

121/121 tests passed


Capture_V1 Tests

✅ 95/95 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint ✅ 515ms
Endpoint And Method.Does Not Use Legacy Endpoints ✅ 510ms
Required Headers.Has Authorization Bearer Header ✅ 509ms
Required Headers.Has Content Type Json ✅ 509ms
Required Headers.Has Posthog Sdk Info Format ✅ 508ms
Required Headers.Has Posthog Attempt Header ✅ 509ms
Required Headers.Has Posthog Request Id ✅ 508ms
Required Headers.Has Posthog Request Timestamp ✅ 509ms
Required Headers.Has User Agent ✅ 508ms
Body Format.Body Has Created At And Batch ✅ 508ms
Body Format.No Api Key In Body ✅ 509ms
Body Format.No Sent At In Body ✅ 508ms
Event Format.Event Has Required Root Fields ✅ 508ms
Event Format.Event Uuid Is Valid ✅ 509ms
Event Format.Event Timestamp Is Rfc3339 ✅ 508ms
Event Format.Non Utc Event Timestamp Is Converted To Utc ✅ 518ms
Event Format.Distinct Id Is String ✅ 509ms
Event Format.Distinct Id At Root Not Properties ✅ 508ms
Event Format.Custom Properties Preserved ✅ 509ms
Event Format.Set Properties Preserved ✅ 509ms
Event Format.Set Once Properties Preserved ✅ 508ms
Event Format.Groups Properties Preserved ✅ 509ms
Event Format.Sdk Generates Uuid If Not Provided ✅ 508ms
Event Format.Event Has Required Root Fields Batch ✅ 511ms
Event Format.Event Uuid Is Valid Batch ✅ 511ms
Event Format.Event Timestamp Is Rfc3339 Batch ✅ 512ms
Event Format.Distinct Id Is String Batch ✅ 511ms
Event Format.Distinct Id At Root Not Properties Batch ✅ 511ms
Event Format.Custom Properties Preserved Batch ✅ 511ms
Event Format.Set Properties Preserved Batch ✅ 511ms
Event Format.Set Once Properties Preserved Batch ✅ 511ms
Event Format.Groups Properties Preserved Batch ✅ 511ms
Event Format.Sdk Generates Uuid If Not Provided Batch ✅ 512ms
Batch Behavior.Multiple Events In Single Batch ✅ 515ms
Batch Behavior.Batch Envelope Smoke ✅ 527ms
Batch Behavior.Flush With No Events Sends Nothing ✅ 506ms
Batch Behavior.Flush At Triggers Batch ✅ 1010ms
Batch Behavior.Created At Reflects Batch Creation Time ✅ 509ms
Deduplication.Generates Unique Uuids ✅ 514ms
Deduplication.Different Events Same Content Different Uuids ✅ 511ms
Deduplication.Preserves Uuid On Retry ✅ 6517ms
Deduplication.Preserves Timestamp On Retry ✅ 6514ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 6516ms
Deduplication.No Duplicate Events In Batch ✅ 515ms
Header Behavior On Retry.Attempt Header Starts At One ✅ 510ms
Header Behavior On Retry.Attempt Header Increments On Retry ✅ 13518ms
Header Behavior On Retry.Request Id Preserved On Retry ✅ 6518ms
Header Behavior On Retry.Different Requests Have Different Request Ids ✅ 3017ms
Header Behavior On Retry.Request Timestamp Changes On Retry ✅ 6518ms
Response Format Validation.Success Response Has Uuid Keyed Results ✅ 511ms
Response Format Validation.Success Response Has Ok For Each Event ✅ 523ms
Response Format Validation.Success No Retry After When All Ok ✅ 511ms
Response Format Validation.Success Retry After Present When Retry Events ✅ 1513ms
Response Format Validation.Success No Retry After When Drop Only ✅ 511ms
Response Format Validation.Response Echoes Request Id ✅ 509ms
Retry Behavior.Retries On 408 ✅ 6512ms
Retry Behavior.Retries On 500 ✅ 6518ms
Retry Behavior.Retries On 503 ✅ 8522ms
Retry Behavior.Retries On 504 ✅ 6517ms
Retry Behavior.Retryable Errors Have Retry After ✅ 3515ms
Retry Behavior.Respects Retry After On Retryable Error ✅ 11530ms
Retry Behavior.Does Not Retry On 400 ✅ 2512ms
Retry Behavior.Does Not Retry On 401 ✅ 2510ms
Retry Behavior.Does Not Retry On 402 ✅ 2512ms
Retry Behavior.Does Not Retry On 413 ✅ 2509ms
Retry Behavior.Does Not Retry On 415 ✅ 2511ms
Retry Behavior.Non Retryable Errors Have No Retry After ✅ 2512ms
Retry Behavior.Implements Backoff ✅ 22533ms
Retry Behavior.Max Retries Respected ✅ 22533ms
Partial Batch Handling.Handles 200 Full Success ✅ 2511ms
Partial Batch Handling.Handles 200 With All Ok ✅ 3514ms
Partial Batch Handling.Does Not Retry Dropped Events ✅ 3511ms
Partial Batch Handling.Does Not Retry Limited Events ✅ 3513ms
Partial Batch Handling.Prunes Ok Events On Partial Retry ✅ 6516ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry ✅ 6519ms
Partial Batch Handling.Retries Only Retry Events From Partial ✅ 6521ms
Partial Batch Handling.Partial Retry Preserves Uuids ✅ 6515ms
Partial Batch Handling.Partial Retry Attempt Header Increments ✅ 6515ms
Partial Batch Handling.Partial Retry Request Id Preserved ✅ 6518ms
Partial Batch Handling.Respects Retry After On Partial ✅ 8518ms
Partial Batch Handling.Unknown Result Treated As Terminal ✅ 3514ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry ✅ 3514ms
Compression.Sends Gzip Content Encoding ✅ 509ms
Compression.No Content Encoding When Disabled ✅ 509ms
Compression.Compressed Body Is Decompressible ✅ 508ms
Error Handling.Does Not Retry On Unknown 4Xx ✅ 2511ms
Event Options.Cookieless Mode Override ✅ 509ms
Event Options.Disable Skew Correction Override ✅ 508ms
Event Options.Process Person Profile Override ✅ 508ms
Event Options.Product Tour Id Override ✅ 509ms
Event Options.Unset Options Omitted ✅ 509ms
Event Options.Options Override In Batch ✅ 511ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties ✅ 508ms
Geoip And Historical Migration.Historical Migration Set In Body ✅ 508ms
Geoip And Historical Migration.Historical Migration Absent By Default ✅ 508ms

Capture_Ai Tests

✅ 5/5 tests passed

View Details
Test Status Duration
Routing.Capture Ai Posts To Ai Endpoint ✅ 508ms
Routing.Capture Does Not Reroute Ai Named Events ✅ 509ms
Identity.Capture Ai Event Has Uuid ✅ 509ms
Identity.Capture Ai Keeps Supplied Uuid ✅ 508ms
Timestamp Format.Non Utc Event Timestamp Is Converted To Utc ✅ 508ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 9ms
Request Payload.Flags Request Uses V2 Query Param ✅ 7ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 6ms
Request Payload.Flags Request Omits Authorization Header ✅ 10ms
Request Payload.Token In Flags Body Matches Init ✅ 7ms
Request Payload.Groups Round Trip ✅ 6ms
Request Payload.Groups Default To Empty Object ✅ 7ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 7ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 7ms
Request Lifecycle.No Flags Request On Init Alone ✅ 2ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 507ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 12ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 7ms
Retry Behavior.Retries Flags On 502 ✅ 309ms
Retry Behavior.Retries Flags On 504 ✅ 310ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 509ms

Feature_Flags_Local_Evaluation Tests

✅ 4/4 tests passed

View Details
Test Status Duration
Versioned Boolean Matching.Matching Version Missing ✅ 49ms
Versioned Boolean Matching.Matching Version 1 ✅ 46ms
Versioned Boolean Matching.Matching Version 2 ✅ 47ms
Versioned Boolean Matching.Version Only Reload 1 2 1 2 Missing ✅ 24ms

@gustavohstrassburger
gustavohstrassburger marked this pull request as ready for review October 28, 2026 16:18
@gustavohstrassburger
gustavohstrassburger requested a review from a team as a code owner October 28, 2026 16:18

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 1bbd5de: no findings. This pull request is large, so the review ran in 4 parts with less depth than usual.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added lock-ownership regression test behaves as intended. The known diagnostic-precedence limitation remains a cross-SDK follow-up, as discussed.

AI-assisted review.

@gustavohstrassburger
gustavohstrassburger merged commit bb2b9d6 into main Oct 28, 2026
43 checks passed
@gustavohstrassburger
gustavohstrassburger deleted the posthog/unresolved-local-flags-v2 branch October 28, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants