Skip to content

[AI-FSSDK] [FSSDK-12958] Continue rule evaluation after local holdout hit - #525

Open
alexjoeyyong wants to merge 3 commits into
masterfrom
ai/alexjoeyyong/FSSDK-12958-local-holdout-flow-fix
Open

alexjoeyyong wants to merge 3 commits into
masterfrom
ai/alexjoeyyong/FSSDK-12958-local-holdout-flow-fix

Conversation

@alexjoeyyong

@alexjoeyyong alexjoeyyong commented Oct 13, 2026 •

Copy link
Copy Markdown

Summary

A local holdout hit used to return the holdout variation and stop evaluating the flag. Now it excludes the user from that rule only: the hit is recorded, the rule is skipped, and evaluation continues with the next rule, so the user can still be served by a later rule.

Changes

  • Local holdout hits on experiment and delivery rules skip that rule and continue to the next rule instead of returning
  • Each local holdout hit sends one holdout impression in the existing format, regardless of sendFlagDecisions
  • Local hits reuse the existing mechanism for global holdouts bypassed by excludeTargetedDeliveries
  • The final decision and its events are unchanged; global holdouts are unchanged

Jira Ticket

FSSDK-12958

Notes

  • Legacy APIs (is_feature_enabled, get_feature_variable_*) now skip a locally held-out rule without sending a holdout impression, matching how they already treat excludeTargetedDeliveries holdouts

@muzahidul-opti muzahidul-opti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I checked it against the [SPEC] in FSSDK-12958:

  • Experiment and delivery rules are both skipped on a local holdout hit, and evaluation moves on to the next rule.
  • If several holdouts cover one rule, the first match wins. Each hit sends one holdout event in the existing format, even when sendFlagDecisions is off.
  • The order of checks per rule is kept (forced decision → local holdout → normal evaluation), and a holdout hit beats a stored variation.
  • Nothing new is added to OptimizelyDecision or the notification. Global holdouts are unchanged. The older APIs skip the rule without sending an event.
  • All 4 required test scenarios are covered, plus tests for the stored variation and the older APIs. The full suite passes locally (1041 tests).

I like that it reuses the existing excludeTargetedDeliveries event code instead of adding a second one.

A few minor points:

  1. Error path drops holdout events. If rule A is held out and rule B then returns an error (e.g. CMAB), the early return at decision_service.py:826 doesn't include holdout_decisions, so rule A's event is never sent. Global holdout events were already dropped here before this PR, so it's an existing gap. Should we attach holdout_decisions here, or leave it as out of scope?
  2. DISABLE_DECISION_EVENT also blocks holdout events. That seems right to me, but the spec doesn't say so. Let's confirm so all 7 SDKs behave the same way.
  3. decision_event_dispatched is True in the notification when only holdout events were sent. For example, when the final decision comes from a delivery rule with sendFlagDecisions off. The excludeTargetedDeliveries case already does this, so it's consistent. I'm just noting it for the other SDKs.

@jaeopt jaeopt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants