Repository navigation
[AI-FSSDK] [FSSDK-12958] Continue rule evaluation after local holdout hit - #525
Open
alexjoeyyong wants to merge 3 commits into
Open
alexjoeyyong wants to merge 3 commits into
alexjoeyyong wants to merge 3 commits into
Conversation
muzahidul-opti
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me. I checked it against the [SPEC] in FSSDK-12958:
- Experiment and delivery rules are both skipped on a local holdout hit, and evaluation moves on to the next rule.
- If several holdouts cover one rule, the first match wins. Each hit sends one holdout event in the existing format, even when sendFlagDecisions is off.
- The order of checks per rule is kept (forced decision → local holdout → normal evaluation), and a holdout hit beats a stored variation.
- Nothing new is added to
OptimizelyDecisionor the notification. Global holdouts are unchanged. The older APIs skip the rule without sending an event. - All 4 required test scenarios are covered, plus tests for the stored variation and the older APIs. The full suite passes locally (1041 tests).
I like that it reuses the existing excludeTargetedDeliveries event code instead of adding a second one.
A few minor points:
- Error path drops holdout events. If rule A is held out and rule B then returns an error (e.g. CMAB), the early return at
decision_service.py:826doesn't includeholdout_decisions, so rule A's event is never sent. Global holdout events were already dropped here before this PR, so it's an existing gap. Should we attachholdout_decisionshere, or leave it as out of scope? DISABLE_DECISION_EVENTalso blocks holdout events. That seems right to me, but the spec doesn't say so. Let's confirm so all 7 SDKs behave the same way.decision_event_dispatchedisTruein the notification when only holdout events were sent. For example, when the final decision comes from a delivery rule with sendFlagDecisions off. The excludeTargetedDeliveries case already does this, so it's consistent. I'm just noting it for the other SDKs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A local holdout hit used to return the holdout variation and stop evaluating the flag. Now it excludes the user from that rule only: the hit is recorded, the rule is skipped, and evaluation continues with the next rule, so the user can still be served by a later rule.
Changes
Jira Ticket
FSSDK-12958
Notes