Skip to content

fix(agent): a group member's holder appends facts through a scheduler stop; fence clock never pauses (#271) - #273

Merged
el-bri-ga merged 3 commits into
mainfrom
fix/stale-facts-271b
Nov 19, 2026
Merged

el-bri-ga merged 3 commits into
mainfrom
fix/stale-facts-271b

Conversation

@el-bri-ga

@el-bri-ga el-bri-ga commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #271

What

#251 (#270) paused a node's fence clock over a confirmed scheduler stop, and the scheduler pauses its forget clock only over a gap with no facts from anyone. A node with a stale watch keeps appending facts through the stop, so on restart there is no gap: a holder that went quiet during the stop is counted silent and forgotten while it still runs its group members, and the group can run split.

The fix uses the holder's own evidence. A node holding a group member appends facts whether or not a scheduler is wanted, and the fence clock never pauses. So the only node the scheduler can forget while it runs a member is one whose facts are not landing, and that node fences (releases its members) first.

Invariant and entry points

Invariant: a node runs a group member only while its own facts land within the fence bound, which is under half the forget window.

Path Where How it is covered Test
Facts writer stops on a scheduler stop crates/agent/src/assignments.rs:498 write_facts, :548 writes_facts writes while the scheduler is wanted OR a group member is held or waiting to be claimed unit a_node_holding_a_group_member_writes_facts_through_a_scheduler_stop
Member count feeding the writer crates/agent/src/agent.rs:379 GroupMember, taken at :936 at the top of own(), before any claim, dropped when the task ends counts a group member's task for its whole life, held or not; a spec change restarts the task unit a_group_member_keeps_the_node_writing_facts_whether_held_or_waiting (revert-checked)
Fence clock crates/agent/src/agent.rs:751 fence_self pause and the scheduler_stopped read removed; only a landed fact (:733 -> facts_due_from :238) moves the clock existing a_node_whose_facts_stop_landing_fences_itself_and_must_be_relisted
Fence readers agent.rs:1038 (claim refused), :1344 (wait to claim), :1810 (held member released) a fence lifts only on a landed fact; a waiting member keeps facts flowing, so a fence does not latch with no scheduler covered by the count above
Scheduler forget crates/core/src/schedule.rs:360 forget_silent, :374 pause_over_gap unchanged; the pause only delays a forget, never causes one #270 tests
End to end crates/felixd/tests/scheduler.rs a stale node appends through a stop past the forget window; the group never runs off its record a_stale_node_s_facts_during_a_scheduler_stop_do_not_forget_a_group_s_holders

TLA: new StopPause constant and paused variable with a Pause(n) action (the #270 behaviour). FelixdPlacementForgetStopPause.cfg sets it and is expected to violate GroupTogether (twins job and check_tla.sh); every other config sets StopPause = FALSE.

Design calls

  1. A holder writes facts through a scheduler stop instead of any pause logic. A stale watch on any node makes a pause unmatchable on the scheduler side, so the holder's own facts are the only sound evidence.
  2. Cost: facts during a scheduler stop (or with no scheduler deployed) from nodes with a group member to hold or claim, about 2,900 a day per node at the default interval. The scheduler trims them once it is back; with no scheduler nothing trims them, filed as sys.facts is never trimmed when no scheduler is deployed #274.
  3. A node with no group member at all goes quiet in a stop and fences after the bound (debug log, not warn); it has nothing to release. Once a member appears it appends again and the fence lifts on the first landed fact. A member the record places on it is claimed there only after the scheduler answers its new relist nonce, as for a restarted node.
  4. The integration test fakes the stale node by appending its facts directly; its id sorts first, so a forget would move the group onto it.
  5. TLA: Pause(n) models the fix(agent): fence a node whose facts stop landing; no forget over a fact gap (#251) #270 behaviour as a free action, so ForgetStopPause shows a pause unmatched by the scheduler breaks GroupTogether. The spec does not model the writer rule itself; with StopPause = FALSE it keeps ForgetLive's assumption that a live node is forgotten only once fenced, which the writer rule is what makes true in code.

Revert check

Draft #272 runs only the new integration test with the fix disabled (writes_facts back to the scheduler watch, fence_self pause restored). Result noted below once CI reports.

Local checks

cargo test -p felixd-agent --lib (210 passed), clippy --workspace --all-targets --all-features -D warnings, fmt. TLC runs in CI.

…ng facts, so a fence lifts on a landed fact (#271)

Review: with no scheduler wanted and no member held, the writer stopped and
the fence latched, so a group member could never be claimed. Count a group
member's task for its whole life instead of only while held. Docs follow;
sys.facts growth with no scheduler is #274.
@el-bri-ga

Copy link
Copy Markdown
Contributor Author

Revert check (#272, fix disabled: writes_facts back to the scheduler watch alone, fence_self pause restored): a_stale_node_s_facts_during_a_scheduler_stop_do_not_forget_a_group_s_holders FAILED at scheduler.rs:1092, sx runs on Some("g11-b") while the record places it on Some("g11-0stale") (run 44383923864). #272 closed, branch deleted.

@el-bri-ga
el-bri-ga enabled auto-merge (squash) November 18, 2026 20:36
@el-bri-ga
el-bri-ga disabled auto-merge November 18, 2026 22:18
…eduler to step a stop (#271)

Every node with a group member task appends facts through the stop, so
the restarted scheduler has nothing to change and, short of a 10 minute
table tick, outputs nothing. Waiting for its output count to grow passed
only when it moved the group, the bug the test checks. A fact carrying a
relist nonce, appended after the stop's facts, gets an answer once the
scheduler has stepped all of them.
@sonarqubecloud

Copy link
Copy Markdown

@el-bri-ga
el-bri-ga enabled auto-merge (squash) November 18, 2026 23:27
@el-bri-ga
el-bri-ga merged commit b8b95b7 into main Nov 19, 2026
37 checks passed
@el-bri-ga
el-bri-ga deleted the fix/stale-facts-271b branch November 19, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Placement: a stale agent's facts during a scheduler stop defeat the gap pause

1 participant