Skip to content

fix(ios): make BannerLayoutTests deterministic on slow runners - #160

Open
blove wants to merge 1 commit into
mainfrom
blove/banner-layout-flakes
Open

blove wants to merge 1 commit into
mainfrom
blove/banner-layout-flakes

Conversation

@blove

@blove blove commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

On #148 (a Rust-only hub change), the "macOS and iOS app builds" job failed twice before it passed, each time in a different BannerLayoutTests case. These changes come from reading both failed attempts' result bundles (the accessibility trees, activity timelines and screen recordings).

Causes

testBlockedMirrorDoesNotPromiseRetry (attempt 2). checkLayout judged the banner from one snapshot, taken as soon as any sync.banner existed. A detail launch shows the list first and then pushes the thread. Partway through that push, the banner already exists in accessibility but is not yet hittable, so XCTAssertEqual(banner != nil, hasBanner) failed. The tree printed in the failure message was captured later, after things settled, and shows the banner in place. I reproduced the same failure locally in testDetailBannerKeepsInlineTitleAndToolbarVisible: one banner found, one hittability snapshot that came back false, 1 of 6 runs on a loaded machine.

testBannerClearsOnHomeAndDetail (attempt 1, banner.waitForNonExistence(timeout: 15)). The recording shows the detail launch, a system notification ("Ready for Apple Intelligence"), and then no screen updates for about 34 s. After that, the recover tap landed: the button highlighted and released on the recording. The detail banner still never cleared. The fixture held the status in an App-level @State that was read only inside the deferred workspaceStatusBanner closure. The shipped app does it differently: it reads MirrorHost.syncStatus through Observation. I could not reproduce this failure locally. A 0.05 s, 0.6 s or 1.2 s press over the composer's menus always cleared the banner, so the menus are not swallowing the tap. The fix here is to make the fixture use the app's update path, and to make the test show which step failed if this ever happens again.

The same stall also tripped the relay's 30 s silence timer. The fixture's fake hub only greeted the first link and never answered pings, so "Connecting to the hub…" stayed on screen for the rest of that test. Any launch that runs past 30 s on a slow runner would show the same thing.

Changes

  • checkLayout first waits for the screen's title, then polls for a drawn and hittable banner (waitForVisible) rather than judging one snapshot. The notice check in testOtherNoticeImplementations and visibleSyncBanner use the same helper.
  • The fixture's status is an @Observable (FixtureSync), so the banner updates the way it does in the app.
  • The recover control disappears once the status has changed. The test waits for that, so a lost tap fails with its own message instead of looking like a banner that didn't clear. It then waits for every sync.banner to go, rather than one element bound by index.
  • The fake hub greets every link the relay opens and answers pings with pong, as the real hub does.
  • No timeouts were raised.

Verification

  • Baseline (unchanged tests, loaded machine): 1 failure in 18 runs, as described above.
  • After the change: the full BannerLayoutTests suite ×3 (75 of 75 passed); the three detail tests ×8 (24 of 24); those three plus testNavigateIntoThreadAndBackWithBannerPresent ×4 with 8 extra busy processes (16 of 16). All on an iPhone 17 Pro Simulator, iOS 26.5.
  • swift format lint --strict --recursive Sources Tests Apps Package.swift passes.

These are fixture and test changes only. No shipped code changes.

🤖 Generated with Claude Code

Two CI flakes on PR #148, each traced in its result bundle:

- testBlockedMirrorDoesNotPromiseRetry: checkLayout judged the banner from
  one snapshot taken as soon as any banner existed. A detail launch shows the
  list and then pushes the thread, and mid-push the banner exists but is not
  hittable, so the check read the unsettled screen. Reproduced locally, same
  signature, in testDetailBannerKeepsInlineTitleAndToolbarVisible (1 of 6
  runs under load). checkLayout now waits for the screen's title and then
  polls for a hittable banner; the notice and banner-label checks share the
  helper.
- testBannerClearsOnHomeAndDetail: the runner stalled the app for ~34 s
  after launch. The recover tap then landed (the button highlighted on the
  recording), yet the detail banner never cleared. The fixture fed the banner
  from an App-level @State read only inside the environment closure, not
  through Observation as the app does with MirrorHost. The status is now an
  @observable, the control goes once the status has changed (so a lost tap
  fails as such), and the test waits for every banner to go rather than one
  bound by index.

The same stall let the relay's 30 s silence timer drop the fixture's link,
which the fake hub never greeted again, leaving "Connecting to the hub..." on
screen. The fake hub now greets every link and answers pings, as the hub does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant