Skip to content

Migrate the 13-case iOS gate to selector action/outcome waits - #298

Draft
piotr-iohk wants to merge 7 commits into
codex/e2e-outcome-pilotfrom
codex/e2e-action-outcome-gate
Draft

piotr-iohk wants to merge 7 commits into
codex/e2e-outcome-pilotfrom
codex/e2e-action-outcome-gate

Conversation

@piotr-iohk

@piotr-iohk piotr-iohk commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Dependency: synonymdev/bitkit-android#1808 — Android app correction for Send2 after a Lightning receipt.
Dependency: synonymdev/bitkit-android#1813 — Android RBF replacement publication before Boost success.

All thirteen active @ios_gate cases use selector action/outcome waits. Actions execute once; polling observes fresh selectors. Ordinary UI waits are 30 seconds. Existing financial, history, metadata, restore, channel and hardware lifecycle assertions, domain wait budgets and declared skips are retained. No OCR or app source changes.

Stacked on #296 (codex/e2e-outcome-pilot). Merge #296 first, then retarget to main. Keep draft: refreshed against E2E main (265a050) through pilot 96d6fca, with final gate candidate 38b1255. Android app fixes #1808 and #1813 are merged. Android Send2, Boost2 and Lightning1 passed first attempt (3/3, 144/144 outcomes) against a master build containing both merged fixes. The full iOS gate is still running. Historical comparisons below retain their original revisions. Staging funding has not been revalidated.

Six-run iOS comparison

All six candidate runs completed successfully at the same app 8c1149be80a5488ea00da159779ca12706b99b10, E2E 6de3b222fc7272087321022229763a14056c0e75 and workflow 90949e4a16d0066e8d3ce94c47b55c6cda567cce. Logs confirm Node 22.23.4, iOS 26.5 and Xcode 26.6. No source changes during the cohort. The historical baseline was not rerun.

Metric Existing six baseline runs Six action/outcome runs
First-attempt case passes 74/78 (94.9%) 77/78 (98.7%)
Whole gates clean first attempt 3/6 5/6
Case failures recovered by normal CI retries 4 1
Eventual case passes 78/78 78/78
Median first-attempt worker time 53m 42.1s 52m 29.4s
Median worker time including case retries 56m 18.4s 53m 04.9s

Observed first-worker median is 1m 12.7s / 2.3% lower; including retries, the median is 3m 13.5s / 5.7% lower. This small observational sample supports continuing the experiment; it does not establish a causal speedup or a durable lower flake rate. The failed first attempt ended its RBF case early, so first-worker timing alone is especially limited.

Candidate run First attempt Recovered cases First worker Worker including retries
42299518347 13/13 None 51m 41.7s 51m 41.7s
42370124830 13/13 None 54m 4.0s 54m 4.0s
42370157344 13/13 None 53m 17.1s 53m 17.1s
42370162355 13/13 None 53m 33.2s 53m 33.2s
42370167733 13/13 None 50m 41.4s 50m 41.4s
42370172726 12/13 boost_2 on attempt 2 49m 4.1s 52m 52.6s

The only candidate iOS failure was boost_2 in run 42370172726: BoostSuccessToast was not observed within 30s; the unchanged case passed on attempt 2. Its matching recording visibly shows the green Boosted! toast briefly, followed by a blue Transaction Replaced toast, while the final screenshot shows a Boosting transaction. This is a missed transient-success observation; the exact accessibility/poll scheduling cause remains unproven. No action was replayed within the failed attempt.

Baseline references: 42201867907, 42199157743, 42164497264, 42119619712, 42118872824, 42116447900. The first is explicitly pinned to the same app and had 11/13 first-attempt passes. Other historical app revisions vary. Baseline CI already used 30s UI waits; this is not evidence that increasing the CI timeout caused the difference.

Timings use reported worker summaries, excluding queue/build/setup time. Baseline logs do not expose exact per-case end times. Five additional candidates ran concurrently on isolated runners with separate fresh backends; runtime variation remains a caveat.

Android at frozen 6de3b22, before the fix

Pinned Android app/workflows ebe9bc09850704d1e96a2bb1697fd5d5a664fd1a, same E2E 6de3b22, 30s UI waits, Pixel 6 API 33 and Node 22. Both workflows completed with failure after their normal three-attempt retry policy. Skips are excluded from pass counts.

  • Full seven-shard local suite: 34/39 first attempt, 36/39 eventual. send_1, send_2, send_3 fail in all three attempts. All nine screenshots and usable video tails remain on the populated recipient form with Continue visible. Lightning and Transfer hit the same transition on their first attempts and recover on attempt 2. Exact click/readiness cause is unproven; this is not a MoveToSpark intro in the inspected artifacts.
  • Supplemental staging suite: 5/9 first and eventual. The five Pubky profile cases pass first attempt. Hardware pairing/receive reject the new scoped mixed-case wallet label although screenshots show E2E TREZOR; pinned Android source renders this caption uppercase. This is a test/UI-contract mismatch. An existing same-app hardware baseline passed all three hardware cases first attempt with old E2E source; no new baseline execution.
  • hardware_wallet_4 and the unchanged paykit_1 fail their initial software-wallet funding balance waits in all three attempts, before the migrated hardware flow/Paykit path. Screenshots show zero balance; deposit API calls return transaction IDs. Funding/sync root cause remains unresolved and is not attributed to the rewrite from these artifacts.

Across the thirteen migrated cases on Android: 5/13 first attempt, 7/13 eventual. The local suite covers ten; staging supplies the three hardware cases. Five declared skips across both workflows remain skipped. All five failed Android shard archives, attempt screenshots, usable recordings and device/app logs were inspected. Some hardware/Paykit recordings are header-only or cannot decode their tails; screenshots/logs remain available. These failures prompted the Android follow-up below.

Frozen 6de3b22 per-case comparison

Case Baseline iOS first passes Candidate iOS first passes Android outcome before the fix
boost_1 6/6 6/6 Pass first attempt
boost_2 5/6 5/6 Pass first attempt
hardware_wallet_1 6/6 6/6 Fail all 3: scoped wallet label
hardware_wallet_2 6/6 6/6 Fail all 3: scoped wallet label
hardware_wallet_4 6/6 6/6 Fail all 3: initial funding before migrated flow
lightning_1 5/6 6/6 Pass on attempt 2
lnurl_1 6/6 6/6 Pass first attempt
multi_address_3 4/6 6/6 Pass first attempt
multi_address_4 6/6 6/6 Pass first attempt
send_1 6/6 6/6 Fail all 3 attempts: recipient transition
send_2 6/6 6/6 Fail all 3 attempts: recipient transition
send_3 6/6 6/6 Fail all 3 attempts: recipient transition
transfer_2 6/6 6/6 Pass on attempt 2

Other validation

At frozen 6de3b22, TypeScript, scoped lint, 78 TypeScript tests and 9 Python tests passed; independent implementation review is complete. The three original implementation commits are signed and Verified. Prior local iOS gate validation was 12/13 first attempt, 13/13 after one unchanged-source multi_address_4 retry. This validation pass changes no source and retains all first failures and retries.

See docs/outcome-gate-migration.md for retained coverage and the comparison protocol. The follow-up below addresses Android recipient readiness and the hardware-name outcome. The original iOS RBF-toast observation and the additional Android issues remain relevant to adoption; this draft is not merge-ready.

Android follow-up at e560ece: completed

The fix waits for Android Recipient Continue to be enabled, with the keyboard hidden and two matching native bounds samples before one click. Invalid/stale samples reset readiness; outcome failure never replays the action. Send validation, manual invoice entry and invoice editing share this path. iOS delegates to the original tap helper. Hardware home captions are verified uppercase on Android; Settings still verifies stored mixed-case names. Existing financial, fee, history, metadata, negative-validation and hardware lifecycle assertions remain.

Targeted local-backend Android results: 12/12 eventual passes, 9/12 first attempt. All three active hardware cases pass first attempt. Logs show 28 recipient-confirmation outcomes and 7 scoped hardware-caption outcomes passed, with zero failures at either corrected boundary, including the staging label checks. This validates the specific fixes; remaining failures below prevent claiming a clean or reliably green Android gate.

Targeted cases First attempt Final result
Send1, Send3 2/2 Pass first attempt
Send2 0/1 Pass on attempt3
Hardware1/2/4, local backend 3/3 Pass first attempt
LNURL, Transfer2 2/2 Pass first attempt
Lightning1 0/1 Pass on attempt2
Security1 (co-located shard) 1/1 Pass first attempt
Boost2 0/1 Pass on attempt2
Multi-address3 1/1 Pass first attempt

Completed workflows:

All builds explicitly check out app ebe9bc09850704d1e96a2bb1697fd5d5a664fd1a; tests are frozen at e560ece2b7cc1fb5bbc1b72e8205abea544e7ae3, with 30s UI waits, Pixel 6, API 33 and Node 22.23.4. Temporary app-repo CI refs filter the affected cases and pin that app checkout. The local hardware build additionally enables the existing TREZOR_BRIDGE=true fixture flag and its standard host bridge URL; app source is unchanged. Staging uses its original network backend/build flags. No Slack post; report job was skipped. This is a targeted rerun, not a new full Android suite.

Remaining Android evidence

  • Send2: attempts1/2 failed the preserved over-limit rejection assertion. Both screenshots and recordings show 10,001 accepted into the field while Available(Spending) is 9,000, with Continue disabled. This is not established as merely a missed/expired toast. Attempt3 observes the warning and rejected digit, then completes all invoice/payment assertions. Subsequent local investigation traced this to cached usable-channel outbound capacity overwriting the fresh spending limit; the Android app correction and targeted CI result are recorded below. These original failures/retries are retained, and the test oracle was not relaxed.
  • Lightning1: first attempt remains on Receive QR after SpecifyInvoiceButton, before any changed recipient-helper call; retry2 passes unchanged. Its screenshot, recording and native command transcript agree on the missing transition; exact cause remains unresolved.
  • Boost2: the boost success toast is observed, but first-attempt detail shows Removed from Mempool and lacks BoostedButton. App logs confirm replacement creation/broadcast/detection; retry2 passes. This is an unresolved activity-state discrepancy after the recipient helper succeeded, not a proven missed tap or missing boost toast.
  • Staging hardware2/4: all three attempts remain at zero balance after deposit API calls return transaction IDs. Hardware2 passes the corrected label check before failing receive; hardware4 fails initial software-wallet funding before pairing. The same app source passes the complete hardware money/lifecycle assertions on the local backend. That validates local behavior and does not establish staging health or its funding/sync root cause.

All four failed-shard archives, ten matching screenshots, usable recordings and retained device/app logs were inspected. Two staging hardware-Send recordings are header-only; another contains only 5.5s. The Boost recording ends before its failed detail state, which is captured in the PNG. Some failed-wallet app log sessions are lost when a later case reinstalls the app; logcat remains. These limits are retained in the triage evidence.

At e560ece, TypeScript and scoped lint pass, 83 TypeScript tests and 9 Python tests pass, and fresh independent review of the complete fix delta, callers and pinned native caption producer has no actionable findings. The fix commit is signed and GitHub Verified. No further source changes were made during validation. The six-run iOS comparison above remains attributed to 6de3b22; no new iOS execution is claimed at e560ece. Keep this PR draft while the remaining discrepancies are assessed.

Android Send2: app fix and targeted CI

Android #1808 refreshes the channel snapshot before invoice preparation trusts a cached usable channel. A fresh spending limit of 9,000 had been overwritten by cached outbound capacity 0 after a 10,000-sat Lightning receipt. Zero selects the number pad’s global limit, so 10,001 could be accepted until periodic balance correction. This is a local LND ↔ Bitkit channel; Send2 does not purchase a Blocktank Spending connection.

The local old/new comparison used the identical original app ebe9bc09850704d1e96a2bb1697fd5d5a664fd1a APK and equivalent fresh backends. Old Send2 at 849806e accepted 10,001 initially, then passed after its internal extra-digit fallback; rewritten Send2 at e560ece correctly failed the preserved warning and retained-field assertion. The fixed app at 8d0fcabfe passed the full unchanged case locally on its first attempt. Android compile, 4,944 unit tests and lint passed, with independent source review complete. The PR head adds only its changelog to that validated app source.

Targeted Android CI run 42573552198: success, Send2 passed first attempt (1/1). Attempts 2/3 were skipped. The Send2 job records 90/90 action/outcome observations passed, including the single rejected final digit: field stayed 1 000 and SendAmountExceededToast was observed in 717ms. All remaining on-chain, Lightning, unified-invoice, quickpay, invoice-edit, balance and history assertions completed.

CI input/result Pinned value
App b5ff8114be9a1392adc2caef371d5b367868c7dd — Android #1808
E2E e560ece2b7cc1fb5bbc1b72e8205abea544e7ae3 — unchanged assertions
Temporary workflow dbd0ae8be421b7be3f68451c77af1b83a7eb5632
Runtime Node 22.23.4, Pixel 6/API33 x86_64, local Docker backend, 30s UI waits
Reported suite / worker time 8m 25.6s / 8m 44s
CI case retries None

This is one targeted Send2 run. It validates the Android correction without changing the E2E test, adding an action retry, or weakening an assertion; it does not establish a new full-gate reliability rate or a speed improvement. The six-run iOS comparison remains attributed to 6de3b22; no new baseline or iOS execution is claimed. Android app builds need to include #1808 for this corrected behavior. The iOS RBF toast observation, Android Lightning Receive transition, Android RBF activity state and staging funding discrepancies remain outside this fix. Android #1808 is ready for review; this E2E PR remains draft.

Boost/Lightning investigation and local follow-up at 46fc6ae

Signed, GitHub-Verified E2E commit 46fc6ae2ee5310d63d21bf84e5add87d7f40cdd6 retains all existing warning, amount, fee/TXID, metadata/history, restore and channel assertions. UI waits remain 30 seconds; no action retries, added sleeps or OCR.

  • iOS Boost feedback: the native replacement notification can replace the four-second Boost success toast early. Old and rewritten tests use the same native gesture; the exact historical XCTest/accessibility timing is unproved. Transient iOS confirmations now share the existing LNURL scope: validate/save idle and animation settings before the gesture, observe the required toast, then restore exact prior values on success or failure. Persistent confirmations and Android gestures retain their previous policy. The candidate runtime observes BoostSuccessToast and restores XCTest settings from 0/0 to 10/2 seconds.
  • Android Lightning Receive: the edit button can be visible while the invoice subtree is being regenerated. The test now waits for a generated Lightning QR different from the already-paid invoice and resolves its enabled Edit button within the same visible pager page. Matching fresh URI, native identities and bounds samples reject placeholder, cached, moving and replaced nodes before one click. The precise historical click-delivery mechanism remains unproved; this closes the established readiness gap without relaxing the destination assertion.
  • Additional rewrite-related tag-sheet race: the unchanged local diagnostic passed Receive editing and payments, then failed at Tag-rtag with the selection sheet still open. Native bounds moved while the new helper was clicking; the old test had a 500ms wait here. Both receive/send tag selections now await an enabled stationary target before one click. Filtered signs/counts and restored notes remain asserted.
  • Android Boost2 remains a known publication/identity gap: retained CI logs show the original-row click preceding replacement publication. The original activity already carries the new fee and Boosting state; its detail stays tied to the old TXID and later shows Removed from Mempool. Home rows expose an index, no transaction identity, so fee/icon/bounds cannot establish replacement readiness. No delay or substitute assertion was added. The successful rerun below does not prove this race fixed; it needs native publication/observability work.

Complete candidate local runs on the same original app binaries:

Case Platform First candidate attempt Action/outcome observations
Boost2 iOS Pass 16/16
Boost2 Android Pass 16/16
Lightning1 Android Pass 38/38

Both Boost2 runs complete increased-fee/changed-TXID, RBF metadata, restore and mined-confirmation checks. Lightning1 completes edited 111-sat receipt/note/tag, outgoing payments, balances, both tag filters, restored history/notes and channel close. No candidate case retry.

Inputs: E2E 46fc6ae; original iOS app 8c1149be80a5488ea00da159779ca12706b99b10 (binary SHA256 dc347b99018e8fc6f0755d754e4ba50bf905c0394db43acda8a0fa579fe63f85), original Android app ebe9bc09850704d1e96a2bb1697fd5d5a664fd1a (APK SHA256 4aca1ec8c663450779d86b226b6b076dc62658f584c362f2175214aaeff63def), Node 22.23.4, iOS26.0/Xcode26.0.1 simulator and Pixel6/API33 ARM64 emulator. Separate isolated local backends; fresh app installation per case. Shared services were untouched. Local runtime/architecture differs from CI.

The pre-fix diagnostic is retained separately at e560ece: iOS Boost2 passed; Android Lightning1 passed Receive/payment steps but failed the moving tag-sheet transition. It is not counted as a candidate attempt or relabeled as a candidate result.

TypeScript, scoped ESLint, 111 TypeScript tests and 9 Python tests pass. Independent implementation review covers all eight changed files, callers, error/cleanup paths and pinned native contracts, with no actionable findings. Local traces, command timing, screenshots, recordings, native logs and exact invocation/build hashes are retained under work/boost-lightning-closeout-20261109.

This is targeted local validation, with no new baseline or CI cohort execution and no speed/reliability claim. The six-run CI comparison above remains pinned to 6de3b22. Keep the E2E PR draft while the Android RBF and staging funding limitations remain visible.

Android Boost2: native publication fix and targeted CI

Separate Android draft #1813 publishes the actual replacement activity, ancestry and metadata before Boost success, preserves the original fee and resolves a stale list row to the published successor. When broadcast returns before the replacement payment is available, it awaits one real wallet sync and resolves that exact returned TXID. No sleep, rebroadcast or E2E assertion change.

The first app-candidate run 42591327878, app 18b041c / E2E 46fc6ae, failed all three attempts waiting for BoostSuccessToast. All failed-attempt artifacts were inspected: the initial strict publication lookup preceded normal native payment discovery. The replacement appeared about 1.1 seconds later, and ancestry about 2.1 seconds after broadcast. That failure is retained; the awaited-sync correction is in app d28d15c.

Corrected Android CI run 42595261569: Boost2 passed first attempt, with attempts 2/3 skipped. The Boost2 job records 16/16 action/outcome observations passed. It completes the unchanged 10,000-sat payment, fee increase 141→422 sats, changed replacement TXID, RBF metadata, restored Boosting history and mined-confirmation assertions. Native logcat shows broadcast → awaited wallet sync → replacement link before success is observed. Suite time is 3m 43.7s; worker time is 4m 04s.

CI input Pinned value
App d28d15c54200679d2abd3c38a8a8d04068a14d77 — Android #1813
E2E 46fc6ae2ee5310d63d21bf84e5add87d7f40cdd6 — unchanged
Temporary workflow cd57c5fb4e15260909274f0ef1fb55b1a22d32fa
Runtime Node 22.23.4, Pixel 6/API33 x86_64, isolated local backend, 30s UI waits

At the final app commit, just compile, all 4,960 unit tests and just lint pass; independent implementation review has no unresolved findings. All app commits are signed and GitHub Verified. The native tests cover delayed publication and deterministic stale-row routing; #1813 retains the gap that J1's immediate manual tap was not driven separately. The passing CI artifact contains logcat and the restored-session app log; it has no screenshot or recording.

This is one targeted Android case, with no new full-suite, baseline or reliability cohort. The six-run iOS comparison remains at its historical pins. Android builds need the app corrections in #1808 and #1813 for the corrected Send2 and Boost2 behavior. Keep this E2E PR draft while those dependencies and staging funding remain open.

Stack refresh against main: CI in progress

Merged current E2E main 265a050057c6cc9cd03f965ba81fb1700f7361ee into pilot #296 at 96d6fcabea2ad3b86c223299c2f83d057c8806d3, then into this branch. Final E2E candidate is 38b12555b12857e760567fa727dcabd1b1923df3. Published history is preserved; all new commits are signed and GitHub Verified. Main's late-introduction recovery, authenticated regtest helpers and removal of obsolete skipped Blocktank cases are retained. All 13 active gate cases and original assertions remain. UI waits stay 30s; no OCR or action retries were added.

Both stack layers pass TypeScript, ESLint on all changed TS/JS and diff checks. Pilot passes 97 TypeScript + 36 Python tests; gate passes 137 TypeScript + 36 Python tests. Full independent reassessment of both layers and the temporary CI configuration found no actionable source defects. YAML, exact commit pins, Android merged-fix ancestry and the 13-case inventory checks pass.

Requested execution App pin CI run
Complete iOS gate, all 13 cases 10231d9e325446ca3be21daf9b7a0214e03af805 42693077441 — pending
Android Send2, Boost2, Lightning1 176430965fe1e27ca572a9c4fe5d48c339f8b1e9 — includes merged #1808 and #1813 42693073741 — 3/3 first-attempt passes; retries skipped

Temporary workflow pins: iOS a6a5a313e79cc7b619bd0867fafa364c2aabdded; Android 27e00181c2e87da8053f4e3bb65d602882cbef5c. Both hard-pin E2E 38b1255, use local backends, Node22 and the existing three-attempt policy. iOS keeps Trezor support; Android uses Pixel6/API33 x86_64 with isolated backends per case. Android logs confirm 90/90 Send2, 16/16 Boost2 and 38/38 Lightning1 outcomes passed. All original assertions completed, including Send2's rejected digit/required warning, Boost2 fee 141→422/new TXID/restore/mined confirmation, and Lightning1 invoice editing, payments, tag filters, restored notes/history and channel close. All three passing archives are retained (native logs, no screenshots/recordings). The full iOS result remains pending; all attempts will be retained. No new baseline or reliability cohort is being run. Both #296 and #298 stay draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant