Repository navigation
Migrate the 13-case iOS gate to selector action/outcome waits - #298
Draft
piotr-iohk wants to merge 7 commits into
Draft
piotr-iohk wants to merge 7 commits into
piotr-iohk wants to merge 7 commits into
Conversation
1 task done
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency: synonymdev/bitkit-android#1808 — Android app correction for Send2 after a Lightning receipt.
Dependency: synonymdev/bitkit-android#1813 — Android RBF replacement publication before Boost success.
All thirteen active
@ios_gatecases use selector action/outcome waits. Actions execute once; polling observes fresh selectors. Ordinary UI waits are 30 seconds. Existing financial, history, metadata, restore, channel and hardware lifecycle assertions, domain wait budgets and declared skips are retained. No OCR or app source changes.Stacked on #296 (
codex/e2e-outcome-pilot). Merge #296 first, then retarget tomain. Keep draft: refreshed against E2Emain(265a050) through pilot96d6fca, with final gate candidate38b1255. Android app fixes #1808 and #1813 are merged. Android Send2, Boost2 and Lightning1 passed first attempt (3/3, 144/144 outcomes) against a master build containing both merged fixes. The full iOS gate is still running. Historical comparisons below retain their original revisions. Staging funding has not been revalidated.Six-run iOS comparison
All six candidate runs completed successfully at the same app
8c1149be80a5488ea00da159779ca12706b99b10, E2E6de3b222fc7272087321022229763a14056c0e75and workflow90949e4a16d0066e8d3ce94c47b55c6cda567cce. Logs confirm Node 22.23.4, iOS 26.5 and Xcode 26.6. No source changes during the cohort. The historical baseline was not rerun.Observed first-worker median is 1m 12.7s / 2.3% lower; including retries, the median is 3m 13.5s / 5.7% lower. This small observational sample supports continuing the experiment; it does not establish a causal speedup or a durable lower flake rate. The failed first attempt ended its RBF case early, so first-worker timing alone is especially limited.
The only candidate iOS failure was
boost_2in run 42370172726:BoostSuccessToastwas not observed within 30s; the unchanged case passed on attempt 2. Its matching recording visibly shows the green Boosted! toast briefly, followed by a blue Transaction Replaced toast, while the final screenshot shows a Boosting transaction. This is a missed transient-success observation; the exact accessibility/poll scheduling cause remains unproven. No action was replayed within the failed attempt.Baseline references: 42201867907, 42199157743, 42164497264, 42119619712, 42118872824, 42116447900. The first is explicitly pinned to the same app and had 11/13 first-attempt passes. Other historical app revisions vary. Baseline CI already used 30s UI waits; this is not evidence that increasing the CI timeout caused the difference.
Timings use reported worker summaries, excluding queue/build/setup time. Baseline logs do not expose exact per-case end times. Five additional candidates ran concurrently on isolated runners with separate fresh backends; runtime variation remains a caveat.
Android at frozen 6de3b22, before the fix
Pinned Android app/workflows
ebe9bc09850704d1e96a2bb1697fd5d5a664fd1a, same E2E6de3b22, 30s UI waits, Pixel 6 API 33 and Node 22. Both workflows completed with failure after their normal three-attempt retry policy. Skips are excluded from pass counts.send_1,send_2,send_3fail in all three attempts. All nine screenshots and usable video tails remain on the populated recipient form with Continue visible. Lightning and Transfer hit the same transition on their first attempts and recover on attempt 2. Exact click/readiness cause is unproven; this is not a MoveToSpark intro in the inspected artifacts.E2E TREZOR; pinned Android source renders this caption uppercase. This is a test/UI-contract mismatch. An existing same-app hardware baseline passed all three hardware cases first attempt with old E2E source; no new baseline execution.hardware_wallet_4and the unchangedpaykit_1fail their initial software-wallet funding balance waits in all three attempts, before the migrated hardware flow/Paykit path. Screenshots show zero balance; deposit API calls return transaction IDs. Funding/sync root cause remains unresolved and is not attributed to the rewrite from these artifacts.Across the thirteen migrated cases on Android: 5/13 first attempt, 7/13 eventual. The local suite covers ten; staging supplies the three hardware cases. Five declared skips across both workflows remain skipped. All five failed Android shard archives, attempt screenshots, usable recordings and device/app logs were inspected. Some hardware/Paykit recordings are header-only or cannot decode their tails; screenshots/logs remain available. These failures prompted the Android follow-up below.
Frozen 6de3b22 per-case comparison
boost_1boost_2hardware_wallet_1hardware_wallet_2hardware_wallet_4lightning_1lnurl_1multi_address_3multi_address_4send_1send_2send_3transfer_2Other validation
At frozen
6de3b22, TypeScript, scoped lint, 78 TypeScript tests and 9 Python tests passed; independent implementation review is complete. The three original implementation commits are signed and Verified. Prior local iOS gate validation was 12/13 first attempt, 13/13 after one unchanged-sourcemulti_address_4retry. This validation pass changes no source and retains all first failures and retries.See
docs/outcome-gate-migration.mdfor retained coverage and the comparison protocol. The follow-up below addresses Android recipient readiness and the hardware-name outcome. The original iOS RBF-toast observation and the additional Android issues remain relevant to adoption; this draft is not merge-ready.Android follow-up at e560ece: completed
The fix waits for Android Recipient Continue to be enabled, with the keyboard hidden and two matching native bounds samples before one click. Invalid/stale samples reset readiness; outcome failure never replays the action. Send validation, manual invoice entry and invoice editing share this path. iOS delegates to the original tap helper. Hardware home captions are verified uppercase on Android; Settings still verifies stored mixed-case names. Existing financial, fee, history, metadata, negative-validation and hardware lifecycle assertions remain.
Targeted local-backend Android results: 12/12 eventual passes, 9/12 first attempt. All three active hardware cases pass first attempt. Logs show 28 recipient-confirmation outcomes and 7 scoped hardware-caption outcomes passed, with zero failures at either corrected boundary, including the staging label checks. This validates the specific fixes; remaining failures below prevent claiming a clean or reliably green Android gate.
Completed workflows:
All builds explicitly check out app
ebe9bc09850704d1e96a2bb1697fd5d5a664fd1a; tests are frozen ate560ece2b7cc1fb5bbc1b72e8205abea544e7ae3, with 30s UI waits, Pixel 6, API 33 and Node 22.23.4. Temporary app-repo CI refs filter the affected cases and pin that app checkout. The local hardware build additionally enables the existingTREZOR_BRIDGE=truefixture flag and its standard host bridge URL; app source is unchanged. Staging uses its original network backend/build flags. No Slack post; report job was skipped. This is a targeted rerun, not a new full Android suite.Remaining Android evidence
10,001accepted into the field while Available(Spending) is9,000, with Continue disabled. This is not established as merely a missed/expired toast. Attempt3 observes the warning and rejected digit, then completes all invoice/payment assertions. Subsequent local investigation traced this to cached usable-channel outbound capacity overwriting the fresh spending limit; the Android app correction and targeted CI result are recorded below. These original failures/retries are retained, and the test oracle was not relaxed.All four failed-shard archives, ten matching screenshots, usable recordings and retained device/app logs were inspected. Two staging hardware-Send recordings are header-only; another contains only 5.5s. The Boost recording ends before its failed detail state, which is captured in the PNG. Some failed-wallet app log sessions are lost when a later case reinstalls the app; logcat remains. These limits are retained in the triage evidence.
At
e560ece, TypeScript and scoped lint pass, 83 TypeScript tests and 9 Python tests pass, and fresh independent review of the complete fix delta, callers and pinned native caption producer has no actionable findings. The fix commit is signed and GitHub Verified. No further source changes were made during validation. The six-run iOS comparison above remains attributed to6de3b22; no new iOS execution is claimed ate560ece. Keep this PR draft while the remaining discrepancies are assessed.Android Send2: app fix and targeted CI
Android #1808 refreshes the channel snapshot before invoice preparation trusts a cached usable channel. A fresh spending limit of 9,000 had been overwritten by cached outbound capacity 0 after a 10,000-sat Lightning receipt. Zero selects the number pad’s global limit, so 10,001 could be accepted until periodic balance correction. This is a local LND ↔ Bitkit channel; Send2 does not purchase a Blocktank Spending connection.
The local old/new comparison used the identical original app
ebe9bc09850704d1e96a2bb1697fd5d5a664fd1aAPK and equivalent fresh backends. Old Send2 at849806eaccepted 10,001 initially, then passed after its internal extra-digit fallback; rewritten Send2 ate560ececorrectly failed the preserved warning and retained-field assertion. The fixed app at8d0fcabfepassed the full unchanged case locally on its first attempt. Android compile, 4,944 unit tests and lint passed, with independent source review complete. The PR head adds only its changelog to that validated app source.Targeted Android CI run 42573552198: success, Send2 passed first attempt (1/1). Attempts 2/3 were skipped. The Send2 job records 90/90 action/outcome observations passed, including the single rejected final digit: field stayed
1 000andSendAmountExceededToastwas observed in 717ms. All remaining on-chain, Lightning, unified-invoice, quickpay, invoice-edit, balance and history assertions completed.b5ff8114be9a1392adc2caef371d5b367868c7dd— Android #1808e560ece2b7cc1fb5bbc1b72e8205abea544e7ae3— unchanged assertionsdbd0ae8be421b7be3f68451c77af1b83a7eb5632This is one targeted Send2 run. It validates the Android correction without changing the E2E test, adding an action retry, or weakening an assertion; it does not establish a new full-gate reliability rate or a speed improvement. The six-run iOS comparison remains attributed to
6de3b22; no new baseline or iOS execution is claimed. Android app builds need to include #1808 for this corrected behavior. The iOS RBF toast observation, Android Lightning Receive transition, Android RBF activity state and staging funding discrepancies remain outside this fix. Android #1808 is ready for review; this E2E PR remains draft.Boost/Lightning investigation and local follow-up at 46fc6ae
Signed, GitHub-Verified E2E commit
46fc6ae2ee5310d63d21bf84e5add87d7f40cdd6retains all existing warning, amount, fee/TXID, metadata/history, restore and channel assertions. UI waits remain 30 seconds; no action retries, added sleeps or OCR.Complete candidate local runs on the same original app binaries:
Both Boost2 runs complete increased-fee/changed-TXID, RBF metadata, restore and mined-confirmation checks. Lightning1 completes edited 111-sat receipt/note/tag, outgoing payments, balances, both tag filters, restored history/notes and channel close. No candidate case retry.
Inputs: E2E
46fc6ae; original iOS app8c1149be80a5488ea00da159779ca12706b99b10(binary SHA256dc347b99018e8fc6f0755d754e4ba50bf905c0394db43acda8a0fa579fe63f85), original Android appebe9bc09850704d1e96a2bb1697fd5d5a664fd1a(APK SHA2564aca1ec8c663450779d86b226b6b076dc62658f584c362f2175214aaeff63def), Node 22.23.4, iOS26.0/Xcode26.0.1 simulator and Pixel6/API33 ARM64 emulator. Separate isolated local backends; fresh app installation per case. Shared services were untouched. Local runtime/architecture differs from CI.The pre-fix diagnostic is retained separately at
e560ece: iOS Boost2 passed; Android Lightning1 passed Receive/payment steps but failed the moving tag-sheet transition. It is not counted as a candidate attempt or relabeled as a candidate result.TypeScript, scoped ESLint, 111 TypeScript tests and 9 Python tests pass. Independent implementation review covers all eight changed files, callers, error/cleanup paths and pinned native contracts, with no actionable findings. Local traces, command timing, screenshots, recordings, native logs and exact invocation/build hashes are retained under
work/boost-lightning-closeout-20261109.This is targeted local validation, with no new baseline or CI cohort execution and no speed/reliability claim. The six-run CI comparison above remains pinned to
6de3b22. Keep the E2E PR draft while the Android RBF and staging funding limitations remain visible.Android Boost2: native publication fix and targeted CI
Separate Android draft #1813 publishes the actual replacement activity, ancestry and metadata before Boost success, preserves the original fee and resolves a stale list row to the published successor. When broadcast returns before the replacement payment is available, it awaits one real wallet sync and resolves that exact returned TXID. No sleep, rebroadcast or E2E assertion change.
The first app-candidate run 42591327878, app
18b041c/ E2E46fc6ae, failed all three attempts waiting for BoostSuccessToast. All failed-attempt artifacts were inspected: the initial strict publication lookup preceded normal native payment discovery. The replacement appeared about 1.1 seconds later, and ancestry about 2.1 seconds after broadcast. That failure is retained; the awaited-sync correction is in appd28d15c.Corrected Android CI run 42595261569: Boost2 passed first attempt, with attempts 2/3 skipped. The Boost2 job records 16/16 action/outcome observations passed. It completes the unchanged 10,000-sat payment, fee increase 141→422 sats, changed replacement TXID, RBF metadata, restored Boosting history and mined-confirmation assertions. Native logcat shows broadcast → awaited wallet sync → replacement link before success is observed. Suite time is 3m 43.7s; worker time is 4m 04s.
d28d15c54200679d2abd3c38a8a8d04068a14d77— Android #181346fc6ae2ee5310d63d21bf84e5add87d7f40cdd6— unchangedcd57c5fb4e15260909274f0ef1fb55b1a22d32faAt the final app commit,
just compile, all 4,960 unit tests andjust lintpass; independent implementation review has no unresolved findings. All app commits are signed and GitHub Verified. The native tests cover delayed publication and deterministic stale-row routing; #1813 retains the gap that J1's immediate manual tap was not driven separately. The passing CI artifact contains logcat and the restored-session app log; it has no screenshot or recording.This is one targeted Android case, with no new full-suite, baseline or reliability cohort. The six-run iOS comparison remains at its historical pins. Android builds need the app corrections in #1808 and #1813 for the corrected Send2 and Boost2 behavior. Keep this E2E PR draft while those dependencies and staging funding remain open.
Stack refresh against main: CI in progress
Merged current E2E main
265a050057c6cc9cd03f965ba81fb1700f7361eeinto pilot #296 at96d6fcabea2ad3b86c223299c2f83d057c8806d3, then into this branch. Final E2E candidate is38b12555b12857e760567fa727dcabd1b1923df3. Published history is preserved; all new commits are signed and GitHub Verified. Main's late-introduction recovery, authenticated regtest helpers and removal of obsolete skipped Blocktank cases are retained. All 13 active gate cases and original assertions remain. UI waits stay 30s; no OCR or action retries were added.Both stack layers pass TypeScript, ESLint on all changed TS/JS and diff checks. Pilot passes 97 TypeScript + 36 Python tests; gate passes 137 TypeScript + 36 Python tests. Full independent reassessment of both layers and the temporary CI configuration found no actionable source defects. YAML, exact commit pins, Android merged-fix ancestry and the 13-case inventory checks pass.
10231d9e325446ca3be21daf9b7a0214e03af805176430965fe1e27ca572a9c4fe5d48c339f8b1e9— includes merged #1808 and #1813Temporary workflow pins: iOS
a6a5a313e79cc7b619bd0867fafa364c2aabdded; Android27e00181c2e87da8053f4e3bb65d602882cbef5c. Both hard-pin E2E38b1255, use local backends, Node22 and the existing three-attempt policy. iOS keeps Trezor support; Android uses Pixel6/API33 x86_64 with isolated backends per case. Android logs confirm 90/90 Send2, 16/16 Boost2 and 38/38 Lightning1 outcomes passed. All original assertions completed, including Send2's rejected digit/required warning, Boost2 fee 141→422/new TXID/restore/mined confirmation, and Lightning1 invoice editing, payments, tag filters, restored notes/history and channel close. All three passing archives are retained (native logs, no screenshots/recordings). The full iOS result remains pending; all attempts will be retained. No new baseline or reliability cohort is being run. Both #296 and #298 stay draft.