Skip to content

fix: report the cause of ghostty-web renderer failures - #372

Open
ThomasK33 wants to merge 2 commits into
mainfrom
fix/ghostty-web-failure-reports
Open

ThomasK33 wants to merge 2 commits into
mainfrom
fix/ghostty-web-failure-reports

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Why

Three things hide the real cause when the ghostty-web renderer fails.

  1. The crash message blames one input. ghostty-web 0.4.0 crashes its page on several inputs, not only the one of coder/ghostty-web#215 (a full reset, ESC c, then ?47h). A plain print of 693 lines that hold combining marks crashes it too. For that input the old text pointed at a reset that was never there.
  2. A live wait --renderer ghostty-web hid the crash. The host retried every failed poll up to 10 times in a row. Each poll boots a page and replays the whole log, so the wait took ten crashes (about 2 s each on this host, longer on a loaded one). When --timeout was shorter than that, the wait ended as WAIT_TIMEOUT (exit 11) and the crash never showed. A live snapshot and screenshot on the same session gave REPLAY_ERROR (exit 10).
  3. An offline REPLAY_ERROR ("Failed to boot or replay renderer for session ...") and a record export --format webm EXPORT_ERROR ended in a bare sentence for every failure that is not a page crash. The RangeError: offset is out of bounds that an empty output event gives was not visible.

What

Crash text (src/renderer/ghosttyWeb/crashError.ts)

  • The first sentence stays. The rest now says that the WASM module of ghostty-web 0.4.0 crashed and that several different inputs do this. It names verify-agent-tty: holder-scan gaps left after #211 #215 as one known cause, not as the cause. It keeps the advice that works: the default libghostty-vt renderer for snapshot and wait, and record export --format svg or asciicast.
  • docs/TROUBLESHOOTING.md, docs/USAGE.md and the READMEs of src/renderer/ghosttyWeb, src/replay, src/cli/commands, src/host and src/util follow the new text. No file under skills/ quotes the crash text, so no public skill changes.

Live render wait (src/host/hostMain.ts, src/host/renderer.ts)

  • Errors that a render poll retries today: every error that the poll throws. Examples are a Chromium launch that fails under load, a page call that times out, a failed replay-input load and any other renderer error. The retry is deliberate. The next poll builds a new page, so a launch failure can pass. The wait rejects with the last error after MAX_CONSECUTIVE_POLL_FAILURES (10) failures in a row. I did not change this.
  • The one exception is a crash of the ghostty-web page. The next poll replays the same output and crashes again, so retrying only costs time. isReplayCrashError (new, in renderer.ts, next to the code that builds the error) recognizes the REPLAY_ERROR that withBackend throws for a crash. The poll loop rejects with it at the first poll. The wait now gives the error that snapshot and screenshot give: REPLAY_ERROR, exit 10, retryable: false, no result.
  • A --timeout shorter than the first poll still ends as WAIT_TIMEOUT, because the timeout fires before the poll ends.

Cause in offline errors (src/util/errorSummary.ts, src/replay/offlineReplay.ts, src/cli/commands/record-export.ts)

  • New summarizeError(error): the first line of the error message, whitespace collapsed, at most 300 characters (ends in … when cut). It never carries a stack frame or the Playwright call log, which follow the first line.
  • The offline replay error now reads Failed to boot or replay renderer for session <id>: <cause>. The record export error reads Failed to export session "<id>" as <format>: <cause>. A page crash keeps its own explanation after a period, as before.
  • The export change sits in the catch of every format, so an unexpected asciicast or SVG failure names its cause too. details keep their shape and the JSON schemas do not change. Only the message text of those errors changes. No golden test pinned it.
  • A first line can name a path that the failing tool prints (for example Playwright prints the browser executable path). A live command already shows the same text as RPC_ERROR, so this adds no new kind of disclosure. Stack frames are never included.

Not touched: replayWithTiming in backend.ts and writeBatch in embeddedHarnessHtml.ts (PR #366). A trial merge with the head of #366 (074b3a31) has no conflicts.

Validation

Machine: shared host, load 11 to 263 during the runs.

  • npm run format:check, npm run lint (0 warnings, 0 errors), npm run typecheck: clean.
  • npx vitest run test/unit: 150 files, 2414 tests pass on the first commit (the second commit only splits one test into two, 2415 tests). See the load note below for the later run.
  • check.sh --drift-only: "Covered code matches the sync record."
  • Integration and e2e, each file alone with --maxWorkers=1 --retry=0: 30 files (20 integration, 10 e2e: ghostty-web-crash, wait-render, renderer-failure-cause, host-renderer-rpc, record-export, empty-session-capture, e2e renderer-errors, record-diff, screenshot-out, screenshot-missing-glyphs, renderer-backend, batch, screen-hash, event-log, slow-replay, backend-selection, cli, run, mouse, artifact-manifest-race, pty-basics, and e2e export-fixtures, renderer-slice, unicode-grid, scrollback-demo, libghostty-vt-renderer, resize-demo, large-snapshot, hello-prompt, batch-fixture). 27 files passed on the first run. Three files had red tests. I judged each by its assertion, and each is a load timeout, not a result of this change:
    • wait-render: "detects screen stability via waitForRender RPC" (Hook timed out in 30000ms in createSession at load 69) and "prints the WAIT_TIMEOUT error to stderr when CLI --text times out without --json" (expected 1 to be 11: runCli killed the CLI at its 10 s limit, which gives status null). Both pass when run alone.
    • mouse: the 5 "delivers ... bytes to a PTY" tests (Test timed out in 30000ms, each took 34 to 41 s at load 89). All 15 pass alone (those tests took 17 to 28 s).
    • backend-selection: "defaults semantic snapshot rendering to libghostty-vt when available" (Test timed out in 5000ms). 7 of 7 pass with --testTimeout=60000. A copy of origin/main fails 7 of 7 at the 5 s default under the same load, so this is not from this change.
    • The full unit run passed once (150 files, 2414 tests, load about 35). A later full run at load 140 to 260 failed 14 tests in 10 files that this PR does not touch (doctor, gc, version, lifecycle, capture, replaySlices, artifactHealth, artifactManifestLock, lineDiff, evals builtins). All were Test timed out in 5000ms or a timing bound (lineDiff: 22178 ms against 10000). All 247 tests of those 10 files pass with --testTimeout=60000 --hookTimeout=60000.

New tests, and the mutant that turns each red (each mutant ran in a git archive copy, an identical-text control passed first):

Fix Mutant Red tests
Crash text restore the old text ghostty-web-crash.test.ts "names the crash of an input that has no reset, and ends a live ghostty-web wait at it", and unit ghosttyWebCrash.test.ts "names the issue as one known cause ..." and "names the commands that do not use ghostty-web"
Wait ends at a crash drop isReplayCrashError(pollError) || from the poll loop wait-render.test.ts "ends a render wait at the first poll that crashes the ghostty-web page" and the real-crash test above
Wait ends at a crash predicate always false, or without the code check unit renderer.test.ts "tells the error of a crashed page from every other poll failure"
Other poll errors stay retried every poll error ends the wait wait-render.test.ts "retries a render poll that fails for another reason until the cap" and "keeps polling through fewer failed render polls than the cap and matches"
Offline REPLAY_ERROR cause drop the cause unit offlineReplay.test.ts x2 (replayTo and boot) and renderer-failure-cause.test.ts x3 (snapshot, screenshot, wait)
WebM EXPORT_ERROR cause drop the cause unit record-export.test.ts x3 and renderer-failure-cause.test.ts "... WebM export"
summarizeError no length bound / whole message instead of first line unit errorSummary.test.ts and offlineReplay.test.ts "bounds the length ..." / "leaves the stack ... out of the message"

Tests added: 8 in errorSummary.test.ts, 2 in ghosttyWebCrash.test.ts, 1 in renderer.test.ts, 2 in offlineReplay.test.ts and 2 changed there, 2 changed and 1 added in record-export.test.ts (unit), and in integration 2 in wait-render.test.ts, 1 in ghostty-web-crash.test.ts and the 4 tests of the new renderer-failure-cause.test.ts.

The wait tests count the host's render polls with test/fixtures/inject-replay-failure.ts. A crash gives 1 poll, an ordinary failure gives 10 (MAX_CONSECUTIVE_POLL_FAILURES). The ghostty-web-crash test uses a real Chromium crash and the same count, so it does not depend on how fast the host is.

Dogfood with the real CLI (node --import tsx src/cli/main.ts, isolated AGENT_TTY_HOME folders under /tmp, sessions destroyed and folders removed). Input "crash A": 693 lines of e\u0301x5 + abcdefghijx7 + abcde, 80x24, no RIS, no resize, no alternate screen. Wall time includes about 4 s of tsx start under load.

Command Before After
live snapshot --renderer ghostty-web exit 10 REPLAY_ERROR, 5.7 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 7.0 s, new text
live screenshot exit 10 REPLAY_ERROR, 6.2 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 6.7 s, new text
live wait --text NEVER (default 30 s timeout) exit 10 REPLAY_ERROR, 21.2 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 7.6 s, new text
live wait --timeout 120000 exit 10 REPLAY_ERROR, 20.2 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 8.0 s, new text
live wait --timeout 8000 exit 11 WAIT_TIMEOUT, 15.6 s, "Wait timed out before its condition was met." exit 10 REPLAY_ERROR, 10.5 s, new text
offline snapshot --renderer ghostty-web exit 10 REPLAY_ERROR, 6.0 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 6.5 s, new text
offline screenshot exit 10 REPLAY_ERROR, 5.8 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 8.8 s, new text
offline wait --renderer ghostty-web exit 10 REPLAY_ERROR, 5.9 s, old text (blames ESC c) exit 10 REPLAY_ERROR, 6.3 s, new text
offline record export webm exit 7 EXPORT_ERROR, 6.3 s, old text (blames ESC c) exit 7 EXPORT_ERROR, 8.4 s, new text
offline record export svg exit 0 ok, 4.8 s exit 0 ok, 5.1 s
offline record export asciicast exit 0 ok, 4.8 s exit 0 ok, 5.7 s

Input without a page crash: an exited session whose log holds an empty output event (RangeError: offset is out of bounds in the page on main).

Command Before After
offline snapshot exit 10 REPLAY_ERROR, 23.2 s: Failed to boot or replay renderer for session <id>. exit 10 REPLAY_ERROR, 5.9 s: Failed to boot or replay renderer for session <id>: page.evaluate: RangeError: offset is out of bounds
offline screenshot (default renderer) exit 10 REPLAY_ERROR, 17.9 s: Failed to boot or replay renderer for session <id>. exit 10 REPLAY_ERROR, 6.3 s: Failed to boot or replay renderer for session <id>: page.evaluate: RangeError: offset is out of bounds
offline wait exit 10 REPLAY_ERROR, 8.8 s: Failed to boot or replay renderer for session <id>. exit 10 REPLAY_ERROR, 6.8 s: Failed to boot or replay renderer for session <id>: page.evaluate: RangeError: offset is out of bounds
offline webm export exit 7 EXPORT_ERROR, 4.8 s: Failed to export session "<id>" as webm. exit 7 EXPORT_ERROR, 5.0 s: Failed to export session "<id>" as webm: page.evaluate: RangeError: offset is out of bounds

The wall times of the offline rows differ by load more than by the change. Only the live wait rows are a before and after comparison: a crash used to cost one replay per poll, and now it costs one. A wait with a timeout shorter than ten polls ended as WAIT_TIMEOUT before the fix. The diagnosis run used --timeout 5000 and my run above used --timeout 8000.


Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

The crash message named one input (RIS, then ?47h) as the known cause, but
ghostty-web 0.4.0 crashes on several inputs. It now says so and names
coder/ghostty-web#215 as one known cause.

A live render wait retried a crashed page for 10 polls. Each poll replays
the log into a new page, so the wait took ten crashes and could return
WAIT_TIMEOUT instead of the error. It now ends at the first crash with the
same REPLAY_ERROR that a live snapshot gives.

An offline REPLAY_ERROR and a WebM EXPORT_ERROR left out the failure that
caused them. Their messages now end with its first line, cut at 300
characters.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T17:18:51.533294Z 5f1939b Manual request
🔒 Security Review ✅ Completed 2026-11-10T17:17:45.492018Z 5f1939b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5f1939b40d

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f1939b40d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"ghostty-web crashed while it replayed this session's output. One known cause is a ghostty-web 0.4.0 bug: after a full reset (ESC c), the alternate screen (?47h or ?1047h) keeps the old cells, and output over a combining mark or other grapheme cluster there crashes the page. The libghostty-vt renderer and `record export --format svg` are not affected.";
"ghostty-web crashed while it replayed this session's output. This is a bug in the WASM module of ghostty-web 0.4.0, which several different inputs trigger. One known cause is https://github.com/coder/ghostty-web/issues/215 (a full reset, ESC c, then ?47h or ?1047h, then output over a combining mark or other grapheme cluster). The default libghostty-vt renderer (`snapshot` and `wait`), `record export --format svg` and `record export --format asciicast` do not use ghostty-web.";

const PAGE_CRASH_PATTERN = /Target crashed/u;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Qualify libghostty-vt as an optional workaround

When the optional native package is unavailable—or the user explicitly configured ghostty-web—semantic snapshot and wait commands use ghostty-web rather than “the default libghostty-vt renderer,” so following this crash message simply reproduces the same failure. Qualify this workaround as applying only when libghostty-vt is installed and selected, rather than implying that these commands universally avoid ghostty-web.

Useful? React with 👍 / 👎.

Comment thread src/host/hostMain.ts
Comment on lines +1181 to +1182
isReplayCrashError(pollError) ||
consecutiveFailures >= MAX_CONSECUTIVE_POLL_FAILURES

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve retries for unclassified Chromium crashes

When Chromium reports Target crashed because of a transient page/process failure such as resource pressure, describeReplayFailure classifies it as GhosttyWebCrashedError even though it cannot establish that the session output hit the deterministic WASM bug. The backend's crash handler marks it unbooted, so a later poll can recreate the page and recover, but this new branch now aborts on the first failure as a non-retryable REPLAY_ERROR; retain the existing retry allowance unless the failure can actually be tied to a deterministic input fault.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant