Skip to content

docs: read Rust's bench and fuzz reports from the runtime's reports branch - #88

Merged
Kinflou merged 3 commits into
mainfrom
docs/verification-report-sources
Oct 25, 2026
Merged

Kinflou merged 3 commits into
mainfrom
docs/verification-report-sources

Conversation

@Kinflou

@Kinflou Kinflou commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Points the language reference at where the Rust runtime now publishes its verification reports, and renders the benchmark and fuzzing results when the site builds instead of committing them. The runtime side is ComlineProject/runtime#48.

Runtime's main only takes pull requests, so a workflow can't commit reports there. Runtime#48 has its CI commit them to an unprotected reports branch instead:

  • rust.bench.json, on every push to main;
  • rust.fuzz.json, from the nightly fuzz run.

Rendered at build time

These results change with every runtime push and every nightly run. Committed, they would make the weekly drift check flag the pages almost every week. So they aren't committed:

  • docs.yml runs scripts/sync-language-support.py --render-reports before zensical build --strict. It writes docs/_generated/reports/<name>.{bench,fuzz}.md, which git ignores.
  • The pages include those files with pymdownx.snippets (--8<--). That covers Rust's bench and fuzz sections and TypeScript's bench section. Zensical has no hook support, and the macros plugin doesn't render under it (the CLI page shows its {{ row.flag }} raw), so a snippet include is the mechanism that works.
  • The build never fails on a fetch. Each fetch has a 10 s timeout. A 404, network error, timeout or bad JSON renders the "not published yet" text and leaves a ::warning:: on the run. The step also has timeout-minutes: 2 and continue-on-error: true. If the step doesn't run at all, the includes are empty, which pymdownx.snippets allows, and the build still passes.
  • The existing daily 06:00 UTC scheduled build picks up new numbers; the runtime's nightly publishes around 03:30. There's also a new repository_dispatch (reports-published) so another repository can ask for a rebuild. Nothing sends that yet, because it needs a token that can write to this repository.
  • --check and the default sync no longer touch these sections. The support tables, runtime tables and divergences stay committed and drift-checked as before.

Other changes

  • scripts/sync-language-support.py:
    • Each verification source is now (repo, rev, path). Rust's bench and fuzz reports are read from runtime@reports.
    • New FUZZ table and --fuzz NAME=FILE, alongside --bench. Both work in --render-reports mode.
    • The fuzz renderer shows the run's date, commit, toolchain and run link, then a table of each target's seconds, executions, corpus size and findings, then the property tests' verdict.
  • reference/languages/rust.md: the fuzz paragraph now introduces the rendered results, and both sections say they're fetched when the site is built.
  • proposals/shared-runtime-core/verification.md: "What's left" no longer says the results are unpublished.
  • conduct/branch-rules.md:
    • A short section on runtime's reports branch.
    • Runtime's owned files are now all of .github/ plus the accepted-regressions ledger, matching runtime#48's CODEOWNERS.
    • No ruleset or required check changes.
  • docs/README.md: how to render the reports for a local build.
  • The sync also picked up drift already on runtime's and comline-typescript's main, in the committed tables.

Testing

  • With the real CI reports from runtime#48's manual runs:

    • 4 fuzz targets at 601 s each, 7 properties passing at 100k cases;
    • 58 benchmarks.

    They render, and zensical build --strict passes. The strict build also passes with no _generated/ at all.

  • With GitHub refused (a dead proxy), the render exits 0 in about 0.1 s with warnings. An unroutable host gives up after the timeout. A malformed report renders as not published, with a warning.

  • --check is clean.

  • Until runtime#48 merges and publishes, the sections say "not published yet". The first push to runtime's main publishes the bench report, and the next nightly publishes the fuzz report. The next docs build after that shows them.

…ranch

The runtime's CI now publishes rust.bench.json (every push to main) and
rust.fuzz.json (nightly) on its unprotected `reports` branch, since main
only takes pull requests. The sync script fetches them from there, and
renders the fuzz report in a new `fuzz` section on the Rust page.

Also picks up the conformance drift already on runtime and
comline-typescript main (JSON-RPC case count, a new divergence class).
They change with every runtime push and nightly run, so committing them
made the weekly drift check flag the pages almost every week. The Docs
workflow now runs sync-language-support.py --render-reports before the
build, writing docs/_generated/reports/ (ignored), which the pages
include with pymdownx.snippets. A report that 404s, times out (10 s) or
doesn't parse renders as "not published yet"; the build never fails on
it. --check no longer covers those sections. The daily scheduled build
picks up new numbers, and a repository_dispatch (reports-published) lets
another repository ask for a rebuild.

Branch rules: runtime's owned files are now all of .github/ and the
accepted-regressions ledger.
@Kinflou
Kinflou force-pushed the docs/verification-report-sources branch from 26d1c2c to 7c5d00c Compare October 25, 2026 19:56
@Kinflou
Kinflou merged commit d809127 into main Oct 25, 2026
3 checks passed
@Kinflou
Kinflou deleted the docs/verification-report-sources branch October 25, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant