Skip to content

ci: publish the bench and fuzz reports to a reports branch - #48

Merged
Kinflou merged 2 commits into
mainfrom
ci/publish-reports
Oct 25, 2026
Merged

Kinflou merged 2 commits into
mainfrom
ci/publish-reports

Conversation

@Kinflou

@Kinflou Kinflou commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Publishes the Rust runtime's benchmark and fuzz reports somewhere the docs can fetch them. Until now rust.bench.json was only a pull-request artifact, and fuzzing left only job summaries.

Where: a reports branch

Each report is committed to an unprotected reports branch, at its root:

  • https://raw.githubusercontent.com/ComlineProject/runtime/reports/rust.bench.json
  • https://raw.githubusercontent.com/ComlineProject/runtime/reports/rust.fuzz.json

Why this target:

  • The rulesets stay as they are. Both rulesets target ~DEFAULT_BRANCH only, so reports isn't covered. Nothing is pushed to main, and no rule or bypass changes.
  • A stable raw URL. The docs' sync script already reads raw.githubusercontent.com, so it only needs a different rev. GitHub Pages would need enabling and a deploy job, and it would replace the site on every deploy, so it keeps no history. Release assets would need a release (or a tag that moves) per push, and their URLs are awkward to fetch.
  • History. Each publish is one fast-forward commit, not a force-push, so git log origin/reports -- rust.bench.json is the series, and each commit's raw URL keeps pointing at its report. The files are a few KB, so the branch stays small.
  • Tied to the commit it measured. Each report carries commit, the main commit it measured, and the publish commit's message names it too. A report older than the one already published (its commit is a strict ancestor) is skipped, so a slow run can't replace a newer one. Bench runs on main also cancel superseded runs.
  • Fork-PR safety. Only push to main, schedule and workflow_dispatch from main publish. The writing jobs are the only ones with contents: write, and they're guarded by github.ref == 'refs/heads/main' && github.event_name != 'pull_request'. Pull requests build the reports as artifacts and dry-run the publish with a read-only token.
  • No setup. The first publish creates the branch.

.github/scripts/publish-report.py writes the commit with git plumbing (hash-object, mktree, commit-tree), so it needs no working tree. A push that loses a race is retried on the new head; each report is its own file, so the retries never conflict. --dry-run does everything except push. The branch gets a README.md (.github/reports-branch.md) saying it's written by CI.

Benchmarks on main (bench.yml)

  • New push: [main] and workflow_dispatch triggers. New job publish instruction counts: it benches the head alone, runs bench-report.py without the gate (no --limit), uploads the artifact and publishes. On a branch other than main it does a dry run.
  • The gate (instruction counts) is unchanged apart from if: github.event_name == 'pull_request', so it still runs, and only runs, on pull requests.
  • bench-report.py: one wording change. With no --limit, the summary says "The head alone, with nothing to compare with." instead of claiming a base exists.

Fuzz report (fuzz.yml, fuzz-report.py)

rust.fuzz.json is documented in runtime/fuzz/README.md ("The report"):

{
  "runtime": "rust", "commit": "…", "toolchain": "rustc … nightly …",
  "date": "2026-10-26T03:41:07Z", "run": "https://github.com/…/actions/runs/…",
  "targets": { "decode_no_panic": { "seconds": 600, "executions": 18734210, "corpus": 2841, "findings": 0 } },
  "proptest": { "cases": 100000, "properties": { "json_round_trips": "pass" } }
}
  • Each fuzz job now builds first, then runs with -print_final_stats=1 and tees the log. A Record the run step (always(), before minimizing) turns the log into a fragment:
    • executions: stat::number_of_executed_units.
    • seconds: from libFuzzer's Done N runs in S second(s), or the step's wall-clock time when a finding stopped it, since libFuzzer prints the stats on a crash but not Done.
    • corpus: the number of inputs in the corpus after the run.
    • findings: the crash-/leak-/timeout-/oom-/slow-unit- files.
  • The proptest job tees its output and records each test … ok|FAILED line.
  • New job fuzz report (!cancelled(), read-only) combines the fragments, writes the summary, uploads the rust.fuzz.json artifact, and dry-runs the publish off main. On a pull request proptest is null, because the long run isn't part of a pull request's run. A target whose job left no fragment gets an error entry with null numbers, not zeros.
  • New job publish fuzz report (contents: write, only on main and never on a pull request) publishes it.
  • Job names fuzz (<target>) are unchanged, so the required checks still match.

Tested

  • Locally: fuzz-report.py on real libFuzzer logs:
    • a clean 10 s run of framing_decode;
    • a forced out-of-memory finding (-rss_limit_mb=40), where the stats are printed, Done isn't, the seconds fall back to wall-clock, and findings: 1.
  • Locally: publish-report.py against a bare clone of this repo:
    • dry run with the branch missing;
    • creating the branch;
    • a no-op republish;
    • adding the second report;
    • skipping an older-commit report;
    • a push rejected once by a hook, then retried and pushed.
  • On this PR: the fuzz workflow's fuzz report job builds rust.fuzz.json as an artifact and dry-runs the publish. A workflow_dispatch of both workflows on this branch runs the full nightly path (10 min per target plus proptest at 100k) and the bench main job, and each ends in a dry run.
  • After merge only: the real push to reports, i.e. that GITHUB_TOKEN with contents: write may create and update the branch. The first push to main (the merge itself) runs it for the bench report, and the next nightly does it for the fuzz report.

The docs side is ComlineProject/docs#88, which reads these URLs.

On every push to main, bench.yml benches the head alone and commits
rust.bench.json to the unprotected `reports` branch. The nightly fuzz
run writes a new rust.fuzz.json (per-target seconds, executions, corpus
and findings from libFuzzer's final stats; proptest verdicts) and
commits it there too. Pull requests build the reports as artifacts and
dry-run the publish; only push to main, the schedule and manual runs
from main write.
@Kinflou
Kinflou merged commit e3b9d77 into main Oct 25, 2026
15 checks passed
@Kinflou
Kinflou deleted the ci/publish-reports branch October 25, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant