Repository navigation
ci: publish the bench and fuzz reports to a reports branch - #48
Merged
Merged
Conversation
On every push to main, bench.yml benches the head alone and commits rust.bench.json to the unprotected `reports` branch. The nightly fuzz run writes a new rust.fuzz.json (per-target seconds, executions, corpus and findings from libFuzzer's final stats; proptest verdicts) and commits it there too. Pull requests build the reports as artifacts and dry-run the publish; only push to main, the schedule and manual runs from main write.
…e required checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishes the Rust runtime's benchmark and fuzz reports somewhere the docs can fetch them. Until now
rust.bench.jsonwas only a pull-request artifact, and fuzzing left only job summaries.Where: a
reportsbranchEach report is committed to an unprotected
reportsbranch, at its root:https://raw.githubusercontent.com/ComlineProject/runtime/reports/rust.bench.jsonhttps://raw.githubusercontent.com/ComlineProject/runtime/reports/rust.fuzz.jsonWhy this target:
~DEFAULT_BRANCHonly, soreportsisn't covered. Nothing is pushed tomain, and no rule or bypass changes.raw.githubusercontent.com, so it only needs a different rev. GitHub Pages would need enabling and a deploy job, and it would replace the site on every deploy, so it keeps no history. Release assets would need a release (or a tag that moves) per push, and their URLs are awkward to fetch.git log origin/reports -- rust.bench.jsonis the series, and each commit's raw URL keeps pointing at its report. The files are a few KB, so the branch stays small.commit, themaincommit it measured, and the publish commit's message names it too. A report older than the one already published (its commit is a strict ancestor) is skipped, so a slow run can't replace a newer one. Bench runs onmainalso cancel superseded runs.pushtomain,scheduleandworkflow_dispatchfrommainpublish. The writing jobs are the only ones withcontents: write, and they're guarded bygithub.ref == 'refs/heads/main' && github.event_name != 'pull_request'. Pull requests build the reports as artifacts and dry-run the publish with a read-only token..github/scripts/publish-report.pywrites the commit with git plumbing (hash-object,mktree,commit-tree), so it needs no working tree. A push that loses a race is retried on the new head; each report is its own file, so the retries never conflict.--dry-rundoes everything except push. The branch gets aREADME.md(.github/reports-branch.md) saying it's written by CI.Benchmarks on
main(bench.yml)push: [main]andworkflow_dispatchtriggers. New jobpublish instruction counts: it benches the head alone, runsbench-report.pywithout the gate (no--limit), uploads the artifact and publishes. On a branch other thanmainit does a dry run.instruction counts) is unchanged apart fromif: github.event_name == 'pull_request', so it still runs, and only runs, on pull requests.bench-report.py: one wording change. With no--limit, the summary says "The head alone, with nothing to compare with." instead of claiming a base exists.Fuzz report (
fuzz.yml,fuzz-report.py)rust.fuzz.jsonis documented inruntime/fuzz/README.md("The report"):{ "runtime": "rust", "commit": "…", "toolchain": "rustc … nightly …", "date": "2026-10-26T03:41:07Z", "run": "https://github.com/…/actions/runs/…", "targets": { "decode_no_panic": { "seconds": 600, "executions": 18734210, "corpus": 2841, "findings": 0 } }, "proptest": { "cases": 100000, "properties": { "json_round_trips": "pass" } } }-print_final_stats=1and tees the log. ARecord the runstep (always(), before minimizing) turns the log into a fragment:executions:stat::number_of_executed_units.seconds: from libFuzzer'sDone N runs in S second(s), or the step's wall-clock time when a finding stopped it, since libFuzzer prints the stats on a crash but notDone.corpus: the number of inputs in the corpus after the run.findings: thecrash-/leak-/timeout-/oom-/slow-unit-files.test … ok|FAILEDline.fuzz report(!cancelled(), read-only) combines the fragments, writes the summary, uploads therust.fuzz.jsonartifact, and dry-runs the publish offmain. On a pull requestproptestisnull, because the long run isn't part of a pull request's run. A target whose job left no fragment gets anerrorentry withnullnumbers, not zeros.publish fuzz report(contents: write, only onmainand never on a pull request) publishes it.fuzz (<target>)are unchanged, so the required checks still match.Tested
fuzz-report.pyon real libFuzzer logs:framing_decode;-rss_limit_mb=40), where the stats are printed,Doneisn't, the seconds fall back to wall-clock, andfindings: 1.publish-report.pyagainst a bare clone of this repo:fuzz reportjob buildsrust.fuzz.jsonas an artifact and dry-runs the publish. Aworkflow_dispatchof both workflows on this branch runs the full nightly path (10 min per target plus proptest at 100k) and the benchmainjob, and each ends in a dry run.reports, i.e. thatGITHUB_TOKENwithcontents: writemay create and update the branch. The first push tomain(the merge itself) runs it for the bench report, and the next nightly does it for the fuzz report.The docs side is ComlineProject/docs#88, which reads these URLs.