From f80b8530ca4b3ab6c26376532462075399aba63d Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 03:32:22 +0800 Subject: [PATCH 1/3] docs: read Rust's bench and fuzz reports from the runtime's reports branch The runtime's CI now publishes rust.bench.json (every push to main) and rust.fuzz.json (nightly) on its unprotected `reports` branch, since main only takes pull requests. The sync script fetches them from there, and renders the fuzz report in a new `fuzz` section on the Rust page. Also picks up the conformance drift already on runtime and comline-typescript main (JSON-RPC case count, a new divergence class). --- docs/docs/conduct/branch-rules.md | 9 ++ .../shared-runtime-core/verification.md | 8 +- docs/docs/reference/languages/rust.md | 23 +++- docs/docs/reference/languages/typescript.md | 1 + scripts/sync-language-support.py | 127 ++++++++++++++---- 5 files changed, 129 insertions(+), 39 deletions(-) diff --git a/docs/docs/conduct/branch-rules.md b/docs/docs/conduct/branch-rules.md index a969b7d..e0d7878 100644 --- a/docs/docs/conduct/branch-rules.md +++ b/docs/docs/conduct/branch-rules.md @@ -21,6 +21,15 @@ repositories follow as they gain gates. - `instruction counts`: no benchmark grows more than 1% unless the pull request accepts it. +## The `reports` branch: `runtime` + +- Holds the latest `rust.bench.json` and `rust.fuzz.json`, which the + language reference reads. +- Not the default branch, so no ruleset covers it. +- Written only by CI: the bench workflow on every push to the default branch, + the fuzz workflow nightly. Never by a pull request's run. +- One commit per report, never force-pushed, so its history is the series. + ## Review - A pull request needs one approval, from the code owner where it touches an diff --git a/docs/docs/proposals/shared-runtime-core/verification.md b/docs/docs/proposals/shared-runtime-core/verification.md index f8fad7e..5d08464 100644 --- a/docs/docs/proposals/shared-runtime-core/verification.md +++ b/docs/docs/proposals/shared-runtime-core/verification.md @@ -157,10 +157,10 @@ Open issues the work turned up: [Rust](../../reference/languages/rust.md#verification) and [TypeScript](../../reference/languages/typescript.md#verification) pages show each runtime's verification, and TypeScript's divergence classes are - rendered from `known-divergences.json`. Benchmark results fill in when a - runtime checks in its `*.bench.json` (Rust's workflow only uploads it as an - artifact so far). Fuzz results wait on a `*.fuzz.json` that no runtime - writes yet. + rendered from `known-divergences.json`. Rust's CI publishes + `rust.bench.json` (every push to `main`) and `rust.fuzz.json` (nightly) to + the runtime's `reports` branch, and the pages render them. TypeScript's + benchmark results fill in when it checks in its `conformance-bench.json`. - **A re-baseline path for the gate.** A correctness fix that costs time needs a deliberate way to move the baseline (see [Budgets](#budgets)). It isn't built yet. diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md index 037557d..6f559a9 100644 --- a/docs/docs/reference/languages/rust.md +++ b/docs/docs/reference/languages/rust.md @@ -63,7 +63,7 @@ Runtime `rust`. | envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | | datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | | datagram@2 framing
`framing-datagram2` | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) | -| JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [11 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | +| JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | | JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | ✅ supported | [46 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | | MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ✅ supported | [45 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-msgpack.json) | | packed wire format
`format-packed` | values as packed@1: fixed-width fields back to back, varint lengths, no tags | ✅ supported | [59 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-packed.json) | @@ -87,9 +87,17 @@ are compared against (see [Verification](../../proposals/shared-runtime-core/ver nightly with the property tests at 100,000 cases. A nightly failure opens an issue. Each finding becomes a `found-*` vector, so every runtime checks it. -Fuzz results aren't published as a report file yet (`reports/rust.fuzz.json` -doesn't exist), so there are no executions or corpus sizes to show here. They -will appear once the runtime publishes them. +The nightly run publishes its results as `rust.fuzz.json` on the runtime's +[`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch +(format: [the fuzz README](https://github.com/ComlineProject/runtime/blob/main/runtime/fuzz/README.md#the-report)). +The latest: + + + + +No fuzzing results are published yet: they will appear here once the runtime's CI publishes [`rust.fuzz.json`](https://github.com/ComlineProject/runtime/blob/reports/rust.fuzz.json) on its `reports` branch. + + ### Divergences @@ -103,11 +111,14 @@ The Rust runtime is the reference the others are compared against, so it has no ### Benchmarks Instruction counts for the formats, framings, handshake and a connection step, -gated on every pull request against its base. +gated on every pull request against its base. Every push to `main` benches the +new head and publishes `rust.bench.json` on the runtime's +[`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch. +The latest: -No benchmark results are published yet: they will appear here once the runtime checks in [`runtime/conformance/reports/rust.bench.json`](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/reports/rust.bench.json). +No benchmark results are published yet: they will appear here once the runtime's CI publishes [`rust.bench.json`](https://github.com/ComlineProject/runtime/blob/reports/rust.bench.json) on its `reports` branch. diff --git a/docs/docs/reference/languages/typescript.md b/docs/docs/reference/languages/typescript.md index 6903737..20b53be 100644 --- a/docs/docs/reference/languages/typescript.md +++ b/docs/docs/reference/languages/typescript.md @@ -103,6 +103,7 @@ Where `typescript` and the reference are known to answer differently, from [`kno | `jsonrpc-decode-lenient` | jsonrpc@1 decoding accepts a declared member twice, and an id or error code spelled 1.0, 1e0 or -0; the reference refuses both | [comline-typescript#75](https://github.com/ComlineProject/comline-typescript/issues/75) | | `reference-jsonrpc-method-escapes` | the reference refuses a JSON-RPC method name written with any escape, and writes a name that needs one unescaped; TypeScript escapes, and reads any spelling | [runtime#41](https://github.com/ComlineProject/runtime/issues/41) | | `reference-jsonrpc-ignored-members-unvalidated` | the reference accepts invalid UTF-8 inside JSON-RPC members it ignores (jsonrpc, error.message, undeclared ones); TypeScript refuses any frame that isn't UTF-8 | [runtime#42](https://github.com/ComlineProject/runtime/issues/42) | +| `connection-replies-in-any-order` | Connection hands a reply to the call it answers in any order; an ordered (nested, blocking) Rust State refuses a reply that skips the innermost waiting call as a framing error. TypeScript has only the async model | [comline-typescript#77](https://github.com/ComlineProject/comline-typescript/issues/77) | diff --git a/scripts/sync-language-support.py b/scripts/sync-language-support.py index 0aa1cfa..ec69a67 100644 --- a/scripts/sync-language-support.py +++ b/scripts/sync-language-support.py @@ -10,18 +10,21 @@ Each runtime's verification results sit next to its support table: the divergence classes a runtime is known to have from the reference -(`known-divergences.json`, see `DIVERGENCES`) and its benchmark results -(`.bench.json`, see `BENCHES`). A report that isn't published yet is not -an error: the page says so, and the section fills in on the first run after the -runtime publishes it. Fuzz results have no report file to read yet, so the -pages describe the fuzzing in prose only. +(`known-divergences.json`, see `DIVERGENCES`), its benchmark results +(`.bench.json`, see `BENCHES`) and its fuzzing results +(`.fuzz.json`, see `FUZZ`). A report that isn't published yet is not an +error: the page says so, and the section fills in on the first run after the +runtime publishes it. Rust's CI publishes its bench and fuzz reports to the +runtime repository's `reports` branch, not to `main`, which only takes pull +requests. This renders, on `reference/languages/`: - `index.md`: the code generation matrix (`` ... ``) and the runtime matrix (`runtime-matrix`); - `.md`: that language's code generation table (`codegen`) and runtime - table (`runtime`), divergences (`divergences`) and benchmarks (`bench`). + table (`runtime`), divergences (`divergences`), benchmarks (`bench`) and, + for a runtime in `FUZZ`, fuzzing (`fuzz`). Issue references (`core#96`) become links. A generator or runtime with no page of its own is an error that names the page to add. @@ -32,7 +35,7 @@ --runtime-features ../runtime/runtime/conformance/features.json \\ --runtime-report rust=../runtime/runtime/conformance/reports/rust.json \\ --divergences typescript=../comline-typescript/runtime/known-divergences.json \\ - --bench rust=../runtime/runtime/conformance/reports/rust.bench.json + --bench rust=rust.bench.json --fuzz rust=rust.fuzz.json Stdlib only: no Rust toolchain needed, so CI runs it per pull request. """ @@ -68,19 +71,25 @@ "typescript": ("comline-typescript", "runtime/conformance-report.json"), } -# Verification results. A runtime with no entry has none to show (Rust is the -# reference the others are compared against, so it has no divergences file). -# A report that isn't there yet (HTTP 404) is shown as "not published yet". -# `rust.bench.json` is what the runtime's bench workflow writes, but it only -# uploads it as an artifact; the runtime has to check it in for it to appear. +# Verification results: (repository, revision, path in it); a revision of +# None is `--runtime-rev`. A runtime with no entry has none to show (Rust is +# the reference the others are compared against, so it has no divergences +# file). A report that isn't there yet (HTTP 404) is shown as "not published +# yet". Rust's are on the runtime repository's `reports` branch: its CI +# commits them there from `main` (bench, every push) and the nightly fuzz run, +# since `main` itself only takes pull requests. +REPORTS_BRANCH = "reports" DIVERGENCES = { - "typescript": ("comline-typescript", "runtime/known-divergences.json"), + "typescript": ("comline-typescript", None, "runtime/known-divergences.json"), } BENCHES = { - "rust": ("runtime", "runtime/conformance/reports/rust.bench.json"), - "typescript": ("comline-typescript", "runtime/conformance-bench.json"), + "rust": ("runtime", REPORTS_BRANCH, "rust.bench.json"), + "typescript": ("comline-typescript", None, "runtime/conformance-bench.json"), } -REPORT_URL = "https://github.com/ComlineProject/{repo}/blob/main/{path}" +FUZZ = { + "rust": ("runtime", REPORTS_BRANCH, "rust.fuzz.json"), +} +REPORT_URL = "https://github.com/ComlineProject/{repo}/blob/{rev}/{path}" VERIFY_NOTICE = ( "" @@ -224,11 +233,11 @@ def divergences_table(name: str, report: dict | None) -> str: if name == "rust" else "No divergences are tracked for this runtime." ) - repo, path = DIVERGENCES[name] + repo, rev, path = DIVERGENCES[name] if report is None: return f"No divergence list is published yet: it will appear here once the runtime checks in `{path}`." classes = report["classes"] - source = f"[`{path.rsplit('/', 1)[-1]}`]({REPORT_URL.format(repo=repo, path=path)})" + source = f"[`{path.rsplit('/', 1)[-1]}`]({REPORT_URL.format(repo=repo, rev=rev or 'main', path=path)})" if not classes: return f"No known divergences from the reference ({source})." lines = [ @@ -243,14 +252,18 @@ def divergences_table(name: str, report: dict | None) -> str: return "\n".join(lines) +def not_published(what: str, repo: str, rev: str | None, path: str) -> str: + link = f"[`{path}`]({REPORT_URL.format(repo=repo, rev=rev or 'main', path=path)})" + if rev is None: + return f"No {what} are published yet: they will appear here once the runtime checks in {link}." + return f"No {what} are published yet: they will appear here once the runtime's CI publishes {link} on its `{rev}` branch." + + def bench_table(name: str, report: dict | None) -> str: - repo, path = BENCHES[name] - source = REPORT_URL.format(repo=repo, path=path) + repo, rev, path = BENCHES[name] + source = REPORT_URL.format(repo=repo, rev=rev or "main", path=path) if report is None: - return ( - "No benchmark results are published yet: they will appear here once the runtime " - f"checks in [`{path}`]({source})." - ) + return not_published("benchmark results", repo, rev, path) metric = {"instructions": "instructions executed"}.get(report.get("metric"), report.get("metric", "value")) lines = [ f"Measured at runtime commit `{report['commit'][:7]}` with `{report['toolchain']}`, " @@ -266,6 +279,47 @@ def bench_table(name: str, report: dict | None) -> str: return "\n".join(lines) +def fuzz_table(name: str, report: dict | None) -> str: + repo, rev, path = FUZZ[name] + source = REPORT_URL.format(repo=repo, rev=rev or "main", path=path) + if report is None: + return not_published("fuzzing results", repo, rev, path) + + def number(n: int | None) -> str: + return "—" if n is None else f"{n:,}" + + lines = [ + f"From the run of {report['date'][:10]} at runtime commit `{report['commit'][:7]}`, " + f"with `{report['toolchain']}` ([run]({report['run']}), " + f"[`{path.rsplit('/', 1)[-1]}`]({source})).", + "", + "| Target | Seconds | Executions | Corpus | Findings |", + "|---|---:|---:|---:|---:|", + ] + for target, t in sorted(report["targets"].items()): + findings = "didn't run" if t.get("error") else number(t["findings"]) + lines.append( + f"| `{target}` | {number(t['seconds'])} | {number(t['executions'])} " + f"| {number(t['corpus'])} | {findings} |" + ) + prop = report.get("proptest") + lines.append("") + if prop is None: + lines.append("This run had no long property-test run.") + elif prop.get("error") or not prop["properties"]: + lines.append(f"The property tests at {prop['cases']:,} cases didn't run.") + else: + failed = [p for p, verdict in sorted(prop["properties"].items()) if verdict != "pass"] + n = len(prop["properties"]) + verdict = ( + "all pass" + if not failed + else f"{len(failed)} fail{'s' if len(failed) == 1 else ''}: " + ", ".join(f"`{p}`" for p in failed) + ) + lines.append(f"Property tests: {n} properties at {prop['cases']:,} cases each, {verdict}.") + return "\n".join(lines) + + def replace_section(text: str, page: Path, name: str, body: str, notice: str = NOTICE) -> str: """`text` of `page` with its `name` section set to `body`.""" start, end = f"", f"" @@ -298,11 +352,20 @@ def main() -> int: metavar="NAME=FILE", help="read runtime NAME's conformance report from FILE (repeatable)", ) - for flag, what in (("--divergences", "known-divergences.json"), ("--bench", "benchmark report")): + for flag, what in ( + ("--divergences", "known-divergences.json"), + ("--bench", "benchmark report"), + ("--fuzz", "fuzzing report"), + ): parser.add_argument( flag, action="append", default=[], metavar="NAME=FILE", help=f"read runtime NAME's {what} from FILE (repeatable)" ) - parser.add_argument("--runtime-rev", default="main", help="revision of the runtime repositories to fetch (default: main)") + parser.add_argument( + "--runtime-rev", + default="main", + help="revision of the runtime repositories to fetch (default: main); reports on the " + f"`{REPORTS_BRANCH}` branch are always fetched from it", + ) args = parser.parse_args() local_reports = dict(arg.split("=", 1) for arg in args.runtime_report) @@ -311,7 +374,11 @@ def main() -> int: sys.exit(f"error: unknown runtime `{name}` (known: {', '.join(RUNTIMES)})") local = {} - for flag, known, given in (("--divergences", DIVERGENCES, args.divergences), ("--bench", BENCHES, args.bench)): + for flag, known, given in ( + ("--divergences", DIVERGENCES, args.divergences), + ("--bench", BENCHES, args.bench), + ("--fuzz", FUZZ, args.fuzz), + ): local[flag] = dict(arg.split("=", 1) for arg in given) for name in local[flag]: if name not in known: @@ -327,9 +394,9 @@ def main() -> int: def optional(flag: str, known: dict, name: str) -> dict | None: if name not in known: return None - repo, path = known[name] + repo, rev, path = known[name] given = local[flag].get(name) - return read_optional_json(Path(given) if given else None, repo, args.runtime_rev, path) + return read_optional_json(Path(given) if given else None, repo, rev or args.runtime_rev, path) # Every page's wanted text, starting from what's on disk; a page gets # its code generation and runtime sections in turn. @@ -349,6 +416,8 @@ def update(page: Path, section: str, body: str, notice: str = NOTICE) -> None: update(page, "runtime", runtime_table(corpus, runtime), RUNTIME_NOTICE) update(page, "divergences", divergences_table(name, optional("--divergences", DIVERGENCES, name)), VERIFY_NOTICE) update(page, "bench", bench_table(name, optional("--bench", BENCHES, name)), VERIFY_NOTICE) + if name in FUZZ: + update(page, "fuzz", fuzz_table(name, optional("--fuzz", FUZZ, name)), VERIFY_NOTICE) stale = [(page, text) for page, text in wanted.items() if page.read_text(encoding="utf-8") != text] if not stale: From 7c5d00c300e88d4b0ddb3b79b1485d8f65fe774c Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 03:56:27 +0800 Subject: [PATCH 2/3] docs: render the runtimes' bench and fuzz results at build time They change with every runtime push and nightly run, so committing them made the weekly drift check flag the pages almost every week. The Docs workflow now runs sync-language-support.py --render-reports before the build, writing docs/_generated/reports/ (ignored), which the pages include with pymdownx.snippets. A report that 404s, times out (10 s) or doesn't parse renders as "not published yet"; the build never fails on it. --check no longer covers those sections. The daily scheduled build picks up new numbers, and a repository_dispatch (reports-published) lets another repository ask for a rebuild. Branch rules: runtime's owned files are now all of .github/ and the accepted-regressions ledger. --- .github/workflows/docs.yml | 20 +++++ docs/.gitignore | 3 + docs/README.md | 8 ++ docs/docs/conduct/branch-rules.md | 7 +- docs/docs/reference/languages/rust.md | 18 +---- docs/docs/reference/languages/typescript.md | 11 +-- scripts/sync-language-support.py | 83 ++++++++++++++++++--- 7 files changed, 113 insertions(+), 37 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 35f8365..e16357b 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -11,7 +11,16 @@ on: # Pages silently stale with no error anywhere. This rebuilds and # redeploys from `main` daily regardless, so staleness self-heals # within a day instead of needing someone to notice. + # + # It also refreshes the verification results rendered at build time (see + # "Render the verification reports"): the runtime's nightly fuzz run + # publishes around 03:30 UTC, so this picks it up the same morning. - cron: '0 6 * * *' + # Lets another repository's workflow ask for a rebuild once it publishes a + # report (`gh api repos/ComlineProject/docs/dispatches -f event_type=reports-published`, + # with a token that may write to this repository). + repository_dispatch: + types: [reports-published] permissions: contents: read @@ -39,6 +48,17 @@ jobs: working-directory: docs run: poetry install + # The runtimes' benchmark and fuzzing results change with every runtime + # push and nightly run, so they're fetched now instead of committed; + # the pages include what this writes (docs/_generated/, ignored). A + # report that can't be fetched renders as "not published yet", so this + # never fails the build; the timeout and continue-on-error are a + # second guard. + - name: Render the verification reports + timeout-minutes: 2 + continue-on-error: true + run: python3 scripts/sync-language-support.py --render-reports + - name: Build site working-directory: docs run: poetry run zensical build --strict diff --git a/docs/.gitignore b/docs/.gitignore index 3ca7271..16f1432 100755 --- a/docs/.gitignore +++ b/docs/.gitignore @@ -5,3 +5,6 @@ site/ # Python bytecode (pygments-comline is a real installed package now) __pycache__/ *.pyc + +# Rendered at build time (scripts/sync-language-support.py --render-reports) +_generated/ diff --git a/docs/README.md b/docs/README.md index e8cc0fc..2099ac7 100755 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,14 @@ poetry run zensical serve -o # live preview at http://localhost:8000 poetry run zensical build # static output in site/ ``` +The runtimes' benchmark and fuzzing results on Reference → Languages are +fetched at build time, not committed. Without this first, those sections are +empty: + +```sh +python3 ../scripts/sync-language-support.py --render-reports # writes _generated/ (ignored) +``` + `mkdocs` / `mkdocs-material` are kept installed as a fallback during Zensical's alpha; `poetry run mkdocs serve` still works against the same `mkdocs.yml`. diff --git a/docs/docs/conduct/branch-rules.md b/docs/docs/conduct/branch-rules.md index e0d7878..b0c8ddf 100644 --- a/docs/docs/conduct/branch-rules.md +++ b/docs/docs/conduct/branch-rules.md @@ -23,8 +23,8 @@ repositories follow as they gain gates. ## The `reports` branch: `runtime` -- Holds the latest `rust.bench.json` and `rust.fuzz.json`, which the - language reference reads. +- Holds the latest `rust.bench.json` and `rust.fuzz.json`. The docs fetch + them each time the site builds. - Not the default branch, so no ruleset covers it. - Written only by CI: the bench workflow on every push to the default branch, the fuzz workflow nightly. Never by a pull request's run. @@ -41,10 +41,9 @@ repositories follow as they gain gates. Listed in [`.github/CODEOWNERS`](https://github.com/ComlineProject/runtime/blob/main/.github/CODEOWNERS): +- `.github/`, all of it: the workflows, their scripts, and `CODEOWNERS`. - `runtime/core/benches/accepted-regressions.toml`: the ledger of accepted regressions. -- `.github/scripts/bench-report.py`, `.github/workflows/bench.yml`: the gate. -- `.github/CODEOWNERS`. ## Accepting a regression diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md index 6f559a9..98aebf8 100644 --- a/docs/docs/reference/languages/rust.md +++ b/docs/docs/reference/languages/rust.md @@ -90,14 +90,9 @@ are compared against (see [Verification](../../proposals/shared-runtime-core/ver The nightly run publishes its results as `rust.fuzz.json` on the runtime's [`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch (format: [the fuzz README](https://github.com/ComlineProject/runtime/blob/main/runtime/fuzz/README.md#the-report)). -The latest: +The latest, fetched when this site was built: - - - -No fuzzing results are published yet: they will appear here once the runtime's CI publishes [`rust.fuzz.json`](https://github.com/ComlineProject/runtime/blob/reports/rust.fuzz.json) on its `reports` branch. - - +--8<-- "_generated/reports/rust.fuzz.md" ### Divergences @@ -114,11 +109,6 @@ Instruction counts for the formats, framings, handshake and a connection step, gated on every pull request against its base. Every push to `main` benches the new head and publishes `rust.bench.json` on the runtime's [`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch. -The latest: - - - - -No benchmark results are published yet: they will appear here once the runtime's CI publishes [`rust.bench.json`](https://github.com/ComlineProject/runtime/blob/reports/rust.bench.json) on its `reports` branch. +The latest, fetched when this site was built: - +--8<-- "_generated/reports/rust.bench.md" diff --git a/docs/docs/reference/languages/typescript.md b/docs/docs/reference/languages/typescript.md index 20b53be..ee46c15 100644 --- a/docs/docs/reference/languages/typescript.md +++ b/docs/docs/reference/languages/typescript.md @@ -63,7 +63,7 @@ Runtime `typescript`. | envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | | datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | | datagram@2 framing
`framing-datagram2` | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) | -| JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [11 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | +| JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | | JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | 🟡 partial: decoding doesn't check enum variants or integer ranges ([comline-typescript#57](https://github.com/ComlineProject/comline-typescript/issues/57))
fails: `format-json/decode-unknown-variant`, `format-json/decode-wider-integer-overflow` | [44 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | | MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ❌ unsupported: JSON and packed@1 ship | — | | packed wire format
`format-packed` | values as packed@1: fixed-width fields back to back, varint lengths, no tags | ✅ supported | [59 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-packed.json) | @@ -101,17 +101,10 @@ Where `typescript` and the reference are known to answer differently, from [`kno | `json-decode-unchecked-shape` | json@1 decoding accepts the wrong JSON kind, a missing or repeated struct field, an integer spelled 1.0, 1e0 or -0, and a float past its range; the reference refuses them | [comline-typescript#73](https://github.com/ComlineProject/comline-typescript/issues/73) | | `json-encode-f32-digits` | json@1 writes an f32 with the shortest digits of a double (1.401298464324817e-45), the reference with those of the f32 (1e-45): the same f32, not the same JSON number | [comline-typescript#74](https://github.com/ComlineProject/comline-typescript/issues/74) | | `jsonrpc-decode-lenient` | jsonrpc@1 decoding accepts a declared member twice, and an id or error code spelled 1.0, 1e0 or -0; the reference refuses both | [comline-typescript#75](https://github.com/ComlineProject/comline-typescript/issues/75) | -| `reference-jsonrpc-method-escapes` | the reference refuses a JSON-RPC method name written with any escape, and writes a name that needs one unescaped; TypeScript escapes, and reads any spelling | [runtime#41](https://github.com/ComlineProject/runtime/issues/41) | -| `reference-jsonrpc-ignored-members-unvalidated` | the reference accepts invalid UTF-8 inside JSON-RPC members it ignores (jsonrpc, error.message, undeclared ones); TypeScript refuses any frame that isn't UTF-8 | [runtime#42](https://github.com/ComlineProject/runtime/issues/42) | | `connection-replies-in-any-order` | Connection hands a reply to the call it answers in any order; an ordered (nested, blocking) Rust State refuses a reply that skips the innermost waiting call as a framing error. TypeScript has only the async model | [comline-typescript#77](https://github.com/ComlineProject/comline-typescript/issues/77) | ### Benchmarks - - - -No benchmark results are published yet: they will appear here once the runtime checks in [`runtime/conformance-bench.json`](https://github.com/ComlineProject/comline-typescript/blob/main/runtime/conformance-bench.json). - - +--8<-- "_generated/reports/typescript.bench.md" diff --git a/scripts/sync-language-support.py b/scripts/sync-language-support.py index ec69a67..be56f41 100644 --- a/scripts/sync-language-support.py +++ b/scripts/sync-language-support.py @@ -13,18 +13,23 @@ (`known-divergences.json`, see `DIVERGENCES`), its benchmark results (`.bench.json`, see `BENCHES`) and its fuzzing results (`.fuzz.json`, see `FUZZ`). A report that isn't published yet is not an -error: the page says so, and the section fills in on the first run after the -runtime publishes it. Rust's CI publishes its bench and fuzz reports to the -runtime repository's `reports` branch, not to `main`, which only takes pull -requests. +error: the page says so, and the section fills in once the runtime publishes +it. Rust's CI publishes its bench and fuzz reports to the runtime repository's +`reports` branch, not to `main`, which only takes pull requests. + +The benchmark and fuzzing results change with every runtime push and every +nightly run, so they aren't committed: `--render-reports` writes them to +`docs/_generated/reports/.{bench,fuzz}.md` (ignored by git) at site +build time, and the pages include those files with `--8<--`. That mode never +fails on a fetch: a report that 404s, times out or doesn't parse renders as +"not published yet". The default mode and `--check` don't touch them. This renders, on `reference/languages/`: - `index.md`: the code generation matrix (`` ... ``) and the runtime matrix (`runtime-matrix`); - `.md`: that language's code generation table (`codegen`) and runtime - table (`runtime`), divergences (`divergences`), benchmarks (`bench`) and, - for a runtime in `FUZZ`, fuzzing (`fuzz`). + table (`runtime`), and divergences (`divergences`). Issue references (`core#96`) become links. A generator or runtime with no page of its own is an error that names the page to add. @@ -34,7 +39,9 @@ python3 scripts/sync-language-support.py --file ../generation/features.json \\ --runtime-features ../runtime/runtime/conformance/features.json \\ --runtime-report rust=../runtime/runtime/conformance/reports/rust.json \\ - --divergences typescript=../comline-typescript/runtime/known-divergences.json \\ + --divergences typescript=../comline-typescript/runtime/known-divergences.json + python3 scripts/sync-language-support.py --render-reports # at build time + python3 scripts/sync-language-support.py --render-reports \\ --bench rust=rust.bench.json --fuzz rust=rust.fuzz.json Stdlib only: no Rust toolchain needed, so CI runs it per pull request. @@ -52,6 +59,11 @@ ROOT = Path(__file__).resolve().parent.parent PAGES = ROOT / "docs/docs/reference/languages" +# Where `--render-reports` writes; the pages include it relative to `docs/`, +# which is `pymdownx.snippets`' base path when the site builds from there. +GENERATED = ROOT / "docs/_generated/reports" +# A build-time fetch gives up quickly: the build must not hang on GitHub. +BUILD_TIMEOUT = 10 RAW = "https://raw.githubusercontent.com/ComlineProject/{repo}/{rev}/{path}" NOTICE = ( "\n\n{body}\n", encoding="utf-8") + print(f"wrote {target.relative_to(ROOT)}") + + def replace_section(text: str, page: Path, name: str, body: str, notice: str = NOTICE) -> str: """`text` of `page` with its `name` section set to `body`.""" start, end = f"", f"" @@ -342,6 +395,15 @@ def page_for(name: str, section: str, what: str) -> Path: def main() -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--check", action="store_true", help="exit 1 if a page is out of date") + parser.add_argument( + "--render-reports", + nargs="?", + const=GENERATED, + type=Path, + metavar="DIR", + help=f"write the bench and fuzz sections for the site build (default: {GENERATED.relative_to(ROOT)}); " + "never fails on a fetch", + ) parser.add_argument("--file", type=Path, help="read generation's features.json from here instead of fetching it") parser.add_argument("--rev", default="master", help="generation revision to fetch (default: master)") parser.add_argument("--runtime-features", type=Path, help="read the wire corpus's features.json from here") @@ -384,6 +446,10 @@ def main() -> int: if name not in known: sys.exit(f"error: unknown runtime `{name}` for {flag} (known: {', '.join(known)})") + if args.render_reports: + render_reports(args.render_reports, args.runtime_rev, local) + return 0 + report = read_json(args.file, "generation", args.rev, "features.json") corpus = read_json(args.runtime_features, CORPUS[0], args.runtime_rev, CORPUS[1]) runtimes = { @@ -415,9 +481,6 @@ def update(page: Path, section: str, body: str, notice: str = NOTICE) -> None: page = page_for(name, "runtime", "runtime") update(page, "runtime", runtime_table(corpus, runtime), RUNTIME_NOTICE) update(page, "divergences", divergences_table(name, optional("--divergences", DIVERGENCES, name)), VERIFY_NOTICE) - update(page, "bench", bench_table(name, optional("--bench", BENCHES, name)), VERIFY_NOTICE) - if name in FUZZ: - update(page, "fuzz", fuzz_table(name, optional("--fuzz", FUZZ, name)), VERIFY_NOTICE) stale = [(page, text) for page, text in wanted.items() if page.read_text(encoding="utf-8") != text] if not stale: From 77c236d3d547cfa4c5a2b6dac78448e94dd6084b Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 04:00:45 +0800 Subject: [PATCH 3/3] ci: drop the repository_dispatch trigger; the daily build picks up new reports --- .github/workflows/docs.yml | 5 ----- 1 file changed, 5 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index e16357b..28fcbb5 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -16,11 +16,6 @@ on: # "Render the verification reports"): the runtime's nightly fuzz run # publishes around 03:30 UTC, so this picks it up the same morning. - cron: '0 6 * * *' - # Lets another repository's workflow ask for a rebuild once it publishes a - # report (`gh api repos/ComlineProject/docs/dispatches -f event_type=reports-published`, - # with a token that may write to this repository). - repository_dispatch: - types: [reports-published] permissions: contents: read