Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ on:
# Pages silently stale with no error anywhere. This rebuilds and
# redeploys from `main` daily regardless, so staleness self-heals
# within a day instead of needing someone to notice.
#
# It also refreshes the verification results rendered at build time (see
# "Render the verification reports"): the runtime's nightly fuzz run
# publishes around 03:30 UTC, so this picks it up the same morning.
- cron: '0 6 * * *'

permissions:
Expand Down Expand Up @@ -39,6 +43,17 @@ jobs:
working-directory: docs
run: poetry install

# The runtimes' benchmark and fuzzing results change with every runtime
# push and nightly run, so they're fetched now instead of committed;
# the pages include what this writes (docs/_generated/, ignored). A
# report that can't be fetched renders as "not published yet", so this
# never fails the build; the timeout and continue-on-error are a
# second guard.
- name: Render the verification reports
timeout-minutes: 2
continue-on-error: true
run: python3 scripts/sync-language-support.py --render-reports

- name: Build site
working-directory: docs
run: poetry run zensical build --strict
Expand Down
3 changes: 3 additions & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,6 @@ site/
# Python bytecode (pygments-comline is a real installed package now)
__pycache__/
*.pyc

# Rendered at build time (scripts/sync-language-support.py --render-reports)
_generated/
8 changes: 8 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@ poetry run zensical serve -o # live preview at http://localhost:8000
poetry run zensical build # static output in site/
```

The runtimes' benchmark and fuzzing results on Reference → Languages are
fetched at build time, not committed. Without this first, those sections are
empty:

```sh
python3 ../scripts/sync-language-support.py --render-reports # writes _generated/ (ignored)
```

`mkdocs` / `mkdocs-material` are kept installed as a fallback during
Zensical's alpha; `poetry run mkdocs serve` still works against the same
`mkdocs.yml`.
Expand Down
12 changes: 10 additions & 2 deletions docs/docs/conduct/branch-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,15 @@ repositories follow as they gain gates.
- `instruction counts`: no benchmark grows more than 1% unless the pull
request accepts it.

## The `reports` branch: `runtime`

- Holds the latest `rust.bench.json` and `rust.fuzz.json`. The docs fetch
them each time the site builds.
- Not the default branch, so no ruleset covers it.
- Written only by CI: the bench workflow on every push to the default branch,
the fuzz workflow nightly. Never by a pull request's run.
- One commit per report, never force-pushed, so its history is the series.

## Review

- A pull request needs one approval, from the code owner where it touches an
Expand All @@ -32,10 +41,9 @@ repositories follow as they gain gates.

Listed in [`.github/CODEOWNERS`](https://github.com/ComlineProject/runtime/blob/main/.github/CODEOWNERS):

- `.github/`, all of it: the workflows, their scripts, and `CODEOWNERS`.
- `runtime/core/benches/accepted-regressions.toml`: the ledger of accepted
regressions.
- `.github/scripts/bench-report.py`, `.github/workflows/bench.yml`: the gate.
- `.github/CODEOWNERS`.

## Accepting a regression

Expand Down
8 changes: 4 additions & 4 deletions docs/docs/proposals/shared-runtime-core/verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,10 +157,10 @@ Open issues the work turned up:
[Rust](../../reference/languages/rust.md#verification) and
[TypeScript](../../reference/languages/typescript.md#verification) pages show
each runtime's verification, and TypeScript's divergence classes are
rendered from `known-divergences.json`. Benchmark results fill in when a
runtime checks in its `*.bench.json` (Rust's workflow only uploads it as an
artifact so far). Fuzz results wait on a `*.fuzz.json` that no runtime
writes yet.
rendered from `known-divergences.json`. Rust's CI publishes
`rust.bench.json` (every push to `main`) and `rust.fuzz.json` (nightly) to
the runtime's `reports` branch, and the pages render them. TypeScript's
benchmark results fill in when it checks in its `conformance-bench.json`.
- **A re-baseline path for the gate.** A correctness fix that costs time needs
a deliberate way to move the baseline (see [Budgets](#budgets)). It isn't
built yet.
Expand Down
23 changes: 12 additions & 11 deletions docs/docs/reference/languages/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Runtime `rust`.
| envelope <br><sub>`envelope`</sub> | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) |
| datagram framing <br><sub>`framing-datagram`</sub> | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) |
| datagram@2 framing <br><sub>`framing-datagram2`</sub> | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) |
| JSON-RPC framing <br><sub>`framing-jsonrpc`</sub> | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [11 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) |
| JSON-RPC framing <br><sub>`framing-jsonrpc`</sub> | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) |
| JSON wire format <br><sub>`format-json`</sub> | values as UTF-8 JSON: a struct as an object | ✅ supported | [46 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) |
| MessagePack wire format <br><sub>`format-msgpack`</sub> | values as MessagePack: a struct as an array | ✅ supported | [45 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-msgpack.json) |
| packed wire format <br><sub>`format-packed`</sub> | values as packed@1: fixed-width fields back to back, varint lengths, no tags | ✅ supported | [59 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-packed.json) |
Expand All @@ -87,9 +87,12 @@ are compared against (see [Verification](../../proposals/shared-runtime-core/ver
nightly with the property tests at 100,000 cases. A nightly failure opens an
issue. Each finding becomes a `found-*` vector, so every runtime checks it.

Fuzz results aren't published as a report file yet (`reports/rust.fuzz.json`
doesn't exist), so there are no executions or corpus sizes to show here. They
will appear once the runtime publishes them.
The nightly run publishes its results as `rust.fuzz.json` on the runtime's
[`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch
(format: [the fuzz README](https://github.com/ComlineProject/runtime/blob/main/runtime/fuzz/README.md#the-report)).
The latest, fetched when this site was built:

--8<-- "_generated/reports/rust.fuzz.md"

### Divergences

Expand All @@ -103,11 +106,9 @@ The Rust runtime is the reference the others are compared against, so it has no
### Benchmarks

Instruction counts for the formats, framings, handshake and a connection step,
gated on every pull request against its base.

<!-- bench:start -->
<!-- Rendered from each runtime's verification reports by scripts/sync-language-support.py - don't edit by hand. -->

No benchmark results are published yet: they will appear here once the runtime checks in [`runtime/conformance/reports/rust.bench.json`](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/reports/rust.bench.json).
gated on every pull request against its base. Every push to `main` benches the
new head and publishes `rust.bench.json` on the runtime's
[`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch.
The latest, fetched when this site was built:

<!-- bench:end -->
--8<-- "_generated/reports/rust.bench.md"
12 changes: 3 additions & 9 deletions docs/docs/reference/languages/typescript.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Runtime `typescript`.
| envelope <br><sub>`envelope`</sub> | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) |
| datagram framing <br><sub>`framing-datagram`</sub> | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) |
| datagram@2 framing <br><sub>`framing-datagram2`</sub> | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) |
| JSON-RPC framing <br><sub>`framing-jsonrpc`</sub> | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [11 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) |
| JSON-RPC framing <br><sub>`framing-jsonrpc`</sub> | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) |
| JSON wire format <br><sub>`format-json`</sub> | values as UTF-8 JSON: a struct as an object | 🟡 partial: decoding doesn't check enum variants or integer ranges ([comline-typescript#57](https://github.com/ComlineProject/comline-typescript/issues/57)) <br>fails: `format-json/decode-unknown-variant`, `format-json/decode-wider-integer-overflow` | [44 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) |
| MessagePack wire format <br><sub>`format-msgpack`</sub> | values as MessagePack: a struct as an array | ❌ unsupported: JSON and packed@1 ship | — |
| packed wire format <br><sub>`format-packed`</sub> | values as packed@1: fixed-width fields back to back, varint lengths, no tags | ✅ supported | [59 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-packed.json) |
Expand Down Expand Up @@ -101,16 +101,10 @@ Where `typescript` and the reference are known to answer differently, from [`kno
| `json-decode-unchecked-shape` | json@1 decoding accepts the wrong JSON kind, a missing or repeated struct field, an integer spelled 1.0, 1e0 or -0, and a float past its range; the reference refuses them | [comline-typescript#73](https://github.com/ComlineProject/comline-typescript/issues/73) |
| `json-encode-f32-digits` | json@1 writes an f32 with the shortest digits of a double (1.401298464324817e-45), the reference with those of the f32 (1e-45): the same f32, not the same JSON number | [comline-typescript#74](https://github.com/ComlineProject/comline-typescript/issues/74) |
| `jsonrpc-decode-lenient` | jsonrpc@1 decoding accepts a declared member twice, and an id or error code spelled 1.0, 1e0 or -0; the reference refuses both | [comline-typescript#75](https://github.com/ComlineProject/comline-typescript/issues/75) |
| `reference-jsonrpc-method-escapes` | the reference refuses a JSON-RPC method name written with any escape, and writes a name that needs one unescaped; TypeScript escapes, and reads any spelling | [runtime#41](https://github.com/ComlineProject/runtime/issues/41) |
| `reference-jsonrpc-ignored-members-unvalidated` | the reference accepts invalid UTF-8 inside JSON-RPC members it ignores (jsonrpc, error.message, undeclared ones); TypeScript refuses any frame that isn't UTF-8 | [runtime#42](https://github.com/ComlineProject/runtime/issues/42) |
| `connection-replies-in-any-order` | Connection hands a reply to the call it answers in any order; an ordered (nested, blocking) Rust State refuses a reply that skips the innermost waiting call as a framing error. TypeScript has only the async model | [comline-typescript#77](https://github.com/ComlineProject/comline-typescript/issues/77) |

<!-- divergences:end -->

### Benchmarks

<!-- bench:start -->
<!-- Rendered from each runtime's verification reports by scripts/sync-language-support.py - don't edit by hand. -->

No benchmark results are published yet: they will appear here once the runtime checks in [`runtime/conformance-bench.json`](https://github.com/ComlineProject/comline-typescript/blob/main/runtime/conformance-bench.json).

<!-- bench:end -->
--8<-- "_generated/reports/typescript.bench.md"
Loading
Loading