diff --git a/.github/workflows/generator-support-drift.yml b/.github/workflows/language-support-drift.yml similarity index 53% rename from .github/workflows/generator-support-drift.yml rename to .github/workflows/language-support-drift.yml index 5f1bdf6..0d71eed 100644 --- a/.github/workflows/generator-support-drift.yml +++ b/.github/workflows/language-support-drift.yml @@ -1,28 +1,28 @@ -name: Generator support drift +name: Language support drift -# `reference/generator-support.md` carries a copy of generation's -# `FEATURES.md`: the matrix its conformance suite proves. This checks the copy -# is current. Like the IDL drift check it needs no Rust build, only a fetch, +# The Reference -> Languages pages are rendered from what the conformance +# suites prove: generation's `features.json`, and the runtime wire corpus +# with each runtime's report. This checks they're current. Like the IDL drift check it needs no Rust build, only a fetch, # so it runs per pull request. It's kept separate from the Docs workflow, # because a fetch from another repo shouldn't be able to block a deploy. on: schedule: - # A generator gaining a feature never touches this repo, so a + # A generator or runtime gaining a feature never touches this repo, so a # push-triggered run would not see it. Weekly catches that side. - cron: '45 7 * * 1' pull_request: paths: - - 'docs/docs/reference/generator-support.md' - - 'scripts/sync-generator-support.py' - - '.github/workflows/generator-support-drift.yml' + - 'docs/docs/reference/languages/**' + - 'scripts/sync-language-support.py' + - '.github/workflows/language-support-drift.yml' workflow_dispatch: permissions: contents: read concurrency: - group: generator-support-drift-${{ github.ref }} + group: language-support-drift-${{ github.ref }} cancel-in-progress: true jobs: @@ -38,9 +38,9 @@ jobs: - name: Check for drift shell: bash run: | - if ! python3 scripts/sync-generator-support.py --check | tee drift.txt; then + if ! python3 scripts/sync-language-support.py --check | tee drift.txt; then { - echo '### Generator support drift' + echo '### Language support drift' echo echo '```' cat drift.txt diff --git a/docs/docs/design/core-target-contract.md b/docs/docs/design/core-target-contract.md index 8b34f91..67517c7 100644 --- a/docs/docs/design/core-target-contract.md +++ b/docs/docs/design/core-target-contract.md @@ -201,6 +201,8 @@ built and tested (no more `todo!()` on this path): **A package-wide default** ([generation#17](https://github.com/ComlineProject/generation/pull/17) + [comline-rust#10](https://github.com/ComlineProject/comline-rust/pull/10) + [cli#27](https://github.com/ComlineProject/cli/pull/27)). `comline.toml`'s `[generate] default_framing` (per-`[[generate.target]]` overridable) rides `GenRequest.default_framing` to the generator and applies to every protocol with no `@framing` of its own. Resolution is `@framing` → package default → datagram. The contract crate treats the value as an opaque name; the target generator validates it. +**The wire is pinned by examples, not only by this prose.** The [wire corpus](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance) in the `runtime` repository holds byte-exact vectors for the name hash, the handshake, the envelope, both framings and both wire formats, including bytes that must be refused. They're the normative examples: where this page and a vector disagree, the vector is what peers speak. They're computed from `comline-runtime`, so a change to them is a wire change and shows in review. Every runtime proves its support against them; what each one passes is on its [language reference page](../reference/languages/index.md#runtimes-side-by-side). + Still open, to fix as part of this design: - The `Alloc` seam for owned bits (`.to_owned()`, decoded collection spines). diff --git a/docs/docs/guide/idl/validator.md b/docs/docs/guide/idl/validator.md index c6a4cd0..6b497ea 100644 --- a/docs/docs/guide/idl/validator.md +++ b/docs/docs/guide/idl/validator.md @@ -7,7 +7,7 @@ without a default, or an undeclared `params.*` reference all fail the build. Generated **Rust** code then [runs them](#in-generated-code). Generated TypeScript doesn't yet: it generates the types without the checks, and `comline generate` warns. See - [Generator support](../../reference/generator-support.md). + [TypeScript's language reference](../../reference/languages/typescript.md#code-generation). A **validator** is a named, parameterised check you attach to a [struct](structure.md) or [error](error.md) field, or to a diff --git a/docs/docs/languages/rust/framing-and-wire-format.md b/docs/docs/languages/rust/framing-and-wire-format.md index 3637107..6b120a9 100644 --- a/docs/docs/languages/rust/framing-and-wire-format.md +++ b/docs/docs/languages/rust/framing-and-wire-format.md @@ -72,6 +72,9 @@ the same format. ## Shipped combinations +What the runtime is proven to speak, feature by feature, is on the +[Rust reference page](../../reference/languages/rust.md#runtime). + | Framing | Wire format | Ships | |---|---|---| | `datagram` | `json` | ✅ | diff --git a/docs/docs/languages/typescript/framing-and-wire-format.md b/docs/docs/languages/typescript/framing-and-wire-format.md index 1e688d1..b5324f8 100644 --- a/docs/docs/languages/typescript/framing-and-wire-format.md +++ b/docs/docs/languages/typescript/framing-and-wire-format.md @@ -35,6 +35,9 @@ const client = await ChatClient.connect(transport, new JsonCodec()); ## Shipped combinations +What the runtime is proven to speak, feature by feature, is on the +[TypeScript reference page](../../reference/languages/typescript.md#runtime). + Narrower than Rust's, for the same reason — no MessagePack codec yet: | Framing | Wire format | Ships | diff --git a/docs/docs/reference/cli.md b/docs/docs/reference/cli.md index 4eff182..439fd26 100644 --- a/docs/docs/reference/cli.md +++ b/docs/docs/reference/cli.md @@ -40,7 +40,7 @@ parse → resolve imports → validate → (freeze into .comline/) → (generate | `comline clean [--dry-run]` | Remove generated code (the `generate` output root or files). Never touches `.comline/`. | | `comline reset [--force] [--dry-run]` | **Destructive.** Delete `.comline/` — the whole version history, no undo — *and* generated code. Next `build` starts over at `0.0.1`. On a terminal it shows how many versions will be lost and makes you type `reset`; without a terminal it refuses (exit `2`) unless `--force`. | | `comline completions ` | Print a completion script to stdout. `bash` `zsh` `fish` `powershell` `elvish`. | -| `comline targets [--features [--json]]` | Print one `language#version` per line to stdout — the token syntax `config.idp`'s `code_generation.languages` expects — for every generator this binary was compiled with. Ignores `--path`: it reflects the build, not a project. `--features` also lists what each generator supports, feature by feature ([Generator support](generator-support.md)); `--json` prints that as JSON. | +| `comline targets [--features [--json]]` | Print one `language#version` per line to stdout — the token syntax `config.idp`'s `code_generation.languages` expects — for every generator this binary was compiled with. Ignores `--path`: it reflects the build, not a project. `--features` also lists what each generator supports, feature by feature ([Languages](languages/index.md)); `--json` prints that as JSON. | `--out` / `--layout` / `--mode` override one target and need `--target` when several are configured. `COMLINE_GENERATE_OUT` / `_LAYOUT` / `_MODE` do the same diff --git a/docs/docs/reference/generator-support.md b/docs/docs/reference/generator-support.md deleted file mode 100644 index f8a8552..0000000 --- a/docs/docs/reference/generator-support.md +++ /dev/null @@ -1,75 +0,0 @@ -# Generator support - -What each code generator implements, feature by feature. The table below is -the one the generators' conformance suite produces, so every entry in it has -been checked, not just claimed. - -## What the states mean - -| State | Generating a schema that uses the feature | -|---|---| -| ✅ **supported** | Works. | -| 🟡 **partial** | Works, with the limit the note states. | -| ⚪ **ignored** | Succeeds, but the output is what it would be without the feature: the types are right, the feature has no effect. `comline generate` warns, and so does the editor. | -| ❌ **unsupported** | Is refused, with an error naming every use. The editor warns at each use. | - -## How a claim is proven - -Each generator states its support in a `SUPPORT` table in its own crate. The -conformance suite in -[`ComlineProject/generation`](https://github.com/ComlineProject/generation) -runs every generator over a corpus of schemas and checks each claim against -what the generator actually does: - -- **supported** or **partial**: a schema in the corpus uses the feature, - generates, and matches its checked-in output. Where the feature can be taken - out of the schema, the output must change without it; otherwise the feature - is really being ignored. -- **ignored**: the output is identical with the feature taken out, apart from - the IR hash, which names the schema. -- **unsupported**: every schema using the feature is refused, with an error - naming the feature. - -A claim that no schema in the corpus can back fails the suite. So a new -feature, or a generator that gains one, comes with a schema that proves it. -The proving schemas are named in each cell below. - -## The matrix - - - - -| Feature | rust 1.70.0 | typescript 5.0 | -|---|---|---| -| structs
`structs` | ✅ supported
proven by `primitives`, `optional_and_arrays`, `type_refs` +8 more | ✅ supported
proven by `primitives`, `optional_and_arrays`, `type_refs` +10 more | -| enums
`enums` | ✅ supported
proven by `enum_basic` | ✅ supported
proven by `enum_basic` | -| optional fields
`optional-fields` | ✅ supported
proven by `optional_and_arrays`, `type_refs`, `shapes` +3 more | ✅ supported
proven by `optional_and_arrays`, `type_refs`, `shapes` +4 more | -| arrays
`arrays` | ✅ supported
proven by `optional_and_arrays`, `type_refs`, `shapes` +3 more | ✅ supported
proven by `optional_and_arrays`, `type_refs`, `shapes` +4 more | -| fixed-length arrays
`fixed-arrays` | ✅ supported
proven by `shapes` | 🟡 partial: rendered as `T[]`: the length isn't part of the type
proven by `shapes`, `large_fixed_arrays_nested` | -| fixed arrays over 32 elements, nested
`large-fixed-arrays-nested` | ❌ unsupported: a fixed array over 32 elements renders only as a struct or error field's own, non-optional type (serde's derive stops at 32)
proven by `large_fixed_arrays_nested` | 🟡 partial: rendered as `T[]`: the length isn't part of the type
proven by `large_fixed_arrays_nested` | -| unions
`unions` | ❌ unsupported: Rust has no anonymous sum type to render one as
proven by `unions` | ✅ supported
proven by `unions` | -| type aliases
`type-aliases` | ✅ supported
proven by `validators`, `validators_on_decode` | ✅ supported
proven by `validators`, `validators_on_decode`, `unions` | -| constants
`constants` | ✅ supported
proven by `constant`, `shapes`, `validators` +1 more | ✅ supported
proven by `constant`, `shapes`, `validators` +1 more | -| protocols
`protocols` | ✅ supported
proven by `protocol`, `error_shapes`, `validators` +2 more | ✅ supported
proven by `protocol`, `error_shapes`, `validators` +2 more | -| errors
`errors` | ✅ supported
proven by `protocol`, `error_shapes` | ✅ supported
proven by `protocol`, `error_shapes` | -| functions that throw
`throws` | ✅ supported
proven by `protocol`, `error_shapes` | ✅ supported
proven by `protocol`, `error_shapes` | -| validators
`validators` | ✅ supported
proven by `validators`, `validators_on_decode` | ⚪ ignored: the types are generated; the checks aren't run yet ([core#96](https://github.com/ComlineProject/core/issues/96))
proven by `validators`, `validators_on_decode` | -| imports between schemas
`imports` | ✅ supported
proven by `cross_schema`, `import_alias`, `import_glob` +4 more | ✅ supported
proven by `cross_schema`, `import_alias`, `import_glob` +4 more | -| nested namespaces
`nested-namespaces` | ✅ supported
proven by `nested_namespace`, `external_std`, `validators` +1 more | ✅ supported
proven by `nested_namespace`, `external_std`, `validators` +1 more | -| `lib` mode
`lib-mode` | ✅ supported
proven by `nested_namespace`, `external_std` | ✅ supported
proven by `nested_namespace`, `external_std` | -| `external_std`
`external-std` | ✅ supported
proven by `external_std` | ✅ supported
proven by `external_std` | -| `validate_on_decode`
`validate-on-decode` | ✅ supported
proven by `validators_on_decode` | ⚪ ignored: no checks are generated to run on decode yet ([core#96](https://github.com/ComlineProject/core/issues/96))
proven by `validators_on_decode` | -| JSON-RPC framing
`framing-jsonrpc` | ✅ supported
proven by `framing_jsonrpc` | ✅ supported
proven by `framing_jsonrpc` | - - - -## Elsewhere - -- `comline targets --features` prints this for the generators your `comline` - was built with, and `--json` prints it as JSON (see the [CLI](cli.md)). -- In the editor, a schema using a feature a language declared in `config.idp` - can't generate gets a warning at the use. Hovering the language in - `config.idp` shows what its generator supports. -- Each generator's repository README carries its own column: - [comline-rust](https://github.com/ComlineProject/comline-rust#features), - [comline-typescript](https://github.com/ComlineProject/comline-typescript#features). diff --git a/docs/docs/reference/languages/index.md b/docs/docs/reference/languages/index.md new file mode 100644 index 0000000..79556c1 --- /dev/null +++ b/docs/docs/reference/languages/index.md @@ -0,0 +1,120 @@ +# Languages + +What Comline supports in each target language. Every language has its own +page, with what its **code generator** emits and what its **runtime** speaks +on the wire. The matrices below compare the languages side by side. + +Nothing here is written by hand. Each entry comes from a conformance suite, +which checks the claim against what the generator actually does, so a +language page can't claim something that isn't true. + +## What the states mean + +| State | Generating a schema that uses the feature | +|---|---| +| ✅ **supported** | Works. | +| 🟡 **partial** | Works, with the limit the note states. | +| ⚪ **ignored** | Succeeds, but the output is what it would be without the feature: the types are right, the feature has no effect. `comline generate` warns, and so does the editor. | +| ❌ **unsupported** | Is refused, with an error naming every use. The editor warns at each use. | + +A runtime's claims use the same words, about the wire: + +| State | The runtime, for that wire feature | +|---|---| +| ✅ **supported** | Reproduces every example in the wire corpus, and decodes each back. | +| 🟡 **partial** | Does the same, except for the examples the note lists, which it's known to get wrong. | +| ❌ **unsupported** | Doesn't implement it. A peer must agree on something else at the handshake. | + +## How a claim is proven + +Each generator states its support in a `SUPPORT` table in its own crate. The +conformance suite in +[`ComlineProject/generation`](https://github.com/ComlineProject/generation) +runs every generator over a corpus of schemas and checks each claim against +what the generator actually does: + +- **supported** or **partial**: a schema in the corpus uses the feature, + generates, and matches its checked-in output. Where the feature can be taken + out of the schema, the output must change without it; otherwise the feature + is really being ignored. +- **ignored**: the output is identical with the feature taken out, apart from + the IR hash, which names the schema. +- **unsupported**: every schema using the feature is refused, with an error + naming the feature. + +A claim that no schema in the corpus can back fails the suite. So a new +feature, or a generator that gains one, comes with a schema that proves it. +The proving schemas are named in each cell below. + +Runtimes are proven against a shared wire corpus in +[`ComlineProject/runtime`](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance): +for each wire feature, examples of exact bytes and the input that produces +them, plus bytes that must be refused. The Rust runtime is the reference that +the examples are taken from; every other runtime vendors them at a pinned +commit. A runtime's suite runs every example against it: + +- **supported**: every example encodes to its bytes and decodes back, and + every malformed one is refused. JSON compares as JSON, with numbers exact, + so `1` and `1.0` are equal but a rounded large integer isn't. +- **partial**: as for supported, except for the listed examples, which must + still fail. When one starts passing, the suite fails until the claim is + upgraded. +- **unsupported**: the runtime reports every example as not implemented. + +## Code generation, side by side + + + + +| Feature | [rust 1.70.0](rust.md) | [typescript 5.0](typescript.md) | +|---|---|---| +| structs
`structs` | ✅ supported | ✅ supported | +| enums
`enums` | ✅ supported | ✅ supported | +| optional fields
`optional-fields` | ✅ supported | ✅ supported | +| arrays
`arrays` | ✅ supported | ✅ supported | +| fixed-length arrays
`fixed-arrays` | ✅ supported | 🟡 partial: rendered as `T[]`: the length isn't part of the type | +| fixed arrays over 32 elements, nested
`large-fixed-arrays-nested` | ❌ unsupported: a fixed array over 32 elements renders only as a struct or error field's own, non-optional type (serde's derive stops at 32) | 🟡 partial: rendered as `T[]`: the length isn't part of the type | +| unions
`unions` | ❌ unsupported: Rust has no anonymous sum type to render one as | ✅ supported | +| type aliases
`type-aliases` | ✅ supported | ✅ supported | +| constants
`constants` | ✅ supported | ✅ supported | +| protocols
`protocols` | ✅ supported | ✅ supported | +| errors
`errors` | ✅ supported | ✅ supported | +| functions that throw
`throws` | ✅ supported | ✅ supported | +| validators
`validators` | ✅ supported | ⚪ ignored: the types are generated; the checks aren't run yet ([core#96](https://github.com/ComlineProject/core/issues/96)) | +| imports between schemas
`imports` | ✅ supported | ✅ supported | +| nested namespaces
`nested-namespaces` | ✅ supported | ✅ supported | +| `lib` mode
`lib-mode` | ✅ supported | ✅ supported | +| `external_std`
`external-std` | ✅ supported | ✅ supported | +| `validate_on_decode`
`validate-on-decode` | ✅ supported | ⚪ ignored: no checks are generated to run on decode yet ([core#96](https://github.com/ComlineProject/core/issues/96)) | +| JSON-RPC framing
`framing-jsonrpc` | ✅ supported | ✅ supported | + + + +## Runtimes, side by side + + + + +| Feature | [rust](rust.md#runtime) | [typescript](typescript.md#runtime) | +|---|---|---| +| name hash
`name-hash` | ✅ supported | ✅ supported | +| handshake
`handshake` | ✅ supported | ✅ supported | +| envelope
`envelope` | ✅ supported | ✅ supported | +| datagram framing
`framing-datagram` | ✅ supported | ✅ supported | +| JSON-RPC framing
`framing-jsonrpc` | ✅ supported | ✅ supported | +| JSON wire format
`format-json` | ✅ supported | 🟡 partial: 64-bit integers are `number`, exact only below 2^53 ([comline-typescript#49](https://github.com/ComlineProject/comline-typescript/issues/49)) | +| MessagePack wire format
`format-msgpack` | ✅ supported | ❌ unsupported: only JSON ships | + + + +## Elsewhere + +- `comline targets --features` prints the code generation support for the + generators your `comline` was built with, and `--json` prints it as JSON (see + the [CLI](../cli.md)). +- In the editor, a schema using a feature that a language declared in + `config.idp` can't generate gets a warning at the use. Hovering the language + in `config.idp` shows what its generator supports. +- Each generator's repository README carries its own table: + [comline-rust](https://github.com/ComlineProject/comline-rust#features), + [comline-typescript](https://github.com/ComlineProject/comline-typescript#features). diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md new file mode 100644 index 0000000..0bbc53c --- /dev/null +++ b/docs/docs/reference/languages/rust.md @@ -0,0 +1,63 @@ +# Rust + +What Comline supports in Rust, as its conformance suites prove it. For how +to use it, see the [Rust guide](../../languages/rust/index.md). The states are explained in the +[overview](index.md#what-the-states-mean). + +## Code generation + +What the [`comline-rust`](https://github.com/ComlineProject/comline-rust) generator emits +for each language feature, and the conformance fixtures that prove it. + + + + +Generator `rust#1.70.0`. + +| Feature | What it covers | State | Proven by | +|---|---|---|---| +| structs
`structs` | `struct` declarations | ✅ supported | `primitives`, `optional_and_arrays`, `type_refs`, `shapes`, `cross_schema`, `import_alias`, `import_glob`, `nested_namespace`, `external_std`, `validators`, `validators_on_decode` | +| enums
`enums` | `enum` declarations | ✅ supported | `enum_basic` | +| optional fields
`optional-fields` | fields marked `optional` | ✅ supported | `optional_and_arrays`, `type_refs`, `shapes`, `cross_schema`, `validators`, `validators_on_decode` | +| arrays
`arrays` | variable-length arrays, `T[]` | ✅ supported | `optional_and_arrays`, `type_refs`, `shapes`, `cross_schema`, `validators`, `validators_on_decode` | +| fixed-length arrays
`fixed-arrays` | fixed-length arrays, `T[N]` | ✅ supported | `shapes` | +| fixed arrays over 32 elements, nested
`large-fixed-arrays-nested` | a `T[N]` with N over 32 anywhere but a field's own, non-optional type: inside another array, optional, an alias, an argument or a return | ❌ unsupported: a fixed array over 32 elements renders only as a struct or error field's own, non-optional type (serde's derive stops at 32) | `large_fixed_arrays_nested` | +| unions
`unions` | `union(A B ...)` types | ❌ unsupported: Rust has no anonymous sum type to render one as | `unions` | +| type aliases
`type-aliases` | `type Name = ...` declarations | ✅ supported | `validators`, `validators_on_decode` | +| constants
`constants` | `const` declarations | ✅ supported | `constant`, `shapes`, `validators`, `validators_on_decode` | +| protocols
`protocols` | `protocol` declarations and their functions | ✅ supported | `protocol`, `error_shapes`, `validators`, `validators_on_decode`, `framing_jsonrpc` | +| errors
`errors` | `error` declarations | ✅ supported | `protocol`, `error_shapes` | +| functions that throw
`throws` | functions declared with `! Error` | ✅ supported | `protocol`, `error_shapes` | +| validators
`validators` | `@validators` on fields and type aliases, checked in generated code | ✅ supported | `validators`, `validators_on_decode` | +| imports between schemas
`imports` | `use` of another schema's declarations | ✅ supported | `cross_schema`, `import_alias`, `import_glob`, `nested_namespace`, `external_std`, `validators`, `validators_on_decode` | +| nested namespaces
`nested-namespaces` | schemas in subdirectories (`chat::admin`) | ✅ supported | `nested_namespace`, `external_std`, `validators`, `validators_on_decode` | +| `lib` mode
`lib-mode` | `--mode lib`: a buildable package, not bare source files | ✅ supported | `nested_namespace`, `external_std` | +| `external_std`
`external-std` | `[generate] external_std`: `std` referenced as its own package | ✅ supported | `external_std` | +| `validate_on_decode`
`validate-on-decode` | `[generate] validate_on_decode`: decoding runs the checks | ✅ supported | `validators_on_decode` | +| JSON-RPC framing
`framing-jsonrpc` | `@framing = "jsonrpc"` (or as the package default) | ✅ supported | `framing_jsonrpc` | + + + +## Runtime + +What the [`comline-runtime`](https://github.com/ComlineProject/runtime) runtime speaks on the wire, proven against the +shared [wire corpus](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance): +byte-exact examples of the handshake, the envelopes, each framing and each +wire format. The claims are in [`rust.json`](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/support/rust.json). + + + + +Runtime `rust`. + +| Feature | What it covers | State | Proven by | +|---|---|---|---| +| name hash
`name-hash` | FNV-1a 64 of a wire format or framing name, as the handshake carries it | ✅ supported | [6 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/name-hash.json) | +| handshake
`handshake` | the fixed 31-byte frame each side sends first: magic, version, IR hash, wire format, framing, capabilities | ✅ supported | [7 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/handshake.json) | +| envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | +| datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | +| JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | +| JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | ✅ supported | [24 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | +| MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ✅ supported | [24 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-msgpack.json) | + + diff --git a/docs/docs/reference/languages/typescript.md b/docs/docs/reference/languages/typescript.md new file mode 100644 index 0000000..b808605 --- /dev/null +++ b/docs/docs/reference/languages/typescript.md @@ -0,0 +1,63 @@ +# TypeScript + +What Comline supports in TypeScript, as its conformance suites prove it. For how +to use it, see the [TypeScript guide](../../languages/typescript/index.md). The states are explained in the +[overview](index.md#what-the-states-mean). + +## Code generation + +What the [`comline-typescript`](https://github.com/ComlineProject/comline-typescript) generator emits +for each language feature, and the conformance fixtures that prove it. + + + + +Generator `typescript#5.0`. + +| Feature | What it covers | State | Proven by | +|---|---|---|---| +| structs
`structs` | `struct` declarations | ✅ supported | `primitives`, `optional_and_arrays`, `type_refs`, `shapes`, `cross_schema`, `import_alias`, `import_glob`, `nested_namespace`, `external_std`, `validators`, `validators_on_decode`, `unions`, `large_fixed_arrays_nested` | +| enums
`enums` | `enum` declarations | ✅ supported | `enum_basic` | +| optional fields
`optional-fields` | fields marked `optional` | ✅ supported | `optional_and_arrays`, `type_refs`, `shapes`, `cross_schema`, `validators`, `validators_on_decode`, `large_fixed_arrays_nested` | +| arrays
`arrays` | variable-length arrays, `T[]` | ✅ supported | `optional_and_arrays`, `type_refs`, `shapes`, `cross_schema`, `validators`, `validators_on_decode`, `large_fixed_arrays_nested` | +| fixed-length arrays
`fixed-arrays` | fixed-length arrays, `T[N]` | 🟡 partial: rendered as `T[]`: the length isn't part of the type | `shapes`, `large_fixed_arrays_nested` | +| fixed arrays over 32 elements, nested
`large-fixed-arrays-nested` | a `T[N]` with N over 32 anywhere but a field's own, non-optional type: inside another array, optional, an alias, an argument or a return | 🟡 partial: rendered as `T[]`: the length isn't part of the type | `large_fixed_arrays_nested` | +| unions
`unions` | `union(A B ...)` types | ✅ supported | `unions` | +| type aliases
`type-aliases` | `type Name = ...` declarations | ✅ supported | `validators`, `validators_on_decode`, `unions` | +| constants
`constants` | `const` declarations | ✅ supported | `constant`, `shapes`, `validators`, `validators_on_decode` | +| protocols
`protocols` | `protocol` declarations and their functions | ✅ supported | `protocol`, `error_shapes`, `validators`, `validators_on_decode`, `framing_jsonrpc` | +| errors
`errors` | `error` declarations | ✅ supported | `protocol`, `error_shapes` | +| functions that throw
`throws` | functions declared with `! Error` | ✅ supported | `protocol`, `error_shapes` | +| validators
`validators` | `@validators` on fields and type aliases, checked in generated code | ⚪ ignored: the types are generated; the checks aren't run yet ([core#96](https://github.com/ComlineProject/core/issues/96)) | `validators`, `validators_on_decode` | +| imports between schemas
`imports` | `use` of another schema's declarations | ✅ supported | `cross_schema`, `import_alias`, `import_glob`, `nested_namespace`, `external_std`, `validators`, `validators_on_decode` | +| nested namespaces
`nested-namespaces` | schemas in subdirectories (`chat::admin`) | ✅ supported | `nested_namespace`, `external_std`, `validators`, `validators_on_decode` | +| `lib` mode
`lib-mode` | `--mode lib`: a buildable package, not bare source files | ✅ supported | `nested_namespace`, `external_std` | +| `external_std`
`external-std` | `[generate] external_std`: `std` referenced as its own package | ✅ supported | `external_std` | +| `validate_on_decode`
`validate-on-decode` | `[generate] validate_on_decode`: decoding runs the checks | ⚪ ignored: no checks are generated to run on decode yet ([core#96](https://github.com/ComlineProject/core/issues/96)) | `validators_on_decode` | +| JSON-RPC framing
`framing-jsonrpc` | `@framing = "jsonrpc"` (or as the package default) | ✅ supported | `framing_jsonrpc` | + + + +## Runtime + +What the [`@comline/runtime`](https://github.com/ComlineProject/comline-typescript/tree/main/runtime) runtime speaks on the wire, proven against the +shared [wire corpus](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance): +byte-exact examples of the handshake, the envelopes, each framing and each +wire format. The claims are in [`conformance-support.json`](https://github.com/ComlineProject/comline-typescript/blob/main/runtime/conformance-support.json). + + + + +Runtime `typescript`. + +| Feature | What it covers | State | Proven by | +|---|---|---|---| +| name hash
`name-hash` | FNV-1a 64 of a wire format or framing name, as the handshake carries it | ✅ supported | [6 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/name-hash.json) | +| handshake
`handshake` | the fixed 31-byte frame each side sends first: magic, version, IR hash, wire format, framing, capabilities | ✅ supported | [7 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/handshake.json) | +| envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | +| datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | +| JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | +| JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | 🟡 partial: 64-bit integers are `number`, exact only below 2^53 ([comline-typescript#49](https://github.com/ComlineProject/comline-typescript/issues/49))
fails: `format-json/u64-above-2-53`, `format-json/u64-max`, `format-json/i64-min` | [21 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | +| MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ❌ unsupported: only JSON ships | — | + + diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 7d77b2e..3601192 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -52,6 +52,11 @@ nav: - Framing & wire format: 'languages/typescript/framing-and-wire-format.md' - Reference: + - Languages: + - Overview: 'reference/languages/index.md' + - Rust: 'reference/languages/rust.md' + - TypeScript: 'reference/languages/typescript.md' + - Congregation: - Congregation: 'reference/congregation/index.md' - comline.toml: 'reference/congregation/comline-toml.md' @@ -64,7 +69,6 @@ nav: - Tools: - Tools: 'reference/tools/index.md' - CLI: 'reference/cli.md' - - Generator support: 'reference/generator-support.md' - Design: diff --git a/scripts/sync-generator-support.py b/scripts/sync-generator-support.py deleted file mode 100644 index a2ef6eb..0000000 --- a/scripts/sync-generator-support.py +++ /dev/null @@ -1,96 +0,0 @@ -#!/usr/bin/env python3 -"""Keep the generator support page in step with what conformance proved. - -`ComlineProject/generation`'s conformance suite checks every generator's -feature claims against its corpus and writes the result to `FEATURES.md` at -the repository root. This copies that file's matrix into -`docs/docs/reference/generator-support.md`, between its -`` / `` markers, so the page says -exactly what was proven. The issue references in it (`core#96`) become links. - - python3 scripts/sync-generator-support.py # rewrite the page - python3 scripts/sync-generator-support.py --check # exit 1 on drift - python3 scripts/sync-generator-support.py --file ../generation/FEATURES.md - -Stdlib only: no Rust toolchain needed, so CI runs it per pull request. -""" - -from __future__ import annotations - -import argparse -import re -import sys -import urllib.error -import urllib.request -from pathlib import Path - -ROOT = Path(__file__).resolve().parent.parent -PAGE = ROOT / "docs/docs/reference/generator-support.md" -RAW = "https://raw.githubusercontent.com/ComlineProject/generation/{rev}/FEATURES.md" -START = "" -END = "" - -# Repositories an issue reference may name; anything else is left as text -# (`rust#1.70.0` is a target, not an issue). -REPOS = ("core", "generation", "cli", "comline-rust", "comline-typescript", "language-server", "runtime", "docs") -ISSUE_RE = re.compile(r"\b(" + "|".join(re.escape(r) for r in REPOS) + r")#(\d+)\b(?![\d.])") - - -def read_features(file: Path | None, rev: str) -> str: - if file: - return file.read_text(encoding="utf-8") - url = RAW.format(rev=rev) - try: - with urllib.request.urlopen(url, timeout=30) as response: - return response.read().decode("utf-8") - except urllib.error.HTTPError as error: - sys.exit(f"error: fetching {url} failed ({error.code} {error.reason})") - except urllib.error.URLError as error: - sys.exit(f"error: fetching {url} failed ({error.reason})") - - -def section(features: str) -> str: - """The marked section the page should hold: the matrix, issues linked.""" - table = [line for line in features.splitlines() if line.startswith("|")] - if not table: - sys.exit("error: FEATURES.md has no table") - linked = "\n".join( - ISSUE_RE.sub(lambda m: f"[{m[0]}](https://github.com/ComlineProject/{m[1]}/issues/{m[2]})", line) - for line in table - ) - return ( - f"{START}\n" - "\n\n" - f"{linked}\n\n{END}" - ) - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument("--check", action="store_true", help="exit 1 if the page is out of date") - parser.add_argument("--file", type=Path, help="read FEATURES.md from here instead of fetching it") - parser.add_argument("--rev", default="master", help="generation revision to fetch (default: master)") - args = parser.parse_args() - - page = PAGE.read_text(encoding="utf-8") - start, end = page.find(START), page.find(END) - if start < 0 or end < 0: - sys.exit(f"error: {PAGE.relative_to(ROOT)} has no {START} ... {END} section") - current = page[start : end + len(END)] - wanted = section(read_features(args.file, args.rev)) - - if current == wanted: - print("generator support page is up to date") - return 0 - if args.check: - print(f"{PAGE.relative_to(ROOT)} is out of date with generation's FEATURES.md.") - print("Run: python3 scripts/sync-generator-support.py") - return 1 - PAGE.write_text(page[:start] + wanted + page[end + len(END) :], encoding="utf-8") - print(f"updated {PAGE.relative_to(ROOT)}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/sync-language-support.py b/scripts/sync-language-support.py new file mode 100644 index 0000000..3a495f7 --- /dev/null +++ b/scripts/sync-language-support.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python3 +"""Keep the Reference → Languages pages in step with what conformance proved. + +Two conformance suites feed the pages: + +- `ComlineProject/generation`'s checks every code generator's feature claims + and writes `features.json` at its root; +- `ComlineProject/runtime`'s wire corpus (`runtime/conformance/`) checks every + runtime's claims; each runtime writes a report (see `RUNTIMES`). + +This renders, on `reference/languages/`: + +- `index.md`: the code generation matrix (`` ... + ``) and the runtime matrix (`runtime-matrix`); +- `.md`: that language's code generation table (`codegen`) and runtime + table (`runtime`). + +Issue references (`core#96`) become links. A generator or runtime with no page +of its own is an error that names the page to add. + + python3 scripts/sync-language-support.py # rewrite the pages + python3 scripts/sync-language-support.py --check # exit 1 on drift + python3 scripts/sync-language-support.py --file ../generation/features.json \\ + --runtime-features ../runtime/runtime/conformance/features.json \\ + --runtime-report rust=../runtime/runtime/conformance/reports/rust.json + +Stdlib only: no Rust toolchain needed, so CI runs it per pull request. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +import urllib.error +import urllib.request +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +PAGES = ROOT / "docs/docs/reference/languages" +RAW = "https://raw.githubusercontent.com/ComlineProject/{repo}/{rev}/{path}" +NOTICE = ( + "" +) +RUNTIME_NOTICE = ( + "" +) + +# The wire corpus, and where each language's runtime writes its report: +# (repository, path in it). +CORPUS = ("runtime", "runtime/conformance/features.json") +VECTORS = "https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/{id}.json" +RUNTIMES = { + "rust": ("runtime", "runtime/conformance/reports/rust.json"), + "typescript": ("comline-typescript", "runtime/conformance-report.json"), +} + +# Repositories an issue reference may name; anything else is left as text +# (`rust#1.70.0` is a target, not an issue). +REPOS = ("core", "generation", "cli", "comline-rust", "comline-typescript", "language-server", "runtime", "docs") +ISSUE_RE = re.compile(r"\b(" + "|".join(re.escape(r) for r in REPOS) + r")#(\d+)\b(?![\d.])") + +STATES = { + "supported": "✅ supported", + "partial": "🟡 partial", + "ignored": "⚪ ignored", + "unsupported": "❌ unsupported", +} + + +def read_json(file: Path | None, repo: str, rev: str, path: str) -> dict: + if file: + return json.loads(file.read_text(encoding="utf-8")) + url = RAW.format(repo=repo, rev=rev, path=path) + try: + with urllib.request.urlopen(url, timeout=30) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + sys.exit(f"error: fetching {url} failed ({error.code} {error.reason})") + except urllib.error.URLError as error: + sys.exit(f"error: fetching {url} failed ({error.reason})") + + +def link_issues(text: str) -> str: + return ISSUE_RE.sub(lambda m: f"[{m[0]}](https://github.com/ComlineProject/{m[1]}/issues/{m[2]})", text) + + +def cell(text: str) -> str: + return link_issues(text).replace("|", "\\|") + + +def state_with_detail(claim: dict) -> str: + out = STATES[claim["state"]] + if claim.get("note"): + out += f": {claim['note']}" + if claim.get("tracking"): + out += f" ({claim['tracking']})" + return cell(out) + + +def proof(claim: dict) -> str: + return ", ".join(f"`{fixture}`" for fixture in claim.get("proven_by") or []) or "—" + + +def matrix(report: dict) -> str: + generators = report["generators"] + lines = ["| Feature | " + " | ".join(f"[{g['name']} {g['version']}]({g['name']}.md)" for g in generators) + " |"] + lines.append("|---|" + "---|" * len(generators)) + for feature in report["features"]: + row = [f"{feature['title']}
`{feature['id']}`"] + row += [state_with_detail(g["support"][feature["id"]]) for g in generators] + lines.append("| " + " | ".join(row) + " |") + return "\n".join(lines) + + +def language_table(report: dict, generator: dict) -> str: + lines = [ + f"Generator `{generator['name']}#{generator['version']}`.", + "", + "| Feature | What it covers | State | Proven by |", + "|---|---|---|---|", + ] + for feature in report["features"]: + claim = generator["support"][feature["id"]] + lines.append( + f"| {feature['title']}
`{feature['id']}` | {cell(feature['description'])} " + f"| {state_with_detail(claim)} | {proof(claim)} |" + ) + return "\n".join(lines) + + +def runtime_state(claim: dict) -> str: + out = state_with_detail(claim) + if claim.get("failing"): + out += "
fails: " + ", ".join(f"`{case}`" for case in claim["failing"]) + return out + + +def runtime_proof(feature: str, claim: dict) -> str: + n = len(claim.get("proven_by") or []) + if not n: + return "—" + return f"[{n} case{'s' if n != 1 else ''}]({VECTORS.format(id=feature)})" + + +def runtime_matrix(corpus: dict, reports: dict[str, dict]) -> str: + lines = ["| Feature | " + " | ".join(f"[{name}]({name}.md#runtime)" for name in reports) + " |"] + lines.append("|---|" + "---|" * len(reports)) + for feature in corpus["features"]: + row = [f"{feature['title']}
`{feature['id']}`"] + row += [state_with_detail(report["support"][feature["id"]]) for report in reports.values()] + lines.append("| " + " | ".join(row) + " |") + return "\n".join(lines) + + +def runtime_table(corpus: dict, report: dict) -> str: + lines = [ + f"Runtime `{report['runtime']}`.", + "", + "| Feature | What it covers | State | Proven by |", + "|---|---|---|---|", + ] + for feature in corpus["features"]: + claim = report["support"][feature["id"]] + lines.append( + f"| {feature['title']}
`{feature['id']}` | {cell(feature['description'])} " + f"| {runtime_state(claim)} | {runtime_proof(feature['id'], claim)} |" + ) + return "\n".join(lines) + + +def replace_section(text: str, page: Path, name: str, body: str, notice: str = NOTICE) -> str: + """`text` of `page` with its `name` section set to `body`.""" + start, end = f"", f"" + i, j = text.find(start), text.find(end) + if i < 0 or j < 0: + sys.exit(f"error: {page.relative_to(ROOT)} has no {start} ... {end} section") + return text[:i] + f"{start}\n{notice}\n\n{body}\n\n{end}" + text[j + len(end) :] + + +def page_for(name: str, section: str, what: str) -> Path: + page = PAGES / f"{name}.md" + if not page.exists(): + sys.exit( + f"error: {what} `{name}` has no page - add {page.relative_to(ROOT)} " + f"with a `` / `` section, and list it in mkdocs.yml" + ) + return page + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--check", action="store_true", help="exit 1 if a page is out of date") + parser.add_argument("--file", type=Path, help="read generation's features.json from here instead of fetching it") + parser.add_argument("--rev", default="master", help="generation revision to fetch (default: master)") + parser.add_argument("--runtime-features", type=Path, help="read the wire corpus's features.json from here") + parser.add_argument( + "--runtime-report", + action="append", + default=[], + metavar="NAME=FILE", + help="read runtime NAME's conformance report from FILE (repeatable)", + ) + parser.add_argument("--runtime-rev", default="main", help="revision of the runtime repositories to fetch (default: main)") + args = parser.parse_args() + + local_reports = dict(arg.split("=", 1) for arg in args.runtime_report) + for name in local_reports: + if name not in RUNTIMES: + sys.exit(f"error: unknown runtime `{name}` (known: {', '.join(RUNTIMES)})") + + report = read_json(args.file, "generation", args.rev, "features.json") + corpus = read_json(args.runtime_features, CORPUS[0], args.runtime_rev, CORPUS[1]) + runtimes = { + name: read_json(Path(local_reports[name]) if name in local_reports else None, repo, args.runtime_rev, path) + for name, (repo, path) in RUNTIMES.items() + } + + # Every page's wanted text, starting from what's on disk; a page gets + # its code generation and runtime sections in turn. + wanted: dict[Path, str] = {} + + def update(page: Path, section: str, body: str, notice: str = NOTICE) -> None: + text = wanted.get(page) or page.read_text(encoding="utf-8") + wanted[page] = replace_section(text, page, section, body, notice) + + index = PAGES / "index.md" + update(index, "matrix", matrix(report)) + update(index, "runtime-matrix", runtime_matrix(corpus, runtimes), RUNTIME_NOTICE) + for generator in report["generators"]: + update(page_for(generator["name"], "codegen", "generator"), "codegen", language_table(report, generator)) + for name, runtime in runtimes.items(): + update(page_for(name, "runtime", "runtime"), "runtime", runtime_table(corpus, runtime), RUNTIME_NOTICE) + + stale = [(page, text) for page, text in wanted.items() if page.read_text(encoding="utf-8") != text] + if not stale: + print("language reference pages are up to date") + return 0 + if args.check: + for page, _ in stale: + print(f"{page.relative_to(ROOT)} is out of date with the conformance results.") + print("Run: python3 scripts/sync-language-support.py") + return 1 + for page, text in stale: + page.write_text(text, encoding="utf-8") + print(f"updated {page.relative_to(ROOT)}") + return 0 + + +if __name__ == "__main__": + sys.exit(main())