docs: documenting your schemas with comline doc

.github/workflows/schema-drift.yml

@@ -2,15 +2,17 @@ name: Schema drift
 
 # `docs/docs/schemas/comline-toml/0.x.json` is served on the site as the
 # `comline.toml` JSON Schema, which `comline new` points a `#:schema` line at.
-# It is a copy: generation derives it from `comline_codegen::config`. This
-# checks the copy still equals generation's file.
+# It is a copy: generation derives it from `comline_codegen::config`.
+# `docs/docs/schemas/doc-model/1.json` is the documentation model's schema,
+# a copy of core's `schemas/doc-model.json`. This checks both copies still
+# equal their sources.
 #
 # Separate from the Docs workflow for the same reason as the other drift
 # checks: a fetch against another repo has no business blocking a deploy.
 
 on:
   schedule:
-    # A schema change lands in `generation` and never touches this repo, so a
+    # A schema change lands in `generation` or `core` and never touches this repo, so a
     # push-triggered run would not see it. Weekly catches that side.
     - cron: '45 7 * * 1'
   pull_request:
@@ -37,7 +39,7 @@ jobs:
         with:
           python-version: "3.14"
 
-      - name: Compare with generation's schema
+      - name: Compare with generation's and core's schemas
         shell: bash
         run: |
           if ! python3 scripts/sync-comline-toml-schema.py | tee drift.txt; then

docs/docs/guide/documenting.md

@@ -0,0 +1,136 @@
+# Documenting your schemas
+
+`comline doc` turns a package's docs into pages: the `//!` headers of
+`config.idp` and of each schema, every declaration's `///` docstring, and what
+each protocol resolves to on the wire. Nothing is written by hand, and
+everything the package declares is documented.
+
+## 1. Write the docs
+
+Docs live in the schemas, next to what they describe:
+
+- **`///` docstrings** document the declaration below them: a struct, a field,
+  an enum, a protocol, a function, an error, a validator, a constant. A
+  `/// @name: ...` line documents one field or argument
+  ([Docstrings](idl/docstrings.md)).
+- **`//!` headers** document a module, at the top of its `.ids` file, and the
+  package, at the top of `config.idp`
+  ([Module docs](idl/docstrings.md#module-docs)).
+
+```ids linenums="1"
+//! The shop's API.
+
+/// One order.
+struct Order {
+    /// What it costs, in cents.
+    total: u64
+}
+
+/// Orders things.
+protocol Orders {
+    /// Place one.
+    /// @order: what to place
+    function place(order: Order) -> bool;
+}
+```
+
+The first line of a docstring or header is its summary: index tables show it
+next to each name.
+
+## 2. Render them
+
+```bash
+comline doc
+```
+
+This compiles the working tree and writes Markdown into `docs/api/`:
+
+| File | What's on it |
+|---|---|
+| `index.md` | The package's name and version, its `config.idp` header, and a table of its modules. |
+| `<module>.md` | One per schema (`types::user` is `types/user.md`): its header, a table of what it declares, then each declaration with its docs, annotations, fields, arguments, variants, and each protocol's wire summary, the same table the editor's hover and `comline inspect` show. |
+| `changelog.md` | With `--changelog-from`: what changed, as `comline diff` reports it. |
+| `nav.yml` | The pages as a `nav` list for mkdocs or zensical. |
+
+Every declaration has an anchor, its name in lowercase (`api.md#order`), and a
+type links to the declaration it names, on the same page or another module's.
+
+### Put them on a site
+
+The pages are plain Markdown, so they drop into an mkdocs or zensical site:
+render into the site's `docs/` tree and copy `nav.yml`'s entries into
+`mkdocs.yml`.
+
+```bash
+comline doc --out site/docs/api
+```
+
+### Link to dependencies
+
+A type from a dependency or `std` is plain text, unless you say where that
+package's docs live. Give each one a base URL in `comline.toml`:
+
+```toml
+[doc]
+links = { std = "https://example.com/std", billing = "https://docs.example.com/billing" }
+```
+
+A type from `billing`'s `api` module, `Invoice`, then links to
+`https://docs.example.com/billing/api/#invoice`: the base, the module, and the
+anchor, in the directory-URL form mkdocs and zensical publish pages at.
+
+## 3. Document a release
+
+`--version` documents a built version instead of the working tree: a version
+string, a commit hash, or `HEAD`.
+
+```bash
+comline doc --version 0.2.0 --out site/docs/api/0.2.0
+```
+
+A build stores the `//!` headers with the version, so a version documents with
+the headers it was built with, even after the sources change. Two things need
+the source and are missing from a built version: protocols have no wire
+summary, and dependency types have no link. A version built before headers
+were stored has none.
+
+`--changelog-from` adds what changed since an earlier built version:
+
+```bash
+comline doc --version 0.2.0 --changelog-from 0.1.0
+```
+
+## 4. Or take the model
+
+`--format json` writes the documentation model itself, `model.json`: the same
+data the pages are rendered from, as JSON. Its shape is a published schema,
+[`doc-model/1.json`](../schemas/doc-model/1.json), so a tool of your own can
+render it any way it likes.
+
+```bash
+comline doc --format json
+```
+
+A renderer of your own can also run inside `comline doc`: a program speaking
+the [generator protocol](../reference/generator-protocol.md#render) that answers
+`Render` requests, named by `[doc] renderer` in `comline.toml`. It runs exactly
+like an [external generator](extending/generators.md): a `.wasm` is sandboxed,
+and every path it returns is checked before anything is written.
+
+## Settings
+
+Every flag has a `[doc]` key in [`comline.toml`](../reference/congregation/comline-toml.md#doc),
+and the flag wins:
+
+| Key | Flag | Default |
+|---|---|---|
+| `out` | `--out` | `docs/api` |
+| `format` | `--format` | `markdown` |
+| `changelog_from` | `--changelog-from` | none |
+| `renderer`, `timeout` | none | the built-in renderer |
+| `links` | none | none |
+
+## What's next
+
+The standard library will be rendered onto this site the same way, so that
+`links = { std = ... }` has somewhere to point.

docs/docs/guide/idl/docstrings.md

@@ -67,6 +67,8 @@ The editor shows them wherever a path segment is. In
 the modules in it, and hovering `validators` shows that module's docs and the
 types it declares. `use` completion shows the same docs beside each module and
 type. Like docstrings, module docs are for readers and tools: they don't change
-a build, a version or generated code.
+generated code or the version bump. A build stores them with the version, so
+[`comline doc`](../documenting.md) documents a built version with the headers
+it was built with; editing only docs makes a new build with no bump.
 
 [The standard library](use.md#the-standard-library) documents itself this way.

docs/docs/reference/cli.md

@@ -40,6 +40,7 @@ parse → resolve imports → validate → (freeze into .comline/) → (generate
 | `comline generate [--target <lang>] [--out <dir>] [--layout <tpl>] [--mode <m>] [--external-std] [--no-external-std] [--validate-on-decode <when>] [--dry-run] [--list] [--watch]` | Validate (no freeze), then write generated code. Location/layout from [`comline.toml`](congregation/comline-toml.md); targets from there or `config.idp`. A target can use an [external generator](../guide/extending/generators.md) (`generator = "<path>"`) and map [encodings](../guide/extending/encodings.md#5-map-it-to-a-type). |
 | `comline release [1.0] [--raise <level> --reason <text>] [--yes]` | Build, then release: create the version from the highest bump since the last release ([builds and releases](versioning.md#builds-and-releases)). `--raise patch\|minor\|major` goes higher, with a `--reason` recorded with the release; it never lowers. `1.0` makes a `0.x` package stable after showing what that commits to (its protocols, and what's worth settling first). Raising to major and `1.0` ask on a terminal; without one they refuse (exit `2`) unless `--yes` (`-y`). |
 | `comline diff <old> [<new>]` | Show schema changes between two built versions. Each argument is a version (`0.2.0`), a commit hash (4+ chars), or `HEAD` (`<new>` default). |
+| `comline doc [--out <dir>] [--version <v>] [--changelog-from <v>] [--format <f>]` | Write the package's documentation: Markdown pages (an index, one page per module with its `//!` header, every declaration's `///` docs and each protocol's wire summary, a changelog, an mkdocs/zensical `nav.yml`), or `--format json` for the documentation model. The working tree by default; `--version` documents a built version from its frozen IR and headers. Defaults from [`comline.toml`](congregation/comline-toml.md#doc)'s `[doc]`. No `.comline/` write. See [comline doc](tools/doc.md). |
 | `comline conformance --impl <command> [--encoding <name>]... [--corpus <dir>] [--timeout <secs>]` | Run an implementation of an encoding against its vectors. The package's declared [`encodings`](../guide/extending/encodings.md) ship vectors beside their profiles (`profiles/x.vectors.json`); `--impl` is a process speaking the [ops protocol](conformance.md#the-ops-protocol), run once with a cleared environment, a time limit (`--timeout`, 60 s) and a capped stdout. `--encoding` limits the run to one encoding (repeatable); `--corpus` runs a corpus laid out like the runtime's `runtime/conformance` instead, which is how the official encodings are proven. A failed or unimplemented case, or an unproven profile fact, exits `1`. See [Formats and framings](../guide/extending/encodings.md#3-write-vectors-and-prove-an-implementation). |
 | `comline clean [--dry-run]` | Remove generated code (the `generate` output root or files). Never touches `.comline/`. |
 | `comline reset [--force] [--dry-run]` | **Destructive.** Delete `.comline/` — the whole version history, no undo — *and* generated code. Next `build` starts over at `0.0.1`. On a terminal it shows how many versions will be lost and makes you type `reset`; without a terminal it refuses (exit `2`) unless `--force`. |

docs/docs/reference/congregation/comline-toml.md

@@ -1,7 +1,8 @@
 # `comline.toml`
 
 Consumer-owned configuration for `comline generate`: where generated code goes,
-in what layout, in what form, and for which package versions. It is committed,
+in what layout, in what form, and for which package versions. Its `[doc]`
+table configures `comline doc`. It is committed,
 never frozen, and never part of a published package. With no `comline.toml` (or
 an all-commented one) the defaults below apply and targets come from
 `config.idp`'s `code_generation.languages`.
@@ -198,6 +199,27 @@ your `comline` build: declaring a language means you mean it, so a missing
 generator is an error rather than a silent skip, and `only` is how you say
 you want fewer.
 
+## `[doc]`
+
+Settings for [`comline doc`](../tools/doc.md). Each key but `renderer`,
+`timeout` and `links` has a flag that overrides it.
+
+| Key | Type | Default | Meaning |
+|---|---|---|---|
+| `out` | string (path) | `"docs/api"` | Output directory, relative to `comline.toml`. `--out` overrides it. |
+| `format` | string | `"markdown"` | `markdown` (pages), `json` (the documentation model, `model.json`), or another renderer id `renderer` offers. `--format` overrides it. |
+| `renderer` | string (path) | — | A renderer outside `comline`: a `.wasm` (run sandboxed under wasmtime) or an executable, relative to `comline.toml`'s directory. Run exactly like an external [`generator`](#generator). Unset, the built-in renderer for `format` is used. |
+| `timeout` | integer (seconds, ≥ 1) | `60` | The wall-time limit for each call to `renderer`. |
+| `links` | table | — | Where each dependency's docs live, by the name the package gives it (or `std`): a base URL. A type from a dependency listed here links to `<base>/<module>/#<anchor>`; one that isn't is plain text. |
+| `changelog_from` | string | — | Add a changelog from this built version (a version or commit hash) to the documented one. `--changelog-from` overrides it. |
+
+```toml
+[doc]
+out = "site/docs/api"
+links = { std = "https://example.com/std" }
+changelog_from = "0.1.0"
+```
+
 ## Layout variables
 
 | Variable | Value |

docs/docs/reference/generator-protocol.md

@@ -1,6 +1,7 @@
 # Generator protocol
 
-How the CLI talks to a code generator. For a walkthrough, see
+How the CLI talks to a code generator, and to a documentation renderer. For a
+walkthrough, see
 [Generators](../guide/extending/generators.md); to point a target at one, see
 the [CLI reference](cli.md#commands) and
 [`comline.toml`](congregation/comline-toml.md#generator).
@@ -12,6 +13,9 @@ The Rust definitions are in `codegen/src/protocol.rs` of the
 
 **Protocol version: `0`** (`PROTOCOL_VERSION`). While Comline is in its early
 days, the protocol may change incompatibly, and the number moves when it does.
+An additive change keeps the number: a new field with a default, or a new
+request that only goes to a program whose `Description` says it handles it
+(`Render` was added this way).
 
 The protocol is also pinned to an **IR format** (`IR_FORMAT`), the stamp on
 `.comline/` that `comline-core` owns. A generator accepts the IR formats it
@@ -24,7 +28,7 @@ lists and refuses others.
 - Each is a single **CBOR** document, in the encoding `comline-core` stores
   its IR with (name-tagged fields).
 - A generator process serves one request per run. The CLI starts it once for
-  `Describe` and once for each `Generate`.
+  `Describe` and once for each `Generate` or `Render`.
 - Exit successfully whenever a response was written, an error response
   included. A nonzero exit or a response that isn't a protocol response is a
   failure.
@@ -35,7 +39,7 @@ lists and refuses others.
 |---|---|---|
 | `protocol` | integer | the protocol version the sender speaks |
 | `ir_format` | integer | the IR format of every frozen unit in the body |
-| `body` | `Describe` or `Generate(GenerateRequest)` | what is asked |
+| `body` | `Describe`, `Generate(GenerateRequest)` or `Render(RenderRequest)` | what is asked |
 
 `protocol` and `ir_format` come first, and a generator reads them before the
 body. A mismatch is reported even when the body has a shape the generator
@@ -72,12 +76,29 @@ Generate one target. A `GenerateRequest` holds:
   ([Map it to a type](../guide/extending/encodings.md#5-map-it-to-a-type)).
 - `mode` is `Code` or `Lib`. `dylib` isn't part of the protocol today.
 
+### `Render`
+
+Render a package's documentation, for [`comline doc`](tools/doc.md). Only sent
+to a program whose `Description` lists the renderer at the model version. A
+`RenderRequest` holds:
+
+| Field | Type | Meaning |
+|---|---|---|
+| `renderer` | string | the renderer's id, from `Description.renderers`: `markdown` |
+| `model_version` | integer | the documentation model's version (`DOC_MODEL_VERSION`, `1`), readable without parsing `model` |
+| `model` | string | the documentation model as JSON text, in the shape of [`doc-model/1.json`](../schemas/doc-model/1.json) |
+| `options` | `{ links }` | `links`: a dependency's name (or `std`) to the base URL of its docs |
+
+A renderer accepts every option and ignores the ones it has no use for. The
+answer is `Render(Generated)`: files, checked like a generator's.
+
 ## Response
 
 | Variant | Carries |
 |---|---|
 | `Describe(Description)` | the answer to `Describe` |
 | `Generate(Generated)` | the answer to `Generate` |
+| `Render(Generated)` | the answer to `Render`: the rendered files |
 | `Error { message }` | the request was refused or generation failed |
 
 ### `Description`
@@ -87,6 +108,7 @@ Generate one target. A `GenerateRequest` holds:
 | `languages` | one entry per generator; a language with several floors has several |
 | `protocols` | `(min, max)` protocol versions it accepts, inclusive |
 | `ir_formats` | the IR formats it accepts |
+| `renderers` | the documentation renderers it offers, each `{ id, model_versions }`; empty, or absent from an older program, when it offers none |
 
 A language entry holds:
 
@@ -114,6 +136,9 @@ target's version.
   both sides.
 - **Language:** the generator must offer the target's language at the target's
   version, or the CLI refuses it.
+- **Renderer:** for `comline doc`, the program must list the format in
+  `renderers`, with the CLI's model version in `model_versions`, or the CLI
+  refuses it before sending `Render`.
 - **Output:** every path, then the total size, is checked before anything is
   written.
 
@@ -146,9 +171,13 @@ environment and isn't otherwise sandboxed.
 
 The `comline_codegen::protocol` module exports:
 
-- `Request`, `RequestBody`, `GenerateRequest`, `Response`, `Description`,
-  `Generated` and the types in the tables above;
+- `Request`, `RequestBody`, `GenerateRequest`, `RenderRequest`,
+  `RenderOptions`, `Response`, `Description`, `RendererInfo`, `Generated` and
+  the types in the tables above;
 - `encode` and `decode`, to and from CBOR;
 - `respond(registry, bytes)`, which answers one request without panicking, and
   `handle`, which does the same on bytes;
-- `serve_stdio(registry)`: a generator process's whole `main`.
+- `serve_stdio(registry)`: a generator process's whole `main`. A registry
+  holds renderers too (`Registry::register_renderer`), so one binary can
+  generate, render, or both. The official Markdown renderer is
+  `comline-doc-markdown` in the generation repository.