Skip to content

docs: check the CLI and IDL references against their sources - #28

Merged
Kinflou merged 1 commit into
mainfrom
docs/source-drift-checks
Oct 12, 2026
Merged

Kinflou merged 1 commit into
mainfrom
docs/source-drift-checks

Conversation

@Kinflou

@Kinflou Kinflou commented Oct 12, 2026

Copy link
Copy Markdown
Member

The CLI reference and the IDL guide restate what source owns, in prose deliberately richer than clap's help strings or a token list — so neither page is generated. Two scripts instead check that the surfaces still name the same things, which is the part that rots silently.

scripts/check-cli-drift.py

Parses comline --help and every subcommand's, comparing commands and long flags against the page in both directions. It found three real gaps, fixed here:

  • comline targets was entirely undocumented — a whole command missing from the table.
  • generate --external-std was missing.
  • --verbose was never named, only -v/-vv/-vvv, so searching the long form found nothing.

Documenting targets also corrected a nearby claim that completions is the only command writing to stdout.

It scans _data/*.yaml alongside the markdown — it has to, since moving the Global flags table into YAML made the page unparseable to a text-based checker (that showed up as --quiet/--plain false positives on the first run).

scripts/check-idl-drift.py

Reads core's rust_sitter grammar leaves (36 keywords) and checks the guide documents each. No Rust build — the grammar is a source file, read from a checkout or fetched.

A keyword counts as documented only inside a code fence or inline code span. Both halves matter: searching prose for bare words matches ordinary English ("the type of a field", "a package depends on"), while backticked-only search misses keywords appearing just in examples. A first pass using the latter reported nine undocumented keywords — all nine false positives.

import is excluded with its reason recorded (the grammar marks it legacy, superseded by use). The config.idp grammar is deliberately not checked yet: reference/congregation/config-id.md is still an empty stub, so it would report false as undocumented every run, and an always-red check is one people learn to ignore.

CI

  • IDL drift — per relevant PR plus weekly. Stdlib only, one fetch. Kept out of the Docs workflow on purpose: that one gates the Pages deploy, and a fetch against another repo should not be able to block a deploy.
  • CLI drift — weekly plus PRs touching the CLI docs, since it needs a cargo build. Caches the binary keyed on the CLI's current commit, so an unchanged CLI costs nothing.

Both resolve the target repo's default branch via the API rather than assuming one — core is on master, cli on main, and assuming main for core 404'd.

Also replaces the plain-text comline-typescript#6, #7 in the runtime repo-structure record with live PR-state badges, and fills in the auto-documentation proposal with a full definition of what the feature is (an optional tool documenting a user's schemas and packages, with the CAS supplying per-version docs and a generated changelog) plus its open questions.

Test plan

  • poetry run zensical build --strict — clean
  • Both checkers pass against comline 0.2.0 and core's grammar
  • Verified the pipefail trap in both workflows: a pipeline returns tee's status, so if ! would never fire without shell: bash — demonstrated failing and passing
  • Verified reverse-direction flag detection with an injected --obsolete-flag
  • Verified new-keyword detection with a doctored grammar copy
  • Verified the fetch path (no checkout) against core/master

The CLI reference and the IDL guide restate what source owns, in prose
deliberately richer than clap's help strings or a token list — so neither
is generated. Two scripts instead check the surfaces still name the same
things, which is the part that rots silently.

scripts/check-cli-drift.py parses `comline --help` and every subcommand's,
and compares commands and long flags against the page in both directions.
It found three real gaps, fixed here:

- `comline targets` was entirely undocumented.
- `generate --external-std` was missing.
- `--verbose` was never named, only `-v`/`-vv`/`-vvv`.

Documenting `targets` also corrected the claim that `completions` is the
only command writing to stdout.

scripts/check-idl-drift.py reads core's `rust_sitter` grammar leaves (36
keywords) and checks the guide documents each. A keyword counts only
inside a code fence or inline code span: searching prose for bare words
matches English ("the type of a field"), while backticked-only search
misses keywords that appear just in examples — that first pass reported
nine keywords, all false positives. `import` is excluded with its reason
(legacy, superseded by `use`). The `config.idp` grammar is left out until
`reference/congregation/config-id.md` is written, since it would report
`false` as undocumented every run and an always-red check gets ignored.

CI: IDL drift runs per relevant PR (a source fetch, no Rust), kept out of
the Docs workflow so it cannot block a Pages deploy; CLI drift runs weekly
since it needs a cargo build, caching the binary on the CLI's commit.
Both resolve the default branch rather than assuming it — `core` is on
`master`, `cli` on `main`.

Also replaces the plain-text `comline-typescript#6, #7` in the runtime
repo-structure record with live PR state badges.
@Kinflou
Kinflou merged commit 882e81c into main Oct 12, 2026
4 checks passed
@Kinflou
Kinflou deleted the docs/source-drift-checks branch October 12, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant