Skip to content

conformance: run an encoding's implementation against its package's vectors - #108

Merged
Kinflou merged 2 commits into
mainfrom
conformance/impl-runner
Oct 31, 2026
Merged

Kinflou merged 2 commits into
mainfrom
conformance/impl-runner

Conversation

@Kinflou

@Kinflou Kinflou commented Oct 31, 2026

Copy link
Copy Markdown
Member

Closes #106 (step B5). Depends on ComlineProject/runtime#69: comline-runtime-conformance is pinned at that PR's branch commit 3a9b600, to be re-pinned to runtime's merge commit once it merges (the fixture workspace tests/fixtures/conformance/Cargo.toml pins the same rev and moves with it). Other pins (core, generation, comline-rust, comline-typescript) are unchanged.

comline conformance --impl "<command line>" [--encoding <name>]... [--corpus <dir>] [--timeout <secs>]

  • Vectors convention: an encoding declared under encodings ships its vectors beside its profile: the profile path with .json replaced by .vectors.json (profiles/lines.json -> profiles/lines.vectors.json), a corpus vectors file ({feature, cases}) in the spec's case format. No core change: the declared names come from the frozen config, the profile path from the parsed config.idp (ProjectContext::config), which core has already checked.
  • Official encodings: --corpus <dir> runs a runtime-style corpus (profiles/, vectors/) instead; each encoding runs every case of the features its profile's facts name. Naming an official encoding without --corpus says to pass it.
  • Running: the implementation speaks the ops protocol (as comline-reference does). Every question goes in up front (a case's questions never depend on answers), with version first; a protocol other than 1 is refused. It runs once through external::native::run (now taking args): cleared env, time limit (60s default, --timeout), stdout cap, stderr shown prefixed.
  • Report (stderr): per encoding, cases passing, each fact proven or why not, each failed case with a byte diff; a note if version doesn't list the encoding. Exit 1 on any failed, not-implemented case or unproven fact.

Tests (tests/cli/conformance.rs, fixture implementations built from tests/fixtures/conformance):

  • comline-reference at the pinned runtime rev passes every official encoding in the corpus that rev ships, every fact proven;
  • flipper (the reference, flipping the first byte of each request frame it encodes) fails datagram@1 with the diff and the unproven calls fact;
  • tests/fixtures/conformance_package declares com.acme.lines@1 with its profile and vectors; the lines stub passes 5/5 with both facts proven; the reference fails it as not implemented; missing vectors name the expected path;
  • an official name without --corpus points at it.

Locally: cargo build, cargo test (105 + 40 pass), cargo fmt --check (and both fixture workspaces), both clippy runs with -D warnings. CI gains an fmt check for the new fixture workspace. docs/cli.md and the README table document the command.

…ectors

`comline conformance --impl "<command line>"` runs a process speaking the
conformance ops protocol (as comline-reference does) against the vectors
each declared encoding ships beside its profile
(`profiles/x.json` -> `profiles/x.vectors.json`), or, with `--corpus`, a
runtime-style corpus for the official encodings. It reports which profile
facts are proven and which cases failed, with byte diffs, and exits 1 on
any failure. The process runs once, through the external generators'
runner: cleared environment, time limit, capped output.

The runner itself is ComlineProject/runtime's comline-runtime-conformance,
pinned by git rev.

Closes #106
@Kinflou
Kinflou merged commit e22e3ec into main Oct 31, 2026
2 checks passed
@Kinflou
Kinflou deleted the conformance/impl-runner branch October 31, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

comline conformance: run an implementation against an encoding's vectors

1 participant