Repository navigation
conformance: run an encoding's implementation against its package's vectors - #108
Merged
Merged
Conversation
…ectors `comline conformance --impl "<command line>"` runs a process speaking the conformance ops protocol (as comline-reference does) against the vectors each declared encoding ships beside its profile (`profiles/x.json` -> `profiles/x.vectors.json`), or, with `--corpus`, a runtime-style corpus for the official encodings. It reports which profile facts are proven and which cases failed, with byte diffs, and exits 1 on any failure. The process runs once, through the external generators' runner: cleared environment, time limit, capped output. The runner itself is ComlineProject/runtime's comline-runtime-conformance, pinned by git rev. Closes #106
Closed
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #106 (step B5). Depends on ComlineProject/runtime#69:
comline-runtime-conformanceis pinned at that PR's branch commit3a9b600, to be re-pinned to runtime's merge commit once it merges (the fixture workspacetests/fixtures/conformance/Cargo.tomlpins the same rev and moves with it). Other pins (core, generation, comline-rust, comline-typescript) are unchanged.comline conformance --impl "<command line>" [--encoding <name>]... [--corpus <dir>] [--timeout <secs>]encodingsships its vectors beside its profile: the profile path with.jsonreplaced by.vectors.json(profiles/lines.json->profiles/lines.vectors.json), a corpus vectors file ({feature, cases}) in the spec's case format. No core change: the declared names come from the frozen config, the profile path from the parsedconfig.idp(ProjectContext::config), which core has already checked.--corpus <dir>runs a runtime-style corpus (profiles/,vectors/) instead; each encoding runs every case of the features its profile's facts name. Naming an official encoding without--corpussays to pass it.versionfirst; a protocol other than 1 is refused. It runs once throughexternal::native::run(now taking args): cleared env, time limit (60s default,--timeout), stdout cap, stderr shown prefixed.versiondoesn't list the encoding. Exit 1 on any failed, not-implemented case or unproven fact.Tests (
tests/cli/conformance.rs, fixture implementations built fromtests/fixtures/conformance):flipper(the reference, flipping the first byte of each request frame it encodes) failsdatagram@1with the diff and the unprovencallsfact;tests/fixtures/conformance_packagedeclarescom.acme.lines@1with its profile and vectors; thelinesstub passes 5/5 with both facts proven; the reference fails it as not implemented; missing vectors name the expected path;--corpuspoints at it.Locally:
cargo build,cargo test(105 + 40 pass),cargo fmt --check(and both fixture workspaces), both clippy runs with-D warnings. CI gains an fmt check for the new fixture workspace.docs/cli.mdand the README table document the command.