Skip to content

Preserve source clocks, AV1 descriptions and M2TS arrival timestamps - #10

Open
SteveYu000 wants to merge 1 commit into
DeepRegular:mainfrom
SteveYu000:codex/rendering-correctness
Open

SteveYu000 wants to merge 1 commit into
DeepRegular:mainfrom
SteveYu000:codex/rendering-correctness

Conversation

@SteveYu000

Copy link
Copy Markdown
Contributor

Smart-rendered output can decode successfully while disagreeing with its source
display timestamps, container aspect ratio or AV1 track description. On v0.8.39,
a 40/40/80 ms cadence can be classified from its average, millisecond-rounded
Matroska seams can move later pictures, AV1 container-only SAR can be lost, and
mixed AV1 headers can disagree with the one description written by FFmpeg 7.1.
Variable-rate M2TS output can also have backward arrival timestamps.

This change:

  • propagates effective container SAR through probing, boundary encoding and
    output stream creation using av_guess_sample_aspect_ratio;
  • refines progressive non-TS timing onto the source container clock, detects
    repeated holds using measured median gaps, and separates coarse-clock
    selection slack from measured display coverage; invalidates old seek caches;
  • extracts actual AV1 headers with FFmpeg's BSF, validates the public av1C
    fields for MP4 and a conservative single Sequence Header for Matroska/WebM,
    and initializes the description from the first configuration actually written;
  • explicitly retries with all selected video re-encoded if an incompatible AV1
    seam cannot fit the container. FFmpeg 7.1's MOV muxer has one stsd entry:
    attaching new extradata alone does not add another description. The fallback
    log reports zero copied video; another incompatible configuration is an error;
  • reconstructs M2TS ATS after muxing and SI graft from the final PCR anchors,
    preserving every 188-byte TS packet and the two CPI bits. It handles clock
    wrap and one-PCR cuts without adding a fixed high mux rate or null padding.

Adds seven Rust unit tests, a standalone standard-library Python regression
suite (tests/container_contract_tests.py), its normal CI suite hook, and
docs/technical/container-contracts.md. No new Cargo dependencies.

Validation

Baseline: 683a4ddb2936c72eea03945734c62bac5ff39b78 (v0.8.39). Baseline and
patched CLI built with the same Rust 1.98.0 / Windows x64 MSVC / FFmpeg 7.1.1
SDK and DLLs. Independent media checks used FFmpeg 9.0.2.

Fixed-input exports Baseline Patched Explicit full-video AV1 fallbacks
MP4 / AV1 38/55 55/55 5
Matroska / AVC, HEVC, AV1 71/110 110/110 15
TS and M2TS / AVC, HEVC 31/70 62/70 0
Total 140/235 227/235 20

87 newly passing exports, no formerly passing case regressed. Every paired
input has the same SHA-256. Checks include strict decode, frame identity/count,
display PTS, copy bytes/pixels, duration, metadata and container descriptions.
For MKV/TS, copy assertions cover the interiors of the CLI's declared copy
intervals, with a 1 ms margin because the printed bounds are rounded.
The 20 explicit fallbacks are valid whole-video re-encodes, not successful
partial-copy rendering. Tiny AV1 show-existing commands can match source packet
hashes even after re-encoding, so those hashes alone are not copy evidence.

All 35 M2TS arrival-clock failures disappear. Removing the four-byte prefixes
leaves all 35 outputs byte-identical to baseline; all 35 .ts files are also
byte-identical. Ordinary FFmpeg input seeking was not a criterion in this run.

Additional local validation:

  • New generated container-contract suite: 28 passed (VFR AVC/HEVC/AV1 in
    MP4/MKV, rounded NTSC, SAR, AV1 descriptions, all three M2TS table modes).
    Unmodified baseline fails its first AVC/VFR display-PTS assertion.
  • Existing run_vfr_tests.sh: 8 passed.
  • Existing run_vp9_av1_tests.sh: 5 passed.
  • cargo clippy --release --locked --workspace --all-targets: exit 0, existing
    upstream warnings remain.
  • Final workspace unit tests: 577 passed, 4 explicitly skipped. The same
    four tests fail on the unmodified Windows baseline (core: 535 passed / 4
    failed). Their names are listed below; they were not changed by this PR.
input::tests::a_file_url_is_the_file_it_names
udfw::tests::an_image_being_written_by_another_run_is_left_to_it
udfw::tests::an_image_inside_its_own_folder_is_refused
vobsub::tests::names_follow_the_whole_cut_name

Reproduce the newly added media tests with:

cd rust && cargo build --release --locked && cd ..
bash tests/run_container_contract_tests.sh
# or: bash tests/ci.sh container_contract

Limits and review points

  • Eight TS/M2TS VFR end-duration cases still fail: a final picture expected to
    stay for 80 ms is inferred as 40 ms without a following PTS. This predates
    the patch and is separate from the arrival-clock correction.
  • Source-clock refinement retains the old grid if the signed 32-bit clock
    bound would be exceeded. TS and interlaced timing rules are unchanged.
  • AV1 fallback can cost an additional attempt and re-encode all selected
    video. The GUI/analysis copy estimate can remain from before fallback; the
    rendering log states the actual fallback. The Matroska guard deliberately
    does not optimize the permitted operating-parameters-only exception.
  • M2TS interpolation adds output I/O and does not certify T-STD buffer/rate or
    Blu-ray compliance; missing/non-monotonic PCR is rejected.
  • Linux full CI, GUI compilation and hardware-player certification were not
    run locally. The added suite is integrated for upstream CI to run.

Audio/subtitle strategies and H.26x parameter-set/POC/Open-GOP handling are left
to the existing implementation. The new AV1 FFI declarations are the public
FFmpeg 7.1 BSF API, needed because ffmpeg-sys-next 7.1 does not bind bsf.h.

Carry effective stream SAR into boundary encoders and output streams.
Represent progressive non-TS source clocks without quantizing them to an
average-rate grid, and keep selection slack out of display coverage.
Invalidate seek caches that can misclassify repeated VFR holds.

Use FFmpeg extract_extradata to validate actual AV1 container fields and
Matroska sequence consistency. Retry incompatible selections as explicitly
reported full-video encodes instead of writing an invalid description.

Rebuild final M2TS arrival prefixes from PCR anchors after SI graft while
preserving all 188-byte transport packets. Add generated container-contract
regressions, seven unit tests, the normal CI hook and technical notes.

Validation: 235 fixed-input exports improve from 140 to 227 passes with no
regression (20 explicit AV1 full-video fallbacks). New media tests 28/28;
existing VFR 8/8 and VP9/AV1 5/5; Clippy exit 0. Workspace tests 577 passed,
with four pre-existing Windows failures explicitly skipped. Eight TS/M2TS
VFR final-frame-duration cases remain outside this fix.
@DeepRegular

Copy link
Copy Markdown
Owner

Thank you for the detailed PR and the careful write-up. I built it on Linux (Debian 13, FFmpeg 7.1) and compared it with v0.8.39 on the new suite, the existing suites and some real recordings. Here is what I found.

Confirmed

  • Your 28 container-contract tests pass with this branch, and v0.8.39 fails at the first VFR display-PTS check, as you describe.
  • The M2TS arrival-time fix is real. A standalone .m2ts from v0.8.39 has arrival stamps that jump backwards on almost every packet. With this branch they increase monotonically, and the 188-byte packets are byte-identical. I checked this with two broadcast recordings.
  • .ts output and --bdav disc output are byte-identical to v0.8.39.
  • The existing suites, the workspace unit tests (583) and clippy all pass.

Regression: the last picture of a range is shown for 4 ms

If a progressive MP4 with a 1 ms clock is classified as varying, and a kept range ends a few ms after a picture, that last picture is now shown only until the range bound instead of for its full length. With two ranges, the next range starts 4 ms after the last picture of the previous one (v0.8.39: 41.7 ms). I found it on a web-distributed 23.976 H.264 MP4, and this synthetic file reproduces it:

ffmpeg -f lavfi -i testsrc2=rate=24000/1001:size=1280x720 -t 90 \
  -vf "setpts=(N*1001/24000-if(eq(mod(N\,300)\,299)\,0.025\,0))/TB" \
  -fps_mode passthrough -c:v libx264 -g 120 -video_track_timescale 1000 src.mp4
smartcut src.mp4 --keep 20-38 --keep 60-75 -o out.mp4

The last source picture in the first range is at 37.996 s. In the output, the gap between it and the first picture of the second range is 0.0040 s with this branch, and 0.0417 s with v0.8.39. MKV output behaves the same way. As far as I can tell, the next_picture coverage logic in held_to_the_end is skipped for variable_rate sources (and copy_segment always passes None), while on_clock makes the grid fine enough that the hold is no longer rounded up to a whole frame.

Design points I'd like to decide separately

  • AV1 into Matroska: if the sequence headers differ at a seam, the whole selection is re-encoded. On a 720p clip this took 100 s instead of 35 s, and the log reports both "copied 75.8%" and then "copied 0%". I agree the single-sequence rule is in the spec, but I'm not sure yet whether a full re-encode is the right answer, or whether it should be a warning or an option.
  • M2TS for disc output: bdav::stamp rewrites every arrival time afterwards, so m2ts::restamp only adds a pass there. Please skip it when the output is a clip in a disc folder (onto_a_disc). Also, failing outright when a standalone .m2ts has no PCR or a backward PCR is new behaviour; a note and leaving the stamps as they were would be safer.
  • Byte-for-byte output: for progressive non-TS sources, the timescale/grid changes alter the output even where the frames and timestamps are the same. I try to keep outputs identical between releases unless there is a reason, so I'd like this limited to the sources that need it.
  • Docs: each page under docs/technical/ has a Japanese counterpart (*.ja.md). I can write those myself, so no need to translate.

Request

Could you split this into four PRs?

  1. SAR propagation: small and clearly good; I'm happy to take it first.
  2. AV1 configuration guard.
  3. Source clock / VFR classification, including a fix for the 4 ms case above and a test for it.
  4. M2TS arrival times, skipped for disc output.

Smaller PRs are much easier for me to verify against real recordings, and the parts that are ready can go in without waiting on the rest. Thanks again; the M2TS finding in particular was a real bug I had missed.

DeepRegular pushed a commit that referenced this pull request Nov 2, 2026
* Preserve container sample aspect ratio through smart rendering

AV1 MP4/Matroska display aspect can live on AVStream while codecpar has no
SAR. Read the effective ratio using av_guess_sample_aspect_ratio, keep the
existing codecpar fallback, and carry it to boundary encoders and the
output stream.

Add a standalone SAR suite covering copy, mixed and encoded ranges for
AVC/HEVC/AV1 in MP4/MKV, plus public outline checks using conformdiag.
Register only this suite in CI and document its entry point.

Validation: same-input exports improve from 30/36 to 36/36, outline checks
from 10/12 to 12/12. Workspace tests 570 passed with four known Windows
baseline failures filtered; clippy exits 0. Source clocks, VFR handling,
AV1 configuration policy and M2TS behaviour are unchanged.

SAR-only split from PR #10.

* Keep container SAR out of boundary encoder parameters

Preserve the input codecpar SAR for boundary encoders. Use the guessed
display SAR only for the outline and the output stream's metadata. Set
both output SAR fields to satisfy libavformat's consistency check without
changing the source parameters or the copied SPSs.

Reproduce the reviewer's container override: 12 SPSs now all retain 1:1
while the output container keeps 4:3 DAR. Add AVC/HEVC regressions with
square and non-square coded SAR, MP4/MKV outputs and TS header inspection.
Correct the building.md column and state 44 exports plus 16 outlines.

Actually build with Rust 1.98/MSVC/FFmpeg 7.1.1: 60/60 media checks,
570 workspace tests passed with four known Windows baseline failures
filtered; Clippy exits zero. Timing remains unchanged in all 44 outputs,
and the eight new override exports retain all baseline video packets
and decoded pixels. Document the pre-existing HEVC TS remux decode
failure separately from the SAR assertions.

Addresses the review on PR #11; no AV1 configuration, VFR or M2TS changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants