Repository navigation
perf(video-decode): pipeline the Media Foundation GPU readback - #2583
Open
cesarfrancots wants to merge 16 commits into
Open
cesarfrancots wants to merge 16 commits into
cesarfrancots wants to merge 16 commits into
Conversation
Examples live in target/<profile>/examples, next to deps/, but the DXC lookup only stepped up from deps/. Examples fell back to FXC, which took ~13s to compile the shaders on every editor open and skewed every benchmark built as an example.
…marks scripts/capbench.py drives the existing measurement binaries (cap export, cap record + cap-test validate, and the cap-editor playback/decode/startup examples) against the CI performance fixture and records wall time, whole-process-tree CPU time and peak commit (Windows job object), plus per-benchmark metrics parsed from their output. Variants (builds and/or env overrides) are interleaved per repetition with alternating order so drift and thermal state hit both sides equally; `compare` reports the median change with a bootstrap 95% interval and SSIM of the exported files. Single-shot runs on the reference machine varied by ~25%, which is larger than most individual changes. The playback example gains --timer-1ms to measure under the 1ms timer resolution the editor holds while playing.
Backport of gfx-rs/wgpu#8273 (wgpu 27). Device::wait signalled the device's single auto-reset idler event, so when several threads waited on one device a wakeup could reach the wrong waiter and the other one slept until the 60s CLEANUP_WAIT_MS timeout while wgpu-core held the fence lock, blocking every queue.submit on that device. The desktop app shares one device between editor, preparing-preview and screenshot renderers, and readbacks are about to block on the fence, so concurrent waits become routine.
…-polling Preview and export readbacks polled map_async with yield, then sleep(100us), then sleep(1ms). Outside timeBeginPeriod every sleep rounds up to the ~15.6ms Windows timer tick, which pinned render+readback at ~24ms regardless of resolution. Block on the submission's fence in spawn_blocking (device.poll WaitForSubmissionIndex) so the wait ends when the GPU signals. capbench, 5 interleaved reps (Ryzen 9 9950X3D, RTX 4070 Ti, fixture 3024x1964 + camera): playback pipeline 31.3 -> 92.2 fps, render p50 24.2 -> 4.1 ms, p95 46.0 -> 17.0 ms scrub p50/p95 79.6/123.5 -> 63.5/90.2 ms editor first frame 38.7 -> 13.9 ms exports unchanged
After answering a request the MF decoder reads ahead up to 8-30 frames before it looks at its channel again. A sequential reader (export, playback prefetch) therefore waited out the rest of the batch for its next frame, then found the decoder idle while it consumed the cached ones, so decode and render alternated instead of overlapping. Poll for requests between samples, answer cache hits immediately and slide the readahead window with them; misses are handed back to the normal request path. Answering from the cache mid-batch exposed a timing dependence that already existed: the decoders number samples at the recorded fps (59/57/30 on the fixture), so a VFR stream regularly has two samples rounding to the same frame and the later one replaces the first in the cache. Whether a request got the first or the second depended on whether it arrived before the second was decoded, so repeated exports differed by a source frame at a few dozen positions. A frame is now answered only once the decoder has moved past it, which always yields the last sample for that frame; repeated exports are byte-identical. capbench, interleaved, same build with the window slide toggled (n=4): export 1080p60 9.79 -> 8.45 s wall (-14%), 15.7 -> 18.0 s CPU export 720p30 8.11 -> 7.72 s wall (-5%), 13.7 -> 15.2 s CPU and against the previous build (n=5): MF random access 87.7 -> 69.1 ms avg, 119 -> 97 ms p95. The extra CPU is mostly NVIDIA driver threads now running concurrently with rendering; 4K exports are NVENC-bound and unchanged.
…OP_STATS The renderer already accumulates per-stage time (decode wait, render, next-frame decode, join, send) behind CAP_RENDER_LOOP_STATS=1, but only the render farm read it. MP4 exports now log it at completion, together with the time the forwarder spent blocked on the encoder, which is what showed the 1080p export to be decode-bound and 4K to be NVENC-bound.
Every captured frame was copied into a freshly allocated FFmpeg frame (8 MB at 1080p, 33 MB at 4K). Allocations that large come straight from new OS pages on Windows, so each frame also paid the page faults and zeroing, measured at ~2 ms per 1080p frame (~9 ms at 4K) against ~0.45 ms for the copy itself. Take the buffers from an AVBufferPool instead; they return to the pool when the encoder and the duplicate-frame holder drop them. capbench, 10 s recordings of a 60 fps test pattern, 5 interleaved reps (with the cursor change that follows; instant mode has no cursor thread, so its number is this change alone): studio 60 fps 0.309 -> 0.248 cores (-20%), delivered fps unchanged instant 30 fps 0.205 -> 0.167 cores (-18%) peak commit +7 MB (the pooled buffers)
…nchanged The cursor thread rendered the cursor into a DIB, trimmed it, PNG-encoded it and hashed the PNG on every tick (~32 Hz) only to find the same id. That costs ~250 us per tick (~0.8% of a core for the whole recording) against ~0.5 us for returning the previous result when GetCursorInfo reports the same HCURSOR.
Windows exports held a 1 ms timer resolution because the NV12 readback wait slept in 1 ms steps. Readbacks now block on the GPU fence and nothing on the export's per-frame path sleeps: CLI exports (1080p60, 5 interleaved reps) take 8.62 s without the hold and 8.70 s with it, so the process no longer raises its timer resolution for the length of an export.
…ailure logs `cap-test validate` exits 0 for an invalid recording, so read its `valid` and `errors` and count such runs as failures instead of comparing them. Failure logs were written under the work directory that is deleted after the run; copy them to <out>/failures/ and record that path.
…rame ReadSample hands back a null sample for stream ticks (gaps in the stream), and read_sample reported those as Ok(None), the same as the end of the stream. The decoder then stopped a batch early and answered with an older frame. Read again until a sample, the end-of-stream flag or an error.
…est state machine The decoder thread mixed batching, a mid-batch readahead slide, deferred requests and four fallback chains in one loop. The policy now lives in FrameRequests, which never touches Media Foundation: the thread takes in requests, asks it for the next step (seek, decode one sample, or wait) and does it. Tests drive it with a fake sample stream. Once the decoder moves past a requested frame, the request gets the newest sample at or below that frame. Requests are served lowest first, each with its own seek when decoding forward cannot reach it, so the "first sample after" fallback only applies to a frame before the stream's first sample. The cache keeps every sample since the last seek and evicts the oldest, so the newest sample at or below a frame is always cached. Behaviour changes: - Several requests in flight (export transitions between two segments of one recording decode both through the same decoder) each get their own frame. The highest used to be the target, and a lower one far from it got the target's frame. - A request behind the decoder with nothing cached at or below it seeks back. Within half the cache size it used to get the next decoded, later frame. - Every request slides the readahead, cache hits included, so a sequential reader no longer drains the cache while the decoder idles. - Readahead stops at its frame count, or before the next sample would evict the one answering the latest request (frames too big for the cache budget used to get a later frame); the per-batch time budget is gone. - Samples outside the old cache window are cached too, so a frame after a VFR gap past the window gets its own sample, not the one before the gap. - A failed seek answers what waits from the cache instead of reading on from the old position as if it had landed. Export output is byte-identical to the previous decoder on the reference recording.
The decoder thread copied each sample to a staging texture and mapped it straight away, so it sat blocked in Map while the GPU decoded and copied, and the GPU sat idle while the CPU copied the frame out. read_nv12 now queues the copy of the next sample before mapping the current one, through a ring of two staging textures created with the decoder. Samples are copied from Media Foundation's output surface directly into staging, skipping the pool texture that only the disabled zero-copy path needs. The rendering decoder caches the CPU NV12 data itself and drops the zero-copy fields its frames could never fill, along with the two zero-copy constructors that were their only producers.
Contributor
|
@cesarfrancots is attempting to deploy a commit to the Cap Software Inc Team on Vercel. A member of the Team first needs to authorize it. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… job An export that ffprobe cannot read, that has the wrong frame count, or that falls outside the requested size is now an invalid run instead of a timing sample. Benchmark processes start suspended and resume only once they are in the job object, so nothing they spawn escapes its CPU accounting, and a failed assignment kills the process. A note says why playback forces FFmpeg.
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every Media Foundation frame is read back to the CPU, because the zero-copy path has been off since b971194. The decoder thread used to copy each sample to a staging texture and
Mapit straight away. As a result:Mapwhile the GPU decoded and copied the sample.Profiling the 3024×1964 screen track of the reference recording gave these per-frame costs:
ReadSampleMap(waiting for the GPU)In the 1080p60 export, waiting for the next frame's decode (4.4 s) took longer than rendering (3.2 s). Decoding was the critical path.
MediaFoundationDecoder::read_nv12pipelines the readback:read_nv12returns exactly what the one-at-a-time path did. It drains the queued sample before reporting end of stream or a read error, and a seek drops whatever is queued.The rendering decoder now caches
NV12Datadirectly. Its cached frames had zero-copy fields that were never filled; those are gone, along with the texture-read-failure tracking and twoDecodedFramezero-copy constructors that this branch was the last caller of. A failedMapis now reported as a read error. Previously it cached a black frame.Verification
a88dca3b…).pipelined_readback_returns_every_sample_in_orderis#[ignore]like the existing device test, so CI doesn't run it. Run it withcargo test -p cap-video-decode --lib -- --include-ignored.(pts, pixels)pair fromread_nv12against a one-at-a-time readback.cargo test -p cap-rendering --lib media_foundation: 13 pass.cargo clippy -p cap-rendering -p cap-video-decode --all-targets -- -D warningsis clean.scripts/capbench.pyagainst the refactor(rendering): drive the Media Foundation decoder from one request state machine #2545 build (n=8). Each row is the median, with a bootstrap 95% CI.The playback benchmarks run the FFmpeg decoder, because that is what the editor uses on Windows, so they do not exercise this change and are left out of the table. 4K export is limited by the encoder (
encoder_senddominates the loop stats), so this change doesn't move it. Random access needs one extra sample decoded after each seek before the first frame comes back. It leans about 3 ms slower, but the CI still crosses zero.decode-displayrandom access also varies by about 20% between sessions on the same build. The gain is modest; the clearest win is the 1080p export.Notes for reviewers
scripts/capbench.py(from perf(windows): fence-waited readbacks, overlapped MF export decode, cheaper screen capture #2541) now:read_sample,MFDecodedFrame,FramePoolandNv12PlaneConverterare untouched and still public. Nothing in the repo reads frames through them any more, so they are the scaffolding for a future zero-copy path. Don't mixread_samplewithread_nv12on one decoder:read_nv12keeps one sample queued. If zero-copy isn't coming back, they and the plane shader can be deleted in a follow-up.On the reference recording and the
h264-decoder-lifecycle.mp4fixture, Media Foundation ends each stream withMF_SOURCE_READERF_ERRORrather than end-of-stream. That was true before this PR too: theStream errorwarnings at segment ends are not new.🤖 Generated with Claude Code
The latest changes appear safe to merge, with no new actionable defects found.Summary
This PR pipelines Media Foundation readback through two staging textures and caches CPU frame data directly.
Reviews (2) · Last reviewed commit: "fix(scripts): reject broken exports and ..." · Reviewed by Greptile