Repository navigation
feature: the hardware half of USB video capture - #153
Conversation
MJPEG from a USB grabber, decoded by the P4's JPEG engine and area-averaged into the picture at the size the service asks for. The host runs behind CONFIG_MM_USB_VIDEO, so only the P4 images carry it. Modes are read when the device enumerates, so a board with no grabber fails before the open's timeout, and a continuous-range device is offered a rung per rate a picture asks for rather than only the ends of its range. The driver takes the frame size from the negotiation instead of a guess here. noiseFloor cuts a grabber's near-black to black, as one compare on a byte. staleMs drops the picture when a source stops, rather than leaving the room lit from a frozen frame. The status line names the live rate per second and only what is going wrong. Measured on an ESP32-P4 with a 640x480 grabber: 60 fps into a 60x34 picture, nothing skipped, nothing failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The capture's own controls already appear behind hasUsbVideo, while the source offering it did not, so a board without the transport listed a source whose open could only answer that no grabber is attached. A config carrying the capture by label ports across either way: the Select apply matches on the name, and one that is not listed matches nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
 |
|
@ewowi Hi! |
The capture task read noiseFloor straight off the control while the render thread could be writing it, which threadsanitizer reports as a race on the byte. The floor is now taken in prepareBuffers beside bufW_ and bufH_, before the task exists, so the task owns every value it decodes through. An edit re-prepares, so a new floor still reaches the next frame. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e allocation A grabber unplugged and plugged back now streams again without touching a control, and one plugged in after a failed open is picked up. The USB host and the UVC driver live only while USB capture is the source, so a P4 that never selects it holds none of their RAM. **Core** - PictureService watches videoCaptureFormatGeneration() once a second and rebuilds when it moves: an unplug closes the stream, which frees the port, and a replug opens again. The generation only grows and reads 0 while no device is listed. - Scheduler::instance() and requestPrepareTree() are MM_NONBLOCKING, so a tick may ask for a rebuild. **Platform** - videoCaptureShutdown(): the UVC driver uninstalls when the source leaves or the service is released, and the pump then frees the device and uninstalls the host library, as the MIDI host does. - The frame list, format bank and snapshots (about 3 KB) are allocated with the install instead of static internal RAM. - The frame size and MCU come from the JPEG's SOF header (usbvideo::jpegShape), replacing jpeg_decoder_get_info, which allocated per frame. - uvc_host_desc_print removed from the open, about 100 ms on the render thread; the modes are still logged. - The P4 sdkconfig comment wrapped. **Tests** - A grabber unplugged and plugged back is opened again (fails without the generation watch). - A JPEG's size and MCU from its frame header; a truncated or foreign frame has none. - A byte at the noise floor reads as black, one above it passes through. - Desktop 2682/2682; esp32p4rev1-eth builds. Hardware test of replug, source switching and 60 fps: Gohnnyman's rig. **Reviews** - Fable review of MoonModules#153: replug, install lifetime, static internal RAM, the descriptor print, the per-frame allocation, the generation reset and the floor test: done here. Transfer-error restart, capture stack and CONFIG_UVC_INTERVAL_ARRAY_SIZE: left for the bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @Gohnnyman, great work; 60 fps into the picture with nothing skipped is exactly what this needed. I pushed a maintainer commit on top of yours (29671e6) with the fixes from the review, so you don't have to chase them:
Could you test on your rig?
If you can, also check whether a transfer error (e.g. pulling the HDMI side) restarts the stream, and the capture task's stack high-water mark once. The docs how-to from your setup write-up is next; we'll draft it from your PR text. Processes behind the review: MoonWalk. |
Switching away from the capture and back failed the re-open and stayed failed: the UVC driver cannot install while the previous teardown is still unwinding, and nothing asked again. videoCaptureSettling() answers whether a teardown is in flight, the open is not attempted while it is, and tick1s asks again until it has finished. The ESP32 teardown no longer blocks the render thread for a second waiting on the pump; it hands the rest to the pump and returns. The status names the step a failed open stopped at rather than guessing at the mode list, which is what a bus still shutting down had been reported as. A device that goes away now puts the lights out on the tick that notices it: the stale drop runs before the generation branch can return, and prepareBuffers no longer publishes a writer's buffer, which was re-lighting the frame the drop had just put out. Also from the review: the frame list is null-checked in the driver callbacks, since a shutdown may free it between events; two USB host failures that had been reported as no grabber attached name themselves; and the status line reads the size a writer fills, so a dropped picture no longer reads as 0 x 0. Desktop 2684/2684; esp32p4rev3-eth-wifi builds. On the rig: switching away and back after 2 s, 1 s and immediately all recover inside two seconds and hold 60 fps into a 60 x 34 picture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@ewowi Hi! I fixed several things:
Everything is fixed and tested
One thing I noticed is that test pattern is whole white. It doesn't looks like a test pattern for me, for example user want be able to figure out whether he set up Layout properly or not. I'll update it in the next PR, probably I'll just add static colors on four borders |
…e Picture service Brings the branch up to main, which also fixes CI's check_docgen crash in libclang's bindings. A grabber arriving, leaving or a teardown settling now rebuilds the Picture service alone rather than every layer and driver. **Core** - Scheduler.h: the one conflict resolved to main's version (requestPrepare, the live queue), keeping this branch's MM_NONBLOCKING on requestPrepareTree and instance(), and adding it to requestPrepare, which tick1s now calls. - PictureService: tick1s asks requestPrepare(this) instead of requestPrepareTree(): the picture keeps its size, so no reader re-prepares; a comment's "no longer" reads "does not carry". **Reviews** - Maintainer review of 61c17f6: the CI prose finding and the docgen crash fixed here; the suggestion to scope the rebuild applied. - Checks: desktop build and 2764 tests pass; check_docgen 0 errors; specs up to date; esp32p4rev1-eth builds and, on an ESP32-P4 rev 1.3 with an MS2109-type grabber, shows a Windows PC's HDMI on a 128x128 grid sent as Art-Net. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… out Prose rose by one Weasel finding in CI: "just" here meant "a moment ago", which the rule cannot tell from emphasis. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged, thanks @Gohnnyman, and thanks for testing all three on your rig. Recovering inside two seconds at 60 fps is exactly right. I read 61c17f6 and it looks good. The teardown no longer holds the render thread, a failed open says where it stopped, and the unplug drop has its test. The null checks on Before merging I added two maintainer commits to get CI green:
I tested it on an ESP32-P4 (rev 1.3, Ethernet) with an MS2109-type HDMI grabber, a Windows mini PC as the source, a 128×128 grid, and Art-Net out. Video shows end to end. One thing I measured, for your next round: the capture rate falls with the picture's size, whatever the rendering does.
The grabber opened 640×480 @ 60. Is the step from the decoded frame down to the picture done per output pixel in software? On your rig, which mode do you get at 60 fps? I also see A test pattern that shows the layout's edges is a good idea for the next PR. Next we draft the docs how-to from your setup write-up. |
Untitled.mov
|
|
@Gohnnyman , I will see if I can also get this supported on Linux boxes. Or do you also plan to do that ? |
What is Linux box? Edit: I didn't plan to do that because I'm not sure what that is |
|
A raspberry pi or nano pi (but also windows/macos), docker container Moonlight runs on these devices as well |
|
Ohh, right... There is the main point: Then I started investigating how I could do this. The main problem for me was that everyone said that I need raspberry pi or simillar for this. I thought "why do I need the whole pc (mini pc) just to compute some bytes of video?". It looked to me like a real overhead for a backlight setup. Also expensive (for a backlight). Then I found esp32 p4 that could decode mjpeg and process video frames in proper time. I'm not sure, but I didn't find any similar setup with microcontroller instead of mini-pc (raspberry pi-like). I think it's the firs one of the kind. I'm not sure but I'd be happy if you prove me wrong 😄 So I aimed at this specific setup. ESP32 P4 and that's it. I didn't plan to make it work on any other platform, but I'd be happy to see it work anywhere else |
|
I am not trying to prove you wrong as I like the idea we created something unique 🙂. |
What's new
Fills the capture source the previous PR left stubbed. MJPEG from a USB
grabber, decoded by the P4's JPEG engine, area-averaged into the picture
at whatever size the service asks for.
CONFIG_MM_USB_VIDEO, so only the P4 imagescarry it (+126 KB flash).
fails fast instead of waiting on an open.
noiseFloorcuts a grabber's near-black to black;staleMsdrops thepicture when a source stops, rather than leaving the room lit from a
frozen frame.
Measured on an ESP32-P4 with a 640x480 HDMI grabber: 60 fps into a
60x34 picture, nothing skipped, nothing failing.
Note
What's left to implement is HDR decoding and channel balances, it'll be done in next PRs
My Setup
Active Parts
Basically my whole setup was created for PS5. I have these active parts:Connection logic
The logic of the whole setup is:
Note
During my testing I also tried this HDMI USB Grabber. As I remember, I swapped for the model above because sometimes it flickered with green. But it was long time ago and now I'm not sure whether the problem was in grabber or in code itself. Also Hyperion mentioned in their docs that this grabber not so good.
What's working and what's not
Working:
Not working: