Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions bench/camera-passthrough-live-view/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Camera passthrough for live view — reproduction & findings

Harness and findings behind the production code in this PR (`server/lib/camera`,
`server/cmd/camerad`, the wrapper/neko wiring, and the client controller in
`images/chromium-headful/client/src/kernel/camera.ts`). Follows the
guest-kernel / loopback-module spike in kernel-images#515.

## What this enables

A live-view viewer shares their **real** camera into the browser so a page
inside it can use it. On demand: the viewer's camera turns on when a page calls
`getUserMedia()` and off when it stops. This is for a real person completing
their own camera step — **not** for feeding synthetic or recorded video to a
liveness check.

## Findings (validated in a real browser, 2026-11-16)

1. **Passthrough works.** A page's `getUserMedia()` auto-starts the viewer's
camera; the browser receives live 1280×720; the camera stops ~3s after the
page releases it. Verified on the WebRTC getUserMedia sample.

2. **Liveness SDKs reject the virtual camera, by design.** A widely used
browser liveness demo refuses the session with an "a 3rd-party app is
modifying the video / suspicious webcam configuration" error — it detects the
v4l2loopback device. **Camera passthrough alone will not carry a liveness
step on a vendor that rejects virtual cameras.** We do not attempt to
disguise the device to defeat that detection: the check exists to confirm a
real person is physically present. Passthrough is for flows whose vendor
permits a relayed camera, and for non-liveness camera use (video, capture).

3. **Orientation (known limitation).** Capture is pinned to 1280×720 landscape
end to end, so rotating a phone does not change the camera aspect. The client
controller re-requests `getUserMedia` on `orientationchange`; native
per-orientation device caps (reloading the loopback) remain a follow-up.

## Reproduce against a running browser (no image build)

Inside a browser VM with the camera kernel module available (see
kernel-images#515 for the guest-kernel bits):

```sh
# load the loopback pair, run the manager, enable neko webcam ingest
camerad & # prebuilt binary from this PR
NEKO_CAPTURE_WEBCAM_ENABLED=true NEKO_CAPTURE_WEBCAM_DEVICE=/dev/video1 \
supervisorctl restart neko
```

`kernel-camera.injected.js` is the exact client controller used in the live
test, as a drop-in `<script>` for a browser where you can't rebuild the neko
client; `src/kernel/camera.ts` is the same logic as a real client module.
13 changes: 13 additions & 0 deletions bench/camera-passthrough-live-view/camwatch.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash
# Publishes whether a page in Chrome is using the camera (prototype).
STATE=/var/www/camera-state.json
last=""
while true; do
chrome=false
for p in $(pgrep -u kernel chrom); do
if ls -l /proc/$p/fd 2>/dev/null | grep -q '/dev/video0'; then chrome=true; break; fi
done
s="{\"wanted\":$chrome}"
if [ "$s" != "$last" ]; then echo "$s" > $STATE.tmp && mv $STATE.tmp $STATE; chmod 644 $STATE; last="$s"; echo "$(date +%T) $s"; fi
sleep 0.3
done
59 changes: 59 additions & 0 deletions bench/camera-passthrough-live-view/kernel-camera.injected.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Prototype: camera on demand for live view. Not production code.
// When a page inside the Kernel browser asks for the camera, the viewer's camera turns on
// automatically; when the page stops using it, it turns off. The button is a manual override.
(function () {
function client() { const el = document.querySelector("#neko"); return el && el.__vue__ && el.__vue__.$client; }
function connected() { const c = client(); return c && c._peer && ["connected", "completed"].includes(c._peer.iceConnectionState); }
let stream = null, sender = null, manual = false, starting = false, idleSince = 0;
const ui = document.createElement("div");
ui.style.cssText = "position:fixed;left:16px;bottom:16px;z-index:99999;display:flex;flex-direction:column;align-items:flex-start;gap:8px;font:13px system-ui,sans-serif";
const note = document.createElement("div");
note.style.cssText = "padding:6px 10px;border-radius:8px;background:rgba(33,34,37,.88);color:#edeef0;display:none;max-width:280px";
const btn = document.createElement("button");
btn.style.cssText = "padding:9px 15px;border:0;border-radius:999px;font:600 13px system-ui,sans-serif;box-shadow:0 2px 8px rgba(0,0,0,.35);cursor:pointer";
ui.append(note, btn);
const say = (m) => { note.textContent = m || ""; note.style.display = m ? "block" : "none"; };
function render() {
if (stream) { btn.textContent = "● Camera on — stop"; btn.style.background = "#e5484d"; btn.style.color = "#fff"; }
else { btn.textContent = "📷 Camera off"; btn.style.background = "#81b300"; btn.style.color = "#212225"; }
}
async function start(reason) {
if (stream || starting || !connected()) return;
starting = true;
try {
stream = await navigator.mediaDevices.getUserMedia({ video: { width: { ideal: 1280 }, height: { ideal: 720 }, frameRate: { ideal: 30 } }, audio: false });
sender = client()._peer.addTrack(stream.getVideoTracks()[0], stream);
say(reason === "auto" ? "The site asked for a camera, so your camera is on." : "");
if (reason === "auto") setTimeout(() => stream && say(""), 5000);
} catch (e) {
stream = null;
say(e.name === "NotAllowedError" ? "Camera blocked. Allow camera access for this page (icon left of the address bar) and try again." : "Camera error: " + e.name);
} finally { starting = false; render(); }
}
function stop() {
const c = client();
try { if (c && c._peer && sender) c._peer.removeTrack(sender); } catch (e) {}
if (stream) stream.getTracks().forEach((t) => t.stop());
stream = null; sender = null; render();
}
btn.onclick = () => { if (stream) { manual = false; stop(); say(""); } else { manual = true; start("manual"); } };
async function poll() {
try {
const base = location.pathname.replace(/\/?$/, "/");
const q = location.search ? location.search + "&" : "?";
let r = await fetch(base + "camera-state.json" + q + "t=" + Date.now(), { cache: "no-store" });
if (!r.ok) r = await fetch("camera-state.json?t=" + Date.now(), { cache: "no-store" });
if (r.ok) {
const s = await r.json();
if (s.wanted) { idleSince = 0; if (!stream) start("auto"); }
else if (stream && !manual) {
idleSince = idleSince || Date.now();
if (Date.now() - idleSince > 3000) { stop(); say("The site stopped using the camera, so your camera is off."); setTimeout(() => !stream && say(""), 4000); idleSince = 0; }
}
}
} catch (e) {}
setTimeout(poll, 700);
}
function mount() { document.body.appendChild(ui); render(); poll(); }
document.readyState === "loading" ? document.addEventListener("DOMContentLoaded", mount) : mount();
})();
8 changes: 8 additions & 0 deletions images/chromium-headful/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,12 @@ RUN --mount=type=cache,target=/root/.cache/go-build,id=$CACHEIDPREFIX-go-build \
GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH:-amd64} \
go build -ldflags="-s -w" -o /out/kernel-images-supervisord-shim ./cmd/supervisord-shim

# Build camera passthrough daemon (lib/camera)
RUN --mount=type=cache,target=/root/.cache/go-build,id=$CACHEIDPREFIX-go-build \
--mount=type=cache,target=/go/pkg/mod,id=$CACHEIDPREFIX-go-pkg-mod \
GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH:-amd64} \
go build -ldflags="-s -w" -o /out/camerad ./cmd/camerad

# webrtc client
FROM node:22-bullseye-slim AS client
WORKDIR /src
Expand Down Expand Up @@ -401,6 +407,7 @@ ENV WITHDOCKER=true

COPY images/chromium-headful/xorg.conf /etc/neko/xorg.conf
COPY images/chromium-headful/neko.yaml /etc/neko/neko.yaml
COPY images/chromium-headful/camera-placeholder.png /etc/kernel/camera-placeholder.png
COPY images/chromium-headful/99-kernel-keyboard.gschema.override /usr/share/glib-2.0/schemas/
RUN /usr/lib/*/glib-2.0/glib-compile-schemas --strict /usr/share/glib-2.0/schemas
COPY images/chromium-headful/setxkbmap /usr/local/bin/setxkbmap
Expand Down Expand Up @@ -436,6 +443,7 @@ RUN chmod +x /usr/local/bin/init-envoy.sh
COPY --from=server-builder /out/kernel-images-api /usr/local/bin/kernel-images-api
COPY --from=server-builder /out/chromium-launcher /usr/local/bin/chromium-launcher
COPY --from=server-builder /out/kernel-images-supervisord-shim /usr/local/bin/kernel-images-supervisord-shim
COPY --from=server-builder /out/camerad /usr/local/bin/camerad
COPY --from=server-builder /out/wrapper /wrapper

# Copy and compile the Playwright daemon
Expand Down
Binary file added images/chromium-headful/camera-placeholder.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
158 changes: 158 additions & 0 deletions images/chromium-headful/client/src/kernel/camera.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
// Camera passthrough controller.
//
// Lets a live-view viewer share their real camera into the browser so a page
// inside it (e.g. an identity-verification flow) can use it. The browser's
// camera manager (server lib/camera) reports, via /state, whether a page
// currently wants the camera; this controller starts and stops the viewer's
// camera to match (camera on demand), and offers a manual toggle.
//
// Scope: this relays a real person's real camera. It is not a way to feed
// synthetic or recorded video to a liveness check.

import { NekoClient } from '~/neko'

interface CameraState {
wanted: boolean
streaming: boolean
width: number
height: number
}

const STATE_POLL_MS = 700
const STOP_GRACE_MS = 3000

export class CameraController {
private client: NekoClient
private stream: MediaStream | null = null
private sender: RTCRtpSender | null = null
private manual = false
private starting = false
private idleSince = 0
private timer: number | null = null

constructor(client: NekoClient) {
this.client = client
}

// mount begins polling the on-demand signal and re-requesting the camera at a
// matching aspect ratio when the device orientation changes.
mount() {
this.poll()
window.addEventListener('orientationchange', this.onOrientationChange)
window.matchMedia('(orientation: portrait)').addEventListener('change', this.onOrientationChange)
}

unmount() {
if (this.timer !== null) window.clearTimeout(this.timer)
window.removeEventListener('orientationchange', this.onOrientationChange)
this.stop()
}

// toggle is the manual control. Manual-on keeps the camera on regardless of
// whether a page is currently requesting it.
async toggle(): Promise<void> {
if (this.stream) {
this.manual = false
this.stop()
} else {
this.manual = true
await this.start()
}
}

get active(): boolean {
return this.stream !== null
}

private constraints(): MediaStreamConstraints {
// Match the shared camera's aspect to the device orientation so the browser
// receives upright video. The server letterboxes onto its fixed native mode;
// per-orientation native caps are a follow-up (see PR notes).
const portrait = window.matchMedia('(orientation: portrait)').matches
const long = 1280
const short = 720
return {
video: {
width: { ideal: portrait ? short : long },
height: { ideal: portrait ? long : short },
frameRate: { ideal: 30 },
},
audio: false,
}
}

private async start(): Promise<void> {
if (this.stream || this.starting || !this.client.peerConnected) return
this.starting = true
try {
this.stream = await navigator.mediaDevices.getUserMedia(this.constraints())
const track = this.stream.getVideoTracks()[0]
this.sender = this.client.shareLocalCamera(track, this.stream) ?? null
this.emitChange()
} catch (err) {
this.stream = null
this.sender = null
this.emitError(err)
} finally {
this.starting = false
}
}

private stop(): void {
if (this.sender) {
this.client.unshareLocalCamera(this.sender)
this.sender = null
}
if (this.stream) {
this.stream.getTracks().forEach((t) => t.stop())
this.stream = null
}
this.emitChange()
}

private onOrientationChange = async (): Promise<void> => {
// Re-acquire at the new aspect ratio so the shared video stays upright.
if (!this.stream) return
this.stop()
await this.start()
}

private async poll(): Promise<void> {
try {
const state = await this.fetchState()
if (state) {
if (state.wanted) {
this.idleSince = 0
if (!this.stream) await this.start()
} else if (this.stream && !this.manual) {
this.idleSince = this.idleSince || Date.now()
if (Date.now() - this.idleSince > STOP_GRACE_MS) {
this.stop()
this.idleSince = 0
}
}
}
} catch {
// ignore transient fetch errors; the manager may be starting up
}
this.timer = window.setTimeout(() => this.poll(), STATE_POLL_MS)
}

private async fetchState(): Promise<CameraState | null> {
// The state file is served alongside the live-view page, under the same
// authenticated path, so a relative URL carries the page's credentials.
const base = window.location.pathname.replace(/\/?$/, '/')
const sep = window.location.search ? window.location.search + '&' : '?'
const res = await fetch(`${base}camera-state.json${sep}t=${Date.now()}`, { cache: 'no-store' })
if (!res.ok) return null
return (await res.json()) as CameraState
}

private emitChange(): void {
window.dispatchEvent(new CustomEvent('kernel:camera', { detail: { active: this.active } }))
}

private emitError(err: unknown): void {
window.dispatchEvent(new CustomEvent('kernel:camera-error', { detail: { error: err } }))
}
}
21 changes: 21 additions & 0 deletions images/chromium-headful/client/src/neko/base.ts
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,27 @@ export abstract class BaseClient extends EventEmitter<BaseEvents> {
this._ws!.send(JSON.stringify({ event, ...payload }))
}

// shareLocalCamera adds a local camera track to the existing peer so it is
// delivered upstream to the browser VM (where it becomes a webcam device).
// Adding a track fires onnegotiationneeded, which renegotiates over the same
// peer. Returns the sender so the caller can stop sharing later. Used by the
// camera-passthrough controller (see kernel/camera.ts).
public shareLocalCamera(track: MediaStreamTrack, stream: MediaStream): RTCRtpSender | undefined {
if (!this._peer) {
this.emit('warn', `cannot share camera before the peer is connected`)
return undefined
}
return this._peer.addTrack(track, stream)
}

// unshareLocalCamera removes a previously shared camera sender, renegotiating
// the peer so the browser VM stops receiving the track.
public unshareLocalCamera(sender: RTCRtpSender): void {
if (this._peer) {
this._peer.removeTrack(sender)
}
}

public async createPeer(lite: boolean, servers: RTCIceServer[]) {
this.emit('debug', `creating peer`)
if (!this.socketOpen) {
Expand Down
9 changes: 9 additions & 0 deletions images/chromium-headful/supervisor/services/camerad.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[program:camerad]
command=/usr/local/bin/camerad
# Started by the wrapper only when a session enables the camera capability
# (KERNEL_ENABLE_CAMERA=true); see cmd/wrapper.
autostart=false
autorestart=true
startsecs=0
stdout_logfile=/var/log/supervisord/camerad
redirect_stderr=true
Loading
Loading