Repository navigation
feat(desktop): local CopilotKit Intelligence from the app alone, in the CopilotKit visual system - #832
Draft
dustywusty wants to merge 101 commits into
Draft
feat(desktop): local CopilotKit Intelligence from the app alone, in the CopilotKit visual system#832dustywusty wants to merge 101 commits into
dustywusty wants to merge 101 commits into
Conversation
…nded native boundary local_eval.rs runs a SHA-256-pinned Intelligence CLI with fixed arguments only (local status|start|stop --stack ID --json, local list --json, local setup --defer-model --json), no shell, an environment allowlist, an owner-only output file capped at 1 MiB, per-action timeouts, per-attempt cancellation and process-group cleanup, one call at a time. It normalizes the CLI's version-1 JSON to allowlisted facts; readiness and licence stay unknown because v1 cannot state them. Release builds run no CLI; debug builds accept a developer pin by path and digest. local_eval_state.rs records the local evaluation checkpoint (stack, created or shared, CLI digest, phase). preparation.rs shares its atomic checkpoint write. localEvalCli.ts is the window's typed client.
…p and Quit Stop and Quit ask the CLI to stop the local stack after OpenBot's own cleanup and Compose down, only when OpenBot created it. In-flight calls are cancelled and new ones refused while closing; Quit allows the CLI two minutes and no longer waits behind a Stop. Intelligence failures are reported apart from OpenBot's and never mark OpenBot for recovery; a failed stop at Quit is kept for the next launch. Adds the Tauri commands for status/start/stop, list, setup and cancel, and exposes the checkpoint as AlreadyConfigured.localEval.
…ure split Connect offers "Use Intelligence on this computer" only when a pinned CLI answers. The panel shows independent Docker Desktop, Intelligence, OpenBot and provider rows, runs setup for this installation, and keeps Start disabled because the CLI does not report readiness or give OpenBot a credential. A Stop that fails only for Intelligence marks OpenBot stopped and shows the Intelligence failure separately; a failed stop at Quit is shown on the next launch without holding auto-start.
…uests The v2 requirements artifact (decisions, architecture, coding sequence, progress and release gates), the original r21 artifact unchanged, the implementation handoff, and the contract requests for the Intelligence CLI owners.
…utomatic step 4 Local checkpoint for the demo; not for review as-is.
…t step for local Local checkpoint for the demo; not for review as-is.
… and automatic test Local checkpoint for the demo; not for review as-is.
…temporary model-field debug line The debug line logs field names and token length only, never values. Remove before review.
…minal Local checkpoint for the demo; not for review as-is.
…e status for slow actions Local checkpoint for the demo; not for review as-is.
…o menu-bar status text The menu-bar title is gone (tray-icon never clears it on macOS). An Intelligence row at the bottom of the app sidebar opens the local dashboard through a two-command plugin granted only to the owned loopback app origin. After the first sign-in of a Desktop run, Open sends the stored dashboard address straight to the browser with no CLI call. Debug builds hash the CLI pin with an optimised sha2 (4.86 s to 0.29 s).
…n Intelligence; question box with presets The test question went straight to the harness, which keeps its thread in memory, so Intelligence never saw it and View in Intelligence opened a dashboard with no conversation. A picked Bot is now asked through the owned server's runtime: the server mints the thread, the runtime records the run in Intelligence, and the desktop reads the thread's events until RUN_FINISHED or RUN_ERROR. The local test step is a question box again: a question already in it, presets one click away (strawberry r's, Die Hard, Pop-Tarts, 17 x 23), your own typed question, and Ask again after an answer. Nothing is asked until Ask.
…d build time Several worktrees build this app on one machine and the window looks the same from each, so which build is running was a guess. build.rs records the commit (marked -dirty for uncommitted changes), the branch and the build time, rerunning on source and git changes; About OpenBot shows them.
… that cycles through the rest The preset chips crowded the screen. The box starts on one question at random; Another question beside it steps through the others in order and back round; typing your own still works.
…ilotKit hosted, assuming neither Install with a small 'hosted instead' link assumed local. The first screen now offers both as equal choices with nothing selected, and Continue waits for one.
…thin its screen 900x640 was too short for Connect your AI once a provider's sign-in opened under its row: the page scrolled and the step label slid under the title bar. The window opens at 1000x800, brought within the monitor's work area on launch, and the setup page asks the shell to grow it whenever a step is taller than the window. fit_setup_window only grows, never past the work area, and leaves a maximized or full-screen window alone; past that the page scrolls as before. The page gets no core:window permission.
…ndow The root size goes from 14px to 16px; everything else is in rem, so type, column width and spacing scale together. The answer card's fixed px sizes become rem so it scales with them.
…n only Upstream's self-hosted sign-in sets signingIn too and says its own line; both showed at once.
… shared or toolkit manifest #784 added shared and toolkit to the packages whose manifests preparation reads, but the latest release, v0.2.0, predates it: shared is plain source and there is no toolkit. A desktop built from main failed every new install with 'shared/package.json: No such file or directory'. Those two are now read only when the release has them; the four every release has stay required.
…to choose Sam's call: one fewer decision before anything is installed. With a CLI the first screen is Install, which sets up OpenBot and Intelligence on this computer; the hosted choice and the 'hosted instead' link are gone from it. The hosted way out after a failed local setup stays.
After a Claude code is accepted, the ChatGPT redirect lands, or a Gemini/xAI OAuth completes, the screen continues as Continue would. Checked once right after the sign-in, so a field filled in later never moves the screen on from under somebody typing; a saved sign-in and a typed key still wait for Continue.
…hen it is not; never Podman in its place With Docker Desktop installed but quit, setup found no answering engine and installed Podman behind an administrator prompt: a second engine on a machine that already had Docker, and one local Intelligence refuses. On macOS and Windows the engine is now Docker Desktop. Installed and stopped, it is opened and waited for. Missing, Docker's own installer (4.96.3, pinned to the digests Docker publishes) is downloaded with progress, installed without accepting Docker's terms, and opened so the person accepts them in Docker's window. A Podman somebody already has is still used; a stopped docker with no Docker Desktop (OrbStack, Colima) is left for its owner to start. Linux is unchanged.
…hoice moved on by itself
The local sequence starts OpenBot a tick after the go-ahead and read the model from the last render.
A Continue click has rendered by then; a sign-in that finished and continued from an effect may not
have, so with Intelligence already ready Start ran with model null ('invalid type: null, expected
struct ChosenModel'). The choice is now carried in a ref set when it is made, and the sequence reads
that. The Claude plan test again signs in only after Intelligence is ready.
…ing on '✓ Signed in to <provider>.' appears beside Continue, then the screen continues by itself after CONFIRMED_FOR_MS (1 s). Choosing another provider row in that second cancels the move.
…ad of repeating it The tick was a second 'Signed in to <provider>.' line under the first, so for one second the screen (and a screen reader) said it twice, and tests that look for the sentence failed whenever they looked inside that second. The tick now goes in front of the sentence already there.
…d CLI, CopilotKit sign-in and the agreement in the window A customer opens OpenBot and presses Install; nothing is done in Terminal. - A build made with OPENBOT_BUNDLED_CLI=1 carries the Intelligence CLI in its resources (Node and the CLI, tauri.bundled-cli.conf.json) and runs it pinned to the digests build.rs recorded, in a release build too. Other builds are unchanged. - When setup needs a CopilotKit account, the window offers Sign in to CopilotKit, runs 'login --json' (the CLI opens the approval page), and setup carries on by itself. - When setup needs the evaluation agreement, the window reads it with 'onboard local-terms --json' and shows the organization, a link to read it, and Accept. The click is the person's acceptance (product decision, 2026-10-22) and setup runs again with --accept-eula-version and that version; nothing accepts it without the click. The 'run this in Terminal' footnotes are gone. - The CLI gets Docker's own directory on its PATH: opened from Finder the app has the system's minimal PATH, which does not reach Docker Desktop's /usr/local/bin.
… longer than 30 seconds deployment::get read the body with bytes(), which the blocking client bounds by its 30-second default timeout and reports as 'error decoding response body' when that runs out. Any download taking longer than 30 seconds failed however steadily it arrived: Bun's 30 MB on a connection under about 1 MB/s, the Compose and Podman downloads, and the release lookup. Present since get() was written (#392). The body is now read as a stream, where the timeout bounds each read, so a download is given up only after 30 seconds with nothing arriving. A slow-server test fails with the old code's exact message and passes now.
…step of their own in a local first run Sign-in and Accept sat on the AI provider screen, under the Intelligence status line. A local first run is now four steps: install OpenBot, then CopilotKit (sign in, then read and accept the evaluation agreement), then choose the AI while Intelligence sets up behind it, then test. Nothing licensed starts before the account step finishes, and setup runs with the version accepted there. Somebody already signed in with the current agreement accepted passes the step without a click. The provider screen no longer offers sign-in; the Getting ready panel keeps it as a fallback for a session that ends later. step_viewed telemetry gains 'account'.
…ot's CLI calls like its own Quit does Quitting OpenBot from the Dock or with an AppleScript quit left a running 'local setup' behind, orphaned with its sign-in worker and an image pull; the next setup was refused as local_command_busy. Two causes. - Tauri on macOS answers a quit macOS starts only in applicationWillTerminate:, which ends the event loop at once, so RunEvent::ExitRequested, where OpenBot's Quit cleanup lives (CLI calls, containers, the Intelligence stack it made), never ran. The app delegate now also answers applicationShouldTerminate:, cancelling macOS's termination and asking for OpenBot's own Quit, whose cleanup then exits. Tauri leaves with stop: and process::exit, so it is not asked again. - close_for_quit only asked calls to cancel. The CLI forwards SIGTERM to its children instead of exiting, so its tree went only at SIGKILL after the grace, and an app that exited sooner left it running. close_for_quit now waits, up to 10 s, until every CLI process tree it started is gone. A CLI that ignores SIGTERM is tested: the test fails without the wait. The menu's Quit and its Cmd-Q already went through the cleanup and are unchanged.
…and Intelligence sets up alongside the install The order is now: CopilotKit (sign in, accept the agreement), Installing OpenBot, Connect your AI, then Getting ready and Test. The account needs nothing installed, so it comes first. Intelligence starts once the install reports the container engine ready (it runs on the same Docker, which the install starts or installs), not after the whole install, and its status shows on the install screen. Connect your AI follows the install because a plan sign-in runs in the image the install brings; Intelligence carries on behind it. set_up_local_eval creates the installation folder, which may not exist yet when it starts.
dustywusty
marked this pull request as ready for review
October 27, 2026 22:04
dustywusty
requested review from
MikeRyanDev,
davidmckayv,
guidovizoso,
mxmzb and
tylerslaton
as code owners
October 27, 2026 22:04
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6253bce77
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+3449
to
+3450
| for permission in desktop_page::PERMISSIONS { | ||
| capability = capability.permission(permission); |
There was a problem hiding this comment.
If the main webview navigates to another HTTP service on any loopback port, window_page classifies it as OwnedApp or LocalApp, and the page-load handler calls this function without checking deployment ownership. This grants that unrelated origin all of desktop_page::PERMISSIONS, including commands to stop/restart OpenBot and control Intelligence. Unlike show_openbot_on, this path never calls owned_app_url to verify the service. Restrict these grants to the verified app origin rather than treating every loopback page as trusted.
Useful? React with 👍 / 👎.
Comment on lines
+3685
to
+3686
| #[cfg(not(unix))] | ||
| let ended = child.kill().is_ok(); |
There was a problem hiding this comment.
On Windows, Restart kills only the held child process, not its descendants. stack::spawn_host_process launches the frontend with bun run serve, and app/package.json defines that script as bun scripts/serve-or-build.ts && bun serve.ts, so the actual HTTP server is a descendant that survives child.kill(). The supervisor then launches another frontend against the occupied port, causing repeated startup failures while the old frontend remains running. Terminate the verified process tree before allowing the supervisor to replace the parent.
Useful? React with 👍 / 👎.
Takes main's engine choice (#831): Docker Desktop is opened only when it has run before, and each installation stays on the engine it was set up on. The old copy of #831 on this branch is dropped, including after_engine_install, which duplicated main's answering_after_install and ignored the saved engine. Kept from this branch: path_with_docker, downloads chosen by the machine's architecture on Windows on ARM, and the per-user Docker Desktop location on Windows. Takes main's #823 and #821 (StartFresh replaces DatabaseReset; the local Intelligence screen offers it too), #860, and #858's Recorder step label. The web app keeps main's theme and skin picker (#856); this branch's CopilotKit reskin of the app and its two fonts are dropped. Its desktop CSS (room for the macOS window buttons, sidebar resize) stays.
…n addresses SERVICES now includes server and worker, and Stop names the openbot profile they sit behind. The settings point every Bot at server:3001, give the server host.docker.internal addresses for local Intelligence (the browser keeps the localhost ones), and give an installed harness its container address. The macOS Podman overlay publishes the server's two ports on IPv4 only.
… host Start now hands everything to Compose and waits for the `server` container to answer `/health` on both the app and API host ports, for up to 180 s. Docker's restart policy supervises the server and worker, so the host launch, readiness, pid recording and restart loop are gone. - The desktop host token is minted before `up` and reaches Compose, which names it for the server alone; it is still never persisted. - COMPUTER_NETWORK is `<compose project>_default` (new `stack::project_name`), so Bot computers join the server's network. On Linux OPENBOT_SERVER_USER is the deployment folder's uid:gid. - `.openbot/` is created before `up`, since its bind mount does not create it. - App and server ports count as ours when our container publishes them; the pre-spawn port check is gone. - Ownership for showing, restoring and asking OpenBot is now "this root's server container publishes the ports" (`stack::server_publishes`), not recorded host pids. - Status lists server and worker as containers; Restart runs `compose restart server worker`. - What is left of the supervisor polls the server for refused connections (`watch_connections`). - Start no longer requires the host Bun or packages. - Old Bun host processes from earlier installs are still stopped on Start and Stop, which is how an upgrade frees 3001/3010. Also brings tests left stale by the previous commit up to date (server/worker in the up list, `--profile openbot` on down).
Installing OpenBot is now pulling images. Bun and bun install are prepared only when somebody starts the self-hosted Intelligence sign-in, the one thing on the host that still runs a Bun script, and reused after that.
Rootless Podman maps container root to the person who started it and any other uid to one of their subordinate ids, so passing the owner's uid:gid made the server a stranger to the owner-only .openbot sign-in folder. Docker keeps the owner's numbers; Podman now gets 0:0.
A release from before the server and worker were Compose services has no server or worker to raise, and its server image has no worker, so keeping its pin left a working installation unable to start. Such a deployment now reads as needing installation, Install resolves the latest release instead of repairing the old pin, and if even the latest still runs the server on the host the app says so rather than failing inside Compose.
…ring" The Start screen showed the wait for /health as a step named "Answering", "waiting for OpenBot to answer". It is the longest step on a cold machine, so name it for what the person is waiting for and say it can take a few minutes the first time.
StatusWindow.test.tsx registered happy-dom at module scope and then imported @testing-library/react, so React DOM was first loaded with happy-dom's globals installed. bun test shares one module cache across files, so every desktop test file that ran after it used that React DOM, whose scheduler timers stopped firing once this file unregistered. Async updates never rendered and 116 tests timed out, but only when this file ran first, which depends on directory order on Linux. Import Testing Library statically and register in beforeAll, as the other desktop test files do.
…containers Main's OpenBot Browser (#858) runs agent-computer on the host with Bun, to open the person's own Chrome or Edge. The server and worker are containers now, so the browser is the only host process left: - Start prepares Bun and the packages the first time it has a browser to run, as the Intelligence sign-in helper does, after the server answers. It spawns the browser under the startup lock, records it so a later window can end it, and holds it. - The server container reaches it at host.docker.internal (OPENBOT_BROWSER_CONTAINER_URL, which docker-compose.yml prefers), since the computer listens on the host's loopback. - It is restarted with backoff when it dies. Giving up says why, with the tail of its log, and leaves OpenBot running. - Stop closes the held browser before the recorded processes are ended, so Chrome closes its windows and profile first.
…man machine Which engine OpenBot uses is main's (#831): Docker if it answers, Docker Desktop if it has run before, otherwise Podman, saved with the installation. This commit no longer changes that choice. Local Intelligence's k3d needs rootful Podman (CopilotKit/Intelligence#2147), so the machine OpenBot creates is `--rootful`, and OpenBot addresses it through its `<name>-root` connection, the rootful engine, both for its own machine and for a running machine it adopts. On a rootful Windows machine the host-gateway setting goes to the system configuration. An existing rootless machine is never switched; local Intelligence setup and start decline there instead, with a `rootless-podman` failure whose row names `podman machine set --rootful`. Every Intelligence CLI process gets COPILOTKIT_LOCAL_ENGINE for the engine OpenBot runs on, recorded when the engine is ready (also where Docker answers right after the install step) and when Start pins it; before either, the installation's saved engine. For a Podman machine it also gets CONTAINER_HOST and DOCKER_HOST naming the machine's forwarded API, drops an inherited DOCKER_CONTEXT, and has Podman's directory on PATH. While Podman installs, the step row names the password prompt macOS or Linux raises. The window's runtime row no longer says Docker Desktop.
A rootful Podman machine has two engines with separate storage: `<name>` reaches the rootless one and `<name>-root` the rootful one. Releases up to 0.2.0 reached every machine by its bare name, so an OpenBot that adopted a rootful machine, or whose own machine was switched to rootful since, has its database in the rootless storage. Addressing a rootful machine by `<name>-root` opened that OpenBot on an empty database. Now a rootful machine keeps its bare connection when OpenBot's `postgres-data` volume is there and not in the rootful storage, both for a running machine OpenBot adopts and for its own. Local Intelligence still runs on the machine's rootful engine, through the API the machine forwards, when OpenBot stays in the rootless storage of the same machine: one VM, and the only engine there that Intelligence runs on. That choice is now explicit in `CliEngine::of_with` and has a test. Following the window's `podman machine set --rootful` advice therefore no longer costs the person their OpenBot.
dustywusty
marked this pull request as draft
October 29, 2026 07:42
…Podman The server is a container now, and what it calls on this machine listens only on 127.0.0.1: the person's browser, deliberately, and a local Intelligence. Docker Desktop and the Podman machines on macOS and Windows forward host.docker.internal to the host's loopback. Rootless Podman on Linux does not: slirp4netns runs with host loopback disabled and pasta with --no-map-gw, so neither was reachable at any address. - Under rootless Podman, the Desktop writes containers.conf.d/90-openbot-host.conf: allow_host_loopback for slirp4netns and --map-host-loopback 10.0.2.2 for pasta, both appended so the person's own options are kept. A change restarts the API service this session owns. - The first Start after a change brings OpenBot's stack down once, since the rootless network namespace keeps its old options while any container holds it. - OPENBOT_HOST_GATEWAY=10.0.2.2 points OpenBot's own services' host.docker.internal there. The person's other containers keep their host.containers.internal. Measured on Ubuntu 24.04 (Podman 4.9.3, slirp4netns) and Fedora 44 (Podman 5.8.9, pasta), through Docker Compose and the Podman API service: a loopback-only listener was unreachable without this and reached with it, with outbound traffic unaffected.
The reset was a flag in memory, taken before the teardown. A Start whose teardown failed had already cleared it, so "Try Start again" skipped the teardown and left the old rootless network namespace in place; quitting between writing the drop-in and starting forgot it the same way. It is now a file beside Podman's configuration, written before the drop-in changes and removed by a Start only once stack::down succeeded.
7 tasks done
The first screen of a build with local Intelligence had one button, Install, which sets up OpenBot and Intelligence on this computer. Hosted Intelligence was offered only after local had failed, so somebody who already knew local was not for them (Docker Desktop not allowed, no room for the cluster, OrbStack on a Mac) had to fail at it first. A quiet second button, "Use CopilotKit's hosted service instead", the words the failure card already uses, starts the hosted setup a build without local Intelligence starts: the Bot first, and the CLI is never asked. Install stays the default.
Setup followed the system's appearance, and OpenBot keeps its own (Settings). With the Mac in dark and OpenBot set to light, setup showed dark and then OpenBot light, as if they were two apps. OpenBot's page already reports its light or dark to the desktop (`APP_THEME_PAGE`, through `desktop_page::appearance`). That report is now kept in the app's configuration folder (`appearance.json`), and setup opens in it. Until OpenBot has reported one, or when the record cannot be read, setup follows the system as before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
OpenBot Desktop sets up CopilotKit Intelligence on this computer next to OpenBot, from the app alone, and the setup window and OpenBot itself take on the CopilotKit visual system. 65 commits; squash on merge.
Local Intelligence, from the app alone
OPENBOT_BUNDLED_CLI=1) behind a bounded native boundary. Every call is checksum-pinned, enumerated and cancellable, and only failure categories reach the window.Setup window
Desktop chrome
preferredImageVisibility).openbot-desktopplugin granted only to the owned app's origin. The Status window is the fallback.App (
app/)lib/desktop/queries.tsandmutations.ts), shown only inside Desktop.Fixes along the way
Depends on
app/: the desktop downloads v0.2.0, which has none of the app-side changes, and degrades gracefully without them.Not in this PR
dustywusty/local-eval-windows, untested).Tests
worker_exhaustion_does_not_adopt_healthy_survivorswas skipped locally only because a running OpenBot held port 3010; it passes with the port free.tscand build clean.bun test: 1570 pass and 53 fail. The same 53 fail onmain(compared test by test); none are new.