Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,13 @@ Public repo: everything here is visible to anyone.
a laptop.

## Layout
- `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects.
- `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects.
- `src/logic.js`: the allowlist (pure, tested); re-exports the state from `public/lib/state.js`.
- `src/access.js`: checks the Cloudflare Access token on `/ingest` and `/github/queue` (pure, tested).
- `src/github.js`: the GitHub Actions webhook check, the fields it keeps, and the pull queue (pure, tested).
- `src/usage.js`: what `/ingest/usage` accepts and how `/usage/data` sums it per Halifax day (pure, tested);
the `Usage` Durable Object (SQLite, migration `v2`) is in `src/worker.js`. Private: never route it to the Hub.
- `src/usage.js`: what `/ingest/usage` and `/ingest/limits` accept (`cleanUsage`, `cleanLimits`) and how
`/usage/data` sums usage per Halifax day (pure, tested); the `Usage` Durable Object (SQLite, migration `v2`;
`seat_limits` is created in code) is in `src/worker.js`. Private: never route it to the Hub.
- `src/watch.js`: when to post "updates stopped" and "updates resumed" to Buzz (pure, tested).
- `src/buzz.js`: signs and posts a Buzz message as the Worker's own identity (tested).
- `src/evidence.js`: the v4 showcase's publisher (setkernel-live#52), a public pull request's evidence and
Expand Down
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,11 +228,23 @@ Hours older than 400 days are deleted on each push.
cloud, codex; the counts are whole numbers from 0. One bad bucket makes the whole push 400.
A bucket pushed again replaces the stored one, so a pusher can resend an hour's running totals.
The answer is `{ stored, days }`: what is now stored for the hours pushed, per Halifax day and seat.
- `POST /ingest/limits` (setkernel-live#88) takes each seat's newest 5-hour and weekly limit readings,
behind the same Access app and `INGEST_AUD` check: `{ "vega": { "five_hour": { "utilization": 0.84,
"resets_at": "…", "observed_at": "…" }, "seven_day": { … }, "event": { "status": "rejected",
"rate_limit_type": "five_hour", "resets_at": "…", "received_at": "…" } } }`. The seats are vega, rigel,
lyra, orion and polaris (Codex has no Claude limits); `utilization` is a fraction from 0; times are
ISO 8601 with a zone (`resets_at` may be null: unknown); `status` is allowed, allowed_warning or
rejected; any part may be null or left out, and changes nothing then. Other fields are dropped; a bad
seat, value or time makes the whole push 400. The store (`seat_limits`, one row per seat and part)
keeps a part only if it is as new as or newer than the one it has, and answers `{ limits }`: every
seat's as now stored.
- `GET /usage/data?month=YYYY-MM` needs a token for its own Access app, checked against
`USAGE_AUD` (the "live usage" Access app, setkernel-cloudflare#119; its tag is in `wrangler.jsonc`).
With no login Access answers 302 to its login page; if `USAGE_AUD` were ever empty the Worker would
answer 503 to everyone. It returns `{ month, days: { "2026-10-14": { seat: sums } }, totals: { seat: sums } }`
with `Cache-Control: no-store`. Days are Halifax days (`homeDay`), so an hour's usage counts on
with `Cache-Control: no-store`, and for the current Halifax month, once any reading is stored, `limits`:
`{ seat: { five_hour, seven_day, event } }`, each part null until it has a reading. A past month has
none: limits are read live. Days are Halifax days (`homeDay`), so an hour's usage counts on
the day it was in Halifax: 03:00 UTC is 23:00 the day before in winter.

The read is open behind Access: the app for `live.setkernel.com/usage` exists (setkernel-cloudflare#119).
Expand All @@ -246,8 +258,8 @@ before the Worker sees the request). `/usage/data` names the newest hour stored,
are input + output + cache writes, and a day or seat with no data is a dash, not zero. The seats are
the ones the data names. `public/usage/limits.js` reads each seat's 5-hour and weekly windows from a
`limits` object in the answer, with the wall's rules (alerts at 80% and 70%, a limit hit at 100% or on
a rejected request, a reading from before its window reset is stale). `/usage/data` doesn't send
`limits` yet, so the page says it can't check them and shows no figure. `public/usage/index.html`,
a rejected request, a reading from before its window reset is stale). Until a reading is stored,
`/usage/data` sends no `limits`, so the page says it can't check them and shows no figure. `public/usage/index.html`,
`usage.css` (its own palette and the fonts in `public/usage/fonts/`) and `page.js` draw it
(setkernel-live#78, v5 in #86): `?month=YYYY-MM` picks the month, the page also reads the two months
before it for the 30-day trend, the newest hour is the header's, and a 403, 503 or expired login shows a
Expand Down
2 changes: 1 addition & 1 deletion docs/design/usage/shoot.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ const STATES = [
["normal", "", "2026-10-28T17", 15, NORMAL],
["limit", "", "2026-10-28T17", 15, LIMIT],
["stale", "", "2026-10-28T13", 11, STALE],
["unread", "", "2026-10-28T17", 15, null], // /usage/data without limits: what the Worker answers today
["unread", "", "2026-10-28T17", 15, null], // /usage/data without limits: what the Worker answers until a reading is stored
["past", "?month=2026-09", "2026-10-28T17", 15, NORMAL],
["loading", "", null, 0, "hang"],
["refused", "", null, 0, 403],
Expand Down
Binary file added docs/design/usage/usage-worker-1440-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/usage/usage-worker-375-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 5 additions & 3 deletions public/usage/limits.js
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
// live.setkernel.com/usage (setkernel-live#86): each seat's 5-hour and weekly limit windows, as the
// page says them. Pure and tested (test/usage-page.test.mjs). GET /usage/data is to carry them as
// `limits`, from the seat view the wall reads (setkernel/hq observability/wall/limits.py):
// page says them. Pure and tested (test/usage-page.test.mjs). GET /usage/data carries them as `limits`
// (setkernel-live#88, src/usage.js cleanLimits), from the seat view the wall reads (setkernel/hq
// observability/wall/limits.py):
// { <seat>: { five_hour: { utilization: 0.84, resets_at, observed_at } | null, seven_day: { ... } | null,
// event: { status: "rejected", rate_limit_type: "five_hour", resets_at, received_at } | null } }
// with times as ISO 8601 strings. Until it does, the page says it can't check limits and shows no figure.
// with times as ISO 8601 strings. Until a reading is stored, it sends none: the page says it can't check
// limits and shows no figure.
// The rules are the wall's: a window with no reset time resets a window's length after it was read.
import { homeDay } from "../lib/state.js";
import { seatName } from "./table.js";
Expand Down
52 changes: 52 additions & 0 deletions src/usage.js
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,58 @@ export function cleanUsage(body) {
return out;
}

// Each seat's 5-hour and weekly limit windows (setkernel-live#88): the newest reading per seat, pushed
// to POST /ingest/limits and read back as `limits` from GET /usage/data, in the shape
// public/usage/limits.js reads. Codex has no Claude limits, and "other" is no seat, so neither has any.
export const LIMIT_SEATS = SEATS.filter((s) => s !== "codex" && s !== "other");
export const WINDOWS = ["five_hour", "seven_day"];
export const STATUSES = ["allowed", "allowed_warning", "rejected"];
const ISO = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.\d{1,9})?)?(?:Z|[+-](\d{2}):(\d{2}))$/;

// An ISO 8601 time with a zone, "2026-11-04T12:00:00Z" or "...12:00:00.123456+00:00" -> the same
// instant as toISOString() gives it, or null unless it is a real date and time.
export function isoTime(t) {
const m = typeof t === "string" && t.length <= 40 ? ISO.exec(t) : null;
if (!m) return null;
const [y, mo, d, h, mi, s, oh, om] = m.slice(1).map((x) => Number(x ?? 0));
if (mo < 1 || mo > 12 || d < 1 || d > new Date(Date.UTC(y, mo, 0)).getUTCDate() || h > 23 || mi > 59 || s > 59 || oh > 23 || om > 59) return null;
const ms = Date.parse(t);
return Number.isFinite(ms) ? new Date(ms).toISOString() : null;
}

const plain = (o) => !!o && typeof o === "object" && !Array.isArray(o);
// A reset time may be unknown: null or absent is null, a bad one is undefined.
const resetsAt = (t) => (t == null ? null : (isoTime(t) ?? undefined));
// A window or an event with only the fields the page reads; null if null or absent; undefined if bad.
function cleanWindow(w) {
if (w == null) return null;
if (!plain(w) || typeof w.utilization !== "number" || !Number.isFinite(w.utilization) || w.utilization < 0) return undefined;
const resets_at = resetsAt(w.resets_at), observed_at = isoTime(w.observed_at);
return resets_at === undefined || !observed_at ? undefined : { utilization: w.utilization, resets_at, observed_at };
}
function cleanEvent(e) {
if (e == null) return null;
if (!plain(e) || !STATUSES.includes(e.status) || !WINDOWS.includes(e.rate_limit_type)) return undefined;
const resets_at = resetsAt(e.resets_at), received_at = isoTime(e.received_at);
return resets_at === undefined || !received_at ? undefined : { status: e.status, rate_limit_type: e.rate_limit_type, resets_at, received_at };
}

// { <seat>: { five_hour, seven_day, event } } -> the same with only LIMIT_SEATS, each window's
// utilization (a fraction, 0.84), resets_at and observed_at, and the event's status, rate_limit_type,
// resets_at and received_at, times as toISOString() gives them and a missing part null; or null if any
// seat, window or event is not allowed. Fields it doesn't name are dropped.
export function cleanLimits(body) {
if (!plain(body)) return null;
const out = {};
for (const [seat, s] of Object.entries(body)) {
if (!LIMIT_SEATS.includes(seat) || !plain(s)) return null;
const five_hour = cleanWindow(s.five_hour), seven_day = cleanWindow(s.seven_day), event = cleanEvent(s.event);
if (five_hour === undefined || seven_day === undefined || event === undefined) return null;
out[seat] = { five_hour, seven_day, event };
}
return out;
}

// "2026-10" -> the UTC hours to read, [from, to), a day wider on each side than the month, since a
// Halifax day starts at 03:00 or 04:00 UTC; or null for anything else. sumByDay keeps the month.
export function monthRange(month) {
Expand Down
67 changes: 57 additions & 10 deletions src/worker.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,20 @@
// POST /github/webhook GitHub Actions, pull request, review, issue and comment events from the org webhook (public; HMAC-signed)
// GET /github/queue those events, cut down, for sk-obs01 to pull (its own Access token, checked again)
// POST /ingest/usage hourly token usage per seat (the ingest's Access token, checked again; setkernel-live#66)
// GET /usage/data that usage summed per Halifax day for ?month=YYYY-MM (its own Access app; 503 until it exists)
// POST /ingest/limits each seat's newest 5-hour and weekly limit readings (the same token as /ingest/usage; setkernel-live#88)
// GET /usage/data that usage summed per Halifax day for ?month=YYYY-MM, and on the current month the limits (its own Access app)
// GET /usage/ the page that reads it (public/usage/), behind the same Access app and token check
// One Durable Object ("live") holds the picture, a capped 24-hour replay buffer and the GitHub queue,
// and fans batches out to every open WebSocket.
// A Cron Trigger asks it every 5 minutes whether updates have stopped, and it tells Buzz #infra.
// A second, private Durable Object ("usage") holds the usage, in SQLite; the Hub never sees it.
// A second, private Durable Object ("usage") holds the usage and the limits, in SQLite; the Hub never sees them.
import { DurableObject } from "cloudflare:workers";
import { applyBatch, cleanBatch, emptyReplay, emptyState, homeDay, recordReplay, replaySince, view } from "./logic.js";
import { jwksFetcher, verifyAccessJwt } from "./access.js";
import { EVENTS, cleanDelivery, cleanWebhook, emptyQueue, fetchHub, parseCursor, pushEvent, readCapped, readQueue, verifySignature } from "./github.js";
import { delivered, emptyWatch, tick } from "./watch.js";
import { postMessage } from "./buzz.js";
import { COUNTS, cleanUsage, keepFrom, monthRange, sumByDay } from "./usage.js";
import { COUNTS, cleanLimits, cleanUsage, keepFrom, monthRange, sumByDay } from "./usage.js";

const getKeys = jwksFetcher();
const MAX_BODY = 256 * 1024;
Expand Down Expand Up @@ -64,6 +65,22 @@ export default {
if (!buckets) return new Response("Not usage", { status: 400 });
return Response.json(await usage(env).push(buckets, Date.now()), { headers: { "Cache-Control": "no-store" } });
}
if (pathname === "/ingest/limits") {
if (request.method !== "POST") return new Response("Method not allowed", { status: 405 });
if (!env.INGEST_AUD) return new Response("Ingest is not configured", { status: 503 });
const ok = await verifyAccessJwt(request.headers.get("Cf-Access-Jwt-Assertion"), { aud: env.INGEST_AUD, getKeys });
if (!ok) return new Response("Forbidden", { status: 403 });
const body = await request.text();
if (body.length > MAX_BODY) return new Response("Too large", { status: 413 });
let limits;
try {
limits = cleanLimits(JSON.parse(body));
} catch {
return new Response("Bad JSON", { status: 400 });
}
if (!limits) return new Response("Not limits", { status: 400 });
return Response.json(await usage(env).pushLimits(limits), { headers: { "Cache-Control": "no-store" } });
}
if (pathname === "/usage/data") {
if (request.method !== "GET") return new Response("Method not allowed", { status: 405 });
if (!env.USAGE_AUD) return new Response("Usage is not configured", { status: 503 });
Expand All @@ -72,7 +89,7 @@ export default {
const month = new URL(request.url).searchParams.get("month");
if (!monthRange(month)) return new Response("Bad month, want YYYY-MM", { status: 400 });
// the newest hour stored goes in a header, so the body keeps the shape #69 shipped
const { newest, ...body } = await usage(env).month(month);
const { newest, ...body } = await usage(env).month(month, Date.now());
return Response.json(body, { headers: { "Cache-Control": "no-store", ...(newest && { "Usage-Newest-Hour": newest }) } });
}
if (pathname === "/usage" || pathname.startsWith("/usage/")) {
Expand Down Expand Up @@ -131,15 +148,42 @@ async function githubWebhook(request, env, hub) {

const usage = (env) => env.USAGE.get(env.USAGE.idFromName("usage"));

// Hourly usage per seat, model and surface (setkernel-live#66). Only the Worker calls it, over RPC,
// with buckets cleanUsage has already checked. A bucket pushed again replaces the stored one, so a
// pusher can resend an hour's running totals.
// Hourly usage per seat, model and surface (setkernel-live#66), and each seat's newest limit readings
// (setkernel-live#88). Only the Worker calls it, over RPC, with what cleanUsage or cleanLimits has
// already checked. A bucket pushed again replaces the stored one, so a pusher can resend an hour's
// running totals; a limit reading replaces the stored one only if it is as new or newer.
export class Usage extends DurableObject {
constructor(ctx, env) {
super(ctx, env);
this.sql = ctx.storage.sql;
this.sql.exec(`CREATE TABLE IF NOT EXISTS buckets (hour TEXT NOT NULL, seat TEXT NOT NULL, model TEXT NOT NULL,
surface TEXT NOT NULL, ${COUNTS.map((k) => `${k} INTEGER NOT NULL`).join(", ")}, PRIMARY KEY (hour, seat, model, surface))`);
// One row per seat and part (five_hour, seven_day, event): `at` is its observed_at or received_at,
// `value` the cleaned part as JSON. A seat has at most three rows, so nothing needs pruning.
this.sql.exec(`CREATE TABLE IF NOT EXISTS seat_limits (seat TEXT NOT NULL, part TEXT NOT NULL, at TEXT NOT NULL,
value TEXT NOT NULL, PRIMARY KEY (seat, part))`);
}

// { seat: { five_hour, seven_day, event } } as stored, each part null until a reading of it arrives.
#limits() {
const out = {};
for (const { seat, part, value } of this.sql.exec("SELECT seat, part, value FROM seat_limits ORDER BY seat, part")) {
(out[seat] ??= { five_hour: null, seven_day: null, event: null })[part] = JSON.parse(value);
}
return out;
}

// Keeps each part pushed unless the stored one is newer (times are toISOString's, so they sort as
// text), and answers with every seat's limits as now stored. A part pushed as null changes nothing.
pushLimits(limits) {
const upsert = `INSERT INTO seat_limits (seat, part, at, value) VALUES (?, ?, ?, ?)
ON CONFLICT (seat, part) DO UPDATE SET at = excluded.at, value = excluded.value WHERE excluded.at >= seat_limits.at`;
this.ctx.storage.transactionSync(() => {
for (const [seat, parts] of Object.entries(limits)) {
for (const [part, v] of Object.entries(parts)) if (v) this.sql.exec(upsert, seat, part, v.observed_at ?? v.received_at, JSON.stringify(v));
}
});
return { limits: this.#limits() };
}

// Each hour summed per seat over its models and surfaces, oldest first.
Expand All @@ -162,11 +206,14 @@ export class Usage extends DurableObject {
}

// The month's Halifax days, each seat's sums per day, each seat's totals for the month, and the
// newest hour stored in any month (null if none).
month(month) {
// newest hour stored in any month (null if none). The current Halifax month at `now` also carries
// the limits, once any reading is stored: they are read live, so a past month has none, and while
// there are none the page says it can't check them.
month(month, now) {
const { from, to } = monthRange(month);
const newest = this.sql.exec("SELECT MAX(hour) AS h FROM buckets").one().h;
return { month, ...sumByDay(this.#rows("hour >= ? AND hour < ?", from, to), month), newest };
const limits = homeDay(now).startsWith(`${month}-`) ? this.#limits() : {};
return { month, ...sumByDay(this.#rows("hour >= ? AND hour < ?", from, to), month), ...(Object.keys(limits).length && { limits }), newest };
}
}

Expand Down
Loading
Loading