diff --git a/AGENTS.md b/AGENTS.md index 77bccdc..c25648f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,12 +34,13 @@ Public repo: everything here is visible to anyone. a laptop. ## Layout -- `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects. +- `src/worker.js`: routes (`/ws`, `/api/state`, `/api/replay`, `/ingest`, `/ingest/usage`, `/ingest/limits`, `/usage/data`, `/usage/`, `/github/webhook`, `/github/queue`) and the `Hub` and `Usage` Durable Objects. - `src/logic.js`: the allowlist (pure, tested); re-exports the state from `public/lib/state.js`. - `src/access.js`: checks the Cloudflare Access token on `/ingest` and `/github/queue` (pure, tested). - `src/github.js`: the GitHub Actions webhook check, the fields it keeps, and the pull queue (pure, tested). -- `src/usage.js`: what `/ingest/usage` accepts and how `/usage/data` sums it per Halifax day (pure, tested); - the `Usage` Durable Object (SQLite, migration `v2`) is in `src/worker.js`. Private: never route it to the Hub. +- `src/usage.js`: what `/ingest/usage` and `/ingest/limits` accept (`cleanUsage`, `cleanLimits`) and how + `/usage/data` sums usage per Halifax day (pure, tested); the `Usage` Durable Object (SQLite, migration `v2`; + `seat_limits` is created in code) is in `src/worker.js`. Private: never route it to the Hub. - `src/watch.js`: when to post "updates stopped" and "updates resumed" to Buzz (pure, tested). - `src/buzz.js`: signs and posts a Buzz message as the Worker's own identity (tested). - `src/evidence.js`: the v4 showcase's publisher (setkernel-live#52), a public pull request's evidence and diff --git a/README.md b/README.md index c47fb7f..d0501b4 100644 --- a/README.md +++ b/README.md @@ -228,11 +228,23 @@ Hours older than 400 days are deleted on each push. cloud, codex; the counts are whole numbers from 0. One bad bucket makes the whole push 400. A bucket pushed again replaces the stored one, so a pusher can resend an hour's running totals. The answer is `{ stored, days }`: what is now stored for the hours pushed, per Halifax day and seat. +- `POST /ingest/limits` (setkernel-live#88) takes each seat's newest 5-hour and weekly limit readings, + behind the same Access app and `INGEST_AUD` check: `{ "vega": { "five_hour": { "utilization": 0.84, + "resets_at": "…", "observed_at": "…" }, "seven_day": { … }, "event": { "status": "rejected", + "rate_limit_type": "five_hour", "resets_at": "…", "received_at": "…" } } }`. The seats are vega, rigel, + lyra, orion and polaris (Codex has no Claude limits); `utilization` is a fraction from 0; times are + ISO 8601 with a zone (`resets_at` may be null: unknown); `status` is allowed, allowed_warning or + rejected; any part may be null or left out, and changes nothing then. Other fields are dropped; a bad + seat, value or time makes the whole push 400. The store (`seat_limits`, one row per seat and part) + keeps a part only if it is as new as or newer than the one it has, and answers `{ limits }`: every + seat's as now stored. - `GET /usage/data?month=YYYY-MM` needs a token for its own Access app, checked against `USAGE_AUD` (the "live usage" Access app, setkernel-cloudflare#119; its tag is in `wrangler.jsonc`). With no login Access answers 302 to its login page; if `USAGE_AUD` were ever empty the Worker would answer 503 to everyone. It returns `{ month, days: { "2026-10-14": { seat: sums } }, totals: { seat: sums } }` - with `Cache-Control: no-store`. Days are Halifax days (`homeDay`), so an hour's usage counts on + with `Cache-Control: no-store`, and for the current Halifax month, once any reading is stored, `limits`: + `{ seat: { five_hour, seven_day, event } }`, each part null until it has a reading. A past month has + none: limits are read live. Days are Halifax days (`homeDay`), so an hour's usage counts on the day it was in Halifax: 03:00 UTC is 23:00 the day before in winter. The read is open behind Access: the app for `live.setkernel.com/usage` exists (setkernel-cloudflare#119). @@ -246,8 +258,8 @@ before the Worker sees the request). `/usage/data` names the newest hour stored, are input + output + cache writes, and a day or seat with no data is a dash, not zero. The seats are the ones the data names. `public/usage/limits.js` reads each seat's 5-hour and weekly windows from a `limits` object in the answer, with the wall's rules (alerts at 80% and 70%, a limit hit at 100% or on -a rejected request, a reading from before its window reset is stale). `/usage/data` doesn't send -`limits` yet, so the page says it can't check them and shows no figure. `public/usage/index.html`, +a rejected request, a reading from before its window reset is stale). Until a reading is stored, +`/usage/data` sends no `limits`, so the page says it can't check them and shows no figure. `public/usage/index.html`, `usage.css` (its own palette and the fonts in `public/usage/fonts/`) and `page.js` draw it (setkernel-live#78, v5 in #86): `?month=YYYY-MM` picks the month, the page also reads the two months before it for the 30-day trend, the newest hour is the header's, and a 403, 503 or expired login shows a diff --git a/docs/design/usage/shoot.mjs b/docs/design/usage/shoot.mjs index c2eed34..3768f5d 100644 --- a/docs/design/usage/shoot.mjs +++ b/docs/design/usage/shoot.mjs @@ -88,7 +88,7 @@ const STATES = [ ["normal", "", "2026-10-28T17", 15, NORMAL], ["limit", "", "2026-10-28T17", 15, LIMIT], ["stale", "", "2026-10-28T13", 11, STALE], - ["unread", "", "2026-10-28T17", 15, null], // /usage/data without limits: what the Worker answers today + ["unread", "", "2026-10-28T17", 15, null], // /usage/data without limits: what the Worker answers until a reading is stored ["past", "?month=2026-09", "2026-10-28T17", 15, NORMAL], ["loading", "", null, 0, "hang"], ["refused", "", null, 0, 403], diff --git a/docs/design/usage/usage-worker-1440-light.png b/docs/design/usage/usage-worker-1440-light.png new file mode 100644 index 0000000..f7a7989 Binary files /dev/null and b/docs/design/usage/usage-worker-1440-light.png differ diff --git a/docs/design/usage/usage-worker-375-dark.png b/docs/design/usage/usage-worker-375-dark.png new file mode 100644 index 0000000..75d9173 Binary files /dev/null and b/docs/design/usage/usage-worker-375-dark.png differ diff --git a/public/usage/limits.js b/public/usage/limits.js index 67faa34..b40b575 100644 --- a/public/usage/limits.js +++ b/public/usage/limits.js @@ -1,9 +1,11 @@ // live.setkernel.com/usage (setkernel-live#86): each seat's 5-hour and weekly limit windows, as the -// page says them. Pure and tested (test/usage-page.test.mjs). GET /usage/data is to carry them as -// `limits`, from the seat view the wall reads (setkernel/hq observability/wall/limits.py): +// page says them. Pure and tested (test/usage-page.test.mjs). GET /usage/data carries them as `limits` +// (setkernel-live#88, src/usage.js cleanLimits), from the seat view the wall reads (setkernel/hq +// observability/wall/limits.py): // { : { five_hour: { utilization: 0.84, resets_at, observed_at } | null, seven_day: { ... } | null, // event: { status: "rejected", rate_limit_type: "five_hour", resets_at, received_at } | null } } -// with times as ISO 8601 strings. Until it does, the page says it can't check limits and shows no figure. +// with times as ISO 8601 strings. Until a reading is stored, it sends none: the page says it can't check +// limits and shows no figure. // The rules are the wall's: a window with no reset time resets a window's length after it was read. import { homeDay } from "../lib/state.js"; import { seatName } from "./table.js"; diff --git a/src/usage.js b/src/usage.js index 41e2109..674606c 100644 --- a/src/usage.js +++ b/src/usage.js @@ -36,6 +36,58 @@ export function cleanUsage(body) { return out; } +// Each seat's 5-hour and weekly limit windows (setkernel-live#88): the newest reading per seat, pushed +// to POST /ingest/limits and read back as `limits` from GET /usage/data, in the shape +// public/usage/limits.js reads. Codex has no Claude limits, and "other" is no seat, so neither has any. +export const LIMIT_SEATS = SEATS.filter((s) => s !== "codex" && s !== "other"); +export const WINDOWS = ["five_hour", "seven_day"]; +export const STATUSES = ["allowed", "allowed_warning", "rejected"]; +const ISO = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.\d{1,9})?)?(?:Z|[+-](\d{2}):(\d{2}))$/; + +// An ISO 8601 time with a zone, "2026-11-04T12:00:00Z" or "...12:00:00.123456+00:00" -> the same +// instant as toISOString() gives it, or null unless it is a real date and time. +export function isoTime(t) { + const m = typeof t === "string" && t.length <= 40 ? ISO.exec(t) : null; + if (!m) return null; + const [y, mo, d, h, mi, s, oh, om] = m.slice(1).map((x) => Number(x ?? 0)); + if (mo < 1 || mo > 12 || d < 1 || d > new Date(Date.UTC(y, mo, 0)).getUTCDate() || h > 23 || mi > 59 || s > 59 || oh > 23 || om > 59) return null; + const ms = Date.parse(t); + return Number.isFinite(ms) ? new Date(ms).toISOString() : null; +} + +const plain = (o) => !!o && typeof o === "object" && !Array.isArray(o); +// A reset time may be unknown: null or absent is null, a bad one is undefined. +const resetsAt = (t) => (t == null ? null : (isoTime(t) ?? undefined)); +// A window or an event with only the fields the page reads; null if null or absent; undefined if bad. +function cleanWindow(w) { + if (w == null) return null; + if (!plain(w) || typeof w.utilization !== "number" || !Number.isFinite(w.utilization) || w.utilization < 0) return undefined; + const resets_at = resetsAt(w.resets_at), observed_at = isoTime(w.observed_at); + return resets_at === undefined || !observed_at ? undefined : { utilization: w.utilization, resets_at, observed_at }; +} +function cleanEvent(e) { + if (e == null) return null; + if (!plain(e) || !STATUSES.includes(e.status) || !WINDOWS.includes(e.rate_limit_type)) return undefined; + const resets_at = resetsAt(e.resets_at), received_at = isoTime(e.received_at); + return resets_at === undefined || !received_at ? undefined : { status: e.status, rate_limit_type: e.rate_limit_type, resets_at, received_at }; +} + +// { : { five_hour, seven_day, event } } -> the same with only LIMIT_SEATS, each window's +// utilization (a fraction, 0.84), resets_at and observed_at, and the event's status, rate_limit_type, +// resets_at and received_at, times as toISOString() gives them and a missing part null; or null if any +// seat, window or event is not allowed. Fields it doesn't name are dropped. +export function cleanLimits(body) { + if (!plain(body)) return null; + const out = {}; + for (const [seat, s] of Object.entries(body)) { + if (!LIMIT_SEATS.includes(seat) || !plain(s)) return null; + const five_hour = cleanWindow(s.five_hour), seven_day = cleanWindow(s.seven_day), event = cleanEvent(s.event); + if (five_hour === undefined || seven_day === undefined || event === undefined) return null; + out[seat] = { five_hour, seven_day, event }; + } + return out; +} + // "2026-10" -> the UTC hours to read, [from, to), a day wider on each side than the month, since a // Halifax day starts at 03:00 or 04:00 UTC; or null for anything else. sumByDay keeps the month. export function monthRange(month) { diff --git a/src/worker.js b/src/worker.js index 3dd67b0..3bd0b16 100644 --- a/src/worker.js +++ b/src/worker.js @@ -7,19 +7,20 @@ // POST /github/webhook GitHub Actions, pull request, review, issue and comment events from the org webhook (public; HMAC-signed) // GET /github/queue those events, cut down, for sk-obs01 to pull (its own Access token, checked again) // POST /ingest/usage hourly token usage per seat (the ingest's Access token, checked again; setkernel-live#66) -// GET /usage/data that usage summed per Halifax day for ?month=YYYY-MM (its own Access app; 503 until it exists) +// POST /ingest/limits each seat's newest 5-hour and weekly limit readings (the same token as /ingest/usage; setkernel-live#88) +// GET /usage/data that usage summed per Halifax day for ?month=YYYY-MM, and on the current month the limits (its own Access app) // GET /usage/ the page that reads it (public/usage/), behind the same Access app and token check // One Durable Object ("live") holds the picture, a capped 24-hour replay buffer and the GitHub queue, // and fans batches out to every open WebSocket. // A Cron Trigger asks it every 5 minutes whether updates have stopped, and it tells Buzz #infra. -// A second, private Durable Object ("usage") holds the usage, in SQLite; the Hub never sees it. +// A second, private Durable Object ("usage") holds the usage and the limits, in SQLite; the Hub never sees them. import { DurableObject } from "cloudflare:workers"; import { applyBatch, cleanBatch, emptyReplay, emptyState, homeDay, recordReplay, replaySince, view } from "./logic.js"; import { jwksFetcher, verifyAccessJwt } from "./access.js"; import { EVENTS, cleanDelivery, cleanWebhook, emptyQueue, fetchHub, parseCursor, pushEvent, readCapped, readQueue, verifySignature } from "./github.js"; import { delivered, emptyWatch, tick } from "./watch.js"; import { postMessage } from "./buzz.js"; -import { COUNTS, cleanUsage, keepFrom, monthRange, sumByDay } from "./usage.js"; +import { COUNTS, cleanLimits, cleanUsage, keepFrom, monthRange, sumByDay } from "./usage.js"; const getKeys = jwksFetcher(); const MAX_BODY = 256 * 1024; @@ -64,6 +65,22 @@ export default { if (!buckets) return new Response("Not usage", { status: 400 }); return Response.json(await usage(env).push(buckets, Date.now()), { headers: { "Cache-Control": "no-store" } }); } + if (pathname === "/ingest/limits") { + if (request.method !== "POST") return new Response("Method not allowed", { status: 405 }); + if (!env.INGEST_AUD) return new Response("Ingest is not configured", { status: 503 }); + const ok = await verifyAccessJwt(request.headers.get("Cf-Access-Jwt-Assertion"), { aud: env.INGEST_AUD, getKeys }); + if (!ok) return new Response("Forbidden", { status: 403 }); + const body = await request.text(); + if (body.length > MAX_BODY) return new Response("Too large", { status: 413 }); + let limits; + try { + limits = cleanLimits(JSON.parse(body)); + } catch { + return new Response("Bad JSON", { status: 400 }); + } + if (!limits) return new Response("Not limits", { status: 400 }); + return Response.json(await usage(env).pushLimits(limits), { headers: { "Cache-Control": "no-store" } }); + } if (pathname === "/usage/data") { if (request.method !== "GET") return new Response("Method not allowed", { status: 405 }); if (!env.USAGE_AUD) return new Response("Usage is not configured", { status: 503 }); @@ -72,7 +89,7 @@ export default { const month = new URL(request.url).searchParams.get("month"); if (!monthRange(month)) return new Response("Bad month, want YYYY-MM", { status: 400 }); // the newest hour stored goes in a header, so the body keeps the shape #69 shipped - const { newest, ...body } = await usage(env).month(month); + const { newest, ...body } = await usage(env).month(month, Date.now()); return Response.json(body, { headers: { "Cache-Control": "no-store", ...(newest && { "Usage-Newest-Hour": newest }) } }); } if (pathname === "/usage" || pathname.startsWith("/usage/")) { @@ -131,15 +148,42 @@ async function githubWebhook(request, env, hub) { const usage = (env) => env.USAGE.get(env.USAGE.idFromName("usage")); -// Hourly usage per seat, model and surface (setkernel-live#66). Only the Worker calls it, over RPC, -// with buckets cleanUsage has already checked. A bucket pushed again replaces the stored one, so a -// pusher can resend an hour's running totals. +// Hourly usage per seat, model and surface (setkernel-live#66), and each seat's newest limit readings +// (setkernel-live#88). Only the Worker calls it, over RPC, with what cleanUsage or cleanLimits has +// already checked. A bucket pushed again replaces the stored one, so a pusher can resend an hour's +// running totals; a limit reading replaces the stored one only if it is as new or newer. export class Usage extends DurableObject { constructor(ctx, env) { super(ctx, env); this.sql = ctx.storage.sql; this.sql.exec(`CREATE TABLE IF NOT EXISTS buckets (hour TEXT NOT NULL, seat TEXT NOT NULL, model TEXT NOT NULL, surface TEXT NOT NULL, ${COUNTS.map((k) => `${k} INTEGER NOT NULL`).join(", ")}, PRIMARY KEY (hour, seat, model, surface))`); + // One row per seat and part (five_hour, seven_day, event): `at` is its observed_at or received_at, + // `value` the cleaned part as JSON. A seat has at most three rows, so nothing needs pruning. + this.sql.exec(`CREATE TABLE IF NOT EXISTS seat_limits (seat TEXT NOT NULL, part TEXT NOT NULL, at TEXT NOT NULL, + value TEXT NOT NULL, PRIMARY KEY (seat, part))`); + } + + // { seat: { five_hour, seven_day, event } } as stored, each part null until a reading of it arrives. + #limits() { + const out = {}; + for (const { seat, part, value } of this.sql.exec("SELECT seat, part, value FROM seat_limits ORDER BY seat, part")) { + (out[seat] ??= { five_hour: null, seven_day: null, event: null })[part] = JSON.parse(value); + } + return out; + } + + // Keeps each part pushed unless the stored one is newer (times are toISOString's, so they sort as + // text), and answers with every seat's limits as now stored. A part pushed as null changes nothing. + pushLimits(limits) { + const upsert = `INSERT INTO seat_limits (seat, part, at, value) VALUES (?, ?, ?, ?) + ON CONFLICT (seat, part) DO UPDATE SET at = excluded.at, value = excluded.value WHERE excluded.at >= seat_limits.at`; + this.ctx.storage.transactionSync(() => { + for (const [seat, parts] of Object.entries(limits)) { + for (const [part, v] of Object.entries(parts)) if (v) this.sql.exec(upsert, seat, part, v.observed_at ?? v.received_at, JSON.stringify(v)); + } + }); + return { limits: this.#limits() }; } // Each hour summed per seat over its models and surfaces, oldest first. @@ -162,11 +206,14 @@ export class Usage extends DurableObject { } // The month's Halifax days, each seat's sums per day, each seat's totals for the month, and the - // newest hour stored in any month (null if none). - month(month) { + // newest hour stored in any month (null if none). The current Halifax month at `now` also carries + // the limits, once any reading is stored: they are read live, so a past month has none, and while + // there are none the page says it can't check them. + month(month, now) { const { from, to } = monthRange(month); const newest = this.sql.exec("SELECT MAX(hour) AS h FROM buckets").one().h; - return { month, ...sumByDay(this.#rows("hour >= ? AND hour < ?", from, to), month), newest }; + const limits = homeDay(now).startsWith(`${month}-`) ? this.#limits() : {}; + return { month, ...sumByDay(this.#rows("hour >= ? AND hour < ?", from, to), month), ...(Object.keys(limits).length && { limits }), newest }; } } diff --git a/test/usage.test.mjs b/test/usage.test.mjs index 78f3b53..96d8858 100644 --- a/test/usage.test.mjs +++ b/test/usage.test.mjs @@ -2,6 +2,7 @@ import { test } from "node:test"; import assert from "node:assert/strict"; import { COUNTS, SEATS, SURFACES, cleanUsage, hourMs, keepFrom, monthRange, sumByDay } from "../src/usage.js"; +import { LIMIT_SEATS, cleanLimits, isoTime } from "../src/usage.js"; const bucket = (extra = {}) => ({ hour: "2026-10-14T15", seat: "vega", model: "claude-opus-5-5", surface: "vm", input: 10, output: 20, cache_read: 30, cache_write: 40, usd_micros: 50, requests: 1, ...extra }); const counts = (n) => Object.fromEntries(COUNTS.map((k) => [k, n])); @@ -25,6 +26,52 @@ test("cleanUsage refuses the whole push for any bad bucket", () => { for (const body of [null, [], "x", {}, { buckets: {} }, { buckets: [null] }, { buckets: ["x"] }]) assert.equal(cleanUsage(body), null, JSON.stringify(body)); }); +const win = (extra = {}) => ({ utilization: 0.84, resets_at: "2026-11-04T16:10:00.000Z", observed_at: "2026-11-04T14:55:00.000Z", ...extra }); +const ev = (extra = {}) => ({ status: "rejected", rate_limit_type: "five_hour", resets_at: "2026-11-04T16:10:00.000Z", received_at: "2026-11-04T14:47:00.000Z", ...extra }); +const seat = (extra = {}) => ({ five_hour: win(), seven_day: win({ utilization: 0.52 }), event: ev(), ...extra }); + +test("cleanLimits keeps the seats' windows and event, field for field, and drops everything else", () => { + assert.deepEqual(cleanLimits({}), {}); + assert.deepEqual(cleanLimits({ rigel: seat() }), { rigel: seat() }); + const extra = { rigel: { ...seat({ five_hour: { ...win(), name: "five_hour", login: "rigel-setkernel" }, event: { ...ev(), prompt: "secret", utilization: 1 } }), id: 7, seven_day_opus: win() } }; + assert.deepEqual(cleanLimits(extra), { rigel: seat() }); + // a part null or absent is null; a reset time may be unknown + assert.deepEqual(cleanLimits({ lyra: {} }), { lyra: { five_hour: null, seven_day: null, event: null } }); + assert.deepEqual(cleanLimits({ lyra: seat({ seven_day: null, event: ev({ resets_at: null }) }) }).lyra, { five_hour: win(), seven_day: null, event: ev({ resets_at: null }) }); + assert.deepEqual(cleanLimits({ lyra: { five_hour: { utilization: 0, observed_at: "2026-11-04T14:55:00.000Z" } } }).lyra.five_hour, { utilization: 0, resets_at: null, observed_at: "2026-11-04T14:55:00.000Z" }); + // times come back as toISOString gives them, whatever zone and precision they were sent in + assert.deepEqual(cleanLimits({ vega: { five_hour: win({ resets_at: "2026-11-04T13:10:00-03:00", observed_at: "2026-11-04T14:55:00.123456+00:00" }) } }).vega.five_hour, win({ observed_at: "2026-11-04T14:55:00.123Z" })); + for (const s of LIMIT_SEATS) assert.ok(cleanLimits({ [s]: seat() }), s); + for (const status of ["allowed", "allowed_warning"]) assert.ok(cleanLimits({ vega: seat({ event: ev({ status, rate_limit_type: "seven_day" }) }) }), status); + assert.ok(cleanLimits({ vega: seat({ five_hour: win({ utilization: 1.2 }) }) }), "over 100% is a reading too"); +}); + +test("cleanLimits refuses the whole push for any bad seat, window or event", () => { + const bad = [ + { codex: seat() }, { other: seat() }, { raghav: seat() }, { Vega: seat() }, { vega: null }, { vega: [] }, { vega: "x" }, + { vega: seat({ five_hour: win({ utilization: -0.01 }) }) }, { vega: seat({ five_hour: win({ utilization: "0.5" }) }) }, + { vega: seat({ seven_day: win({ utilization: Infinity }) }) }, { vega: seat({ seven_day: win({ utilization: NaN }) }) }, + { vega: seat({ five_hour: win({ utilization: undefined }) }) }, { vega: seat({ five_hour: [] }) }, { vega: seat({ five_hour: 0.5 }) }, + { vega: seat({ five_hour: win({ observed_at: null }) }) }, { vega: seat({ five_hour: win({ observed_at: "2026-11-04" }) }) }, + { vega: seat({ five_hour: win({ resets_at: "soon" }) }) }, { vega: seat({ five_hour: win({ resets_at: 1793793600000 }) }) }, + { vega: seat({ event: ev({ status: "blocked" }) }) }, { vega: seat({ event: ev({ rate_limit_type: "seven_day_opus" }) }) }, + { vega: seat({ event: ev({ rate_limit_type: null }) }) }, { vega: seat({ event: ev({ received_at: undefined }) }) }, + { vega: seat({ event: ev({ resets_at: "2026-02-30T00:00:00Z" }) }) }, { vega: seat({ event: "rejected" }) }, + ]; + for (const body of bad) assert.equal(cleanLimits({ rigel: seat(), ...body }), null, JSON.stringify(body)); + for (const body of [null, [], "x", 5, [seat()]]) assert.equal(cleanLimits(body), null, JSON.stringify(body)); +}); + +test("isoTime reads only real times with a zone", () => { + assert.equal(isoTime("2026-11-04T12:00Z"), "2026-11-04T12:00:00.000Z"); + assert.equal(isoTime("2026-11-01T01:30:00-03:00"), "2026-11-01T04:30:00.000Z"); + assert.equal(isoTime("2028-02-29T00:00:00Z"), "2028-02-29T00:00:00.000Z"); + for (const t of ["2026-02-29T00:00:00Z", "2026-02-30T00:00:00Z", "2026-13-01T00:00:00Z", "2026-11-04T24:00:00Z", "2026-11-04T12:60:00Z", + "2026-11-04T12:00:00", "2026-11-04", "2026-11-04 12:00:00Z", "2026-11-04T12:00:00+24:00", ` 2026-11-04T12:00:00Z`, `2026-11-04T12:00:00.${"1".repeat(30)}Z`, 1793793600000, null]) { + assert.equal(isoTime(t), null, String(t)); + } +}); + test("hourMs reads only real UTC hours", () => { assert.equal(hourMs("2026-10-14T03"), Date.UTC(2026, 9, 14, 3)); for (const h of ["2026-13-01T00", "2026-10-14T3", " 2026-10-14T03", "2026-10-14T03Z"]) assert.ok(Number.isNaN(hourMs(h)), h); diff --git a/test/worker/usage.test.js b/test/worker/usage.test.js index 6f9bba6..973899c 100644 --- a/test/worker/usage.test.js +++ b/test/worker/usage.test.js @@ -1,6 +1,7 @@ // The Usage Durable Object and its two routes in workerd (setkernel-live#66): /ingest/usage on the // ingest's Access app, /usage/data on its own (503 if USAGE_AUD is empty), sums per Halifax // day, and nothing of it on the public routes. +// /ingest/limits (#88) on the ingest's app too, its readings as /usage/data's limits on the current month only. // Run: npm run test:worker import { env, exports } from "cloudflare:workers"; import { evictDurableObject, runInDurableObject } from "cloudflare:test"; @@ -8,6 +9,7 @@ import { afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import worker from "../../src/worker.js"; import { CERTS, ISSUER } from "../../src/access.js"; import { COUNTS, hourOf as hour } from "../../src/usage.js"; +import { homeDay } from "../../public/lib/state.js"; const BASE = "https://live.setkernel.com"; const OTHER_AUD = "c".repeat(64); // another Access app's tag @@ -49,6 +51,11 @@ async function push(body, { token } = {}) { if (token !== null) headers["Cf-Access-Jwt-Assertion"] = token ?? (await sign(env.INGEST_AUD)); return answer(await exports.default.fetch(`${BASE}/ingest/usage`, { method: "POST", headers, body: typeof body === "string" ? body : JSON.stringify(body) })); } +async function pushLimits(body, { token } = {}) { + const headers = {}; + if (token !== null) headers["Cf-Access-Jwt-Assertion"] = token ?? (await sign(env.INGEST_AUD)); + return answer(await exports.default.fetch(`${BASE}/ingest/limits`, { method: "POST", headers, body: typeof body === "string" ? body : JSON.stringify(body) })); +} // GET /usage/data with the shipped USAGE_AUD, unless `aud` says otherwise. async function read(month, { token, aud = USAGE_AUD } = {}) { const headers = {}; @@ -58,6 +65,8 @@ async function read(month, { token, aud = USAGE_AUD } = {}) { // What the store holds, per hour, seat and model. const stored = () => runInDurableObject(store(), (_, ctx) => ctx.storage.sql.exec("SELECT hour, seat, model, input, requests FROM buckets ORDER BY hour, seat, model").toArray()); +const storedLimits = () => runInDurableObject(store(), (_, ctx) => ctx.storage.sql.exec("SELECT seat, part, at FROM seat_limits ORDER BY seat, part").toArray()); + const bucket = (hour, seat, n, extra = {}) => ({ hour, seat, model: "claude-opus-5-5", surface: "vm", ...Object.fromEntries(COUNTS.map((k, i) => [k, n * (i + 1)])), ...extra }); const counts = (n) => Object.fromEntries(COUNTS.map((k, i) => [k, n * (i + 1)])); @@ -160,6 +169,71 @@ describe("usage", () => { expect((await answer(await worker.fetch(req, { ...env, INGEST_AUD: "" }))).status).toBe(503); }); + // limits: each seat's newest reading, as public/usage/limits.js reads them (setkernel-live#88) + const T0 = Date.now(); + const iso = (ago) => new Date(T0 - ago).toISOString(); + const win = (utilization, ago, extra = {}) => ({ utilization, resets_at: iso(ago - 3 * 3_600_000), observed_at: iso(ago), ...extra }); + const rejected = (ago) => ({ status: "rejected", rate_limit_type: "five_hour", resets_at: iso(ago - 3_600_000), received_at: iso(ago) }); + const thisMonth = () => homeDay(Date.now()).slice(0, 7); + const lastMonth = () => new Date(Date.parse(`${thisMonth()}-01T12:00:00Z`) - 5 * 86_400_000).toISOString().slice(0, 7); + + it("stores each seat's newest limit readings and serves them as limits on the current Halifax month only", async () => { + expect((await read(thisMonth())).json).not.toHaveProperty("limits"); // none yet: the page says it can't check them + const first = { rigel: { five_hour: win(0.84, 60_000), seven_day: win(0.52, 60_000, { resets_at: null }), event: null }, lyra: { five_hour: win(0.31, 120_000), note: "dropped" } }; + const pushed = await pushLimits(first); + expect(pushed.status).toBe(200); + expect(pushed.cache).toBe("no-store"); + const want = { rigel: { five_hour: win(0.84, 60_000), seven_day: win(0.52, 60_000, { resets_at: null }), event: null }, lyra: { five_hour: win(0.31, 120_000), seven_day: null, event: null } }; + expect(pushed.json).toEqual({ limits: want }); + + // an older reading changes nothing, a newer one replaces only its own part, and null erases nothing + await pushLimits({ rigel: { five_hour: win(0.10, 3_600_000), seven_day: null, event: rejected(30_000) }, lyra: { seven_day: win(0.73, 90_000) } }); + const got = await read(thisMonth()); + expect(got.status).toBe(200); + expect(got.json.limits).toEqual({ rigel: { ...want.rigel, event: rejected(30_000) }, lyra: { ...want.lyra, seven_day: win(0.73, 90_000) } }); + await pushLimits({ rigel: { five_hour: win(1, 10_000) } }); + expect((await read(thisMonth())).json.limits.rigel.five_hour).toEqual(win(1, 10_000)); + expect(await storedLimits()).toHaveLength(5); + + // read live, so a past month carries none + const past = await read(lastMonth()); + expect(past.status).toBe(200); + expect(past.json).toEqual({ month: lastMonth(), days: {}, totals: {} }); + }); + + it("refuses a bad limits push with 400 and stores none of it", async () => { + const good = { five_hour: win(0.5, 60_000) }; + for (const body of ["{", "[]", "null", { codex: good }, { vega: good, raghav: good }, { vega: { five_hour: win(-1, 60_000) } }, { vega: { five_hour: win(0.5, 60_000, { observed_at: "yesterday" }) } }, { vega: { event: { ...rejected(1000), rate_limit_type: "overage" } } }]) { + expect((await pushLimits(body)).status, JSON.stringify(body)).toBe(400); + } + expect((await pushLimits("x".repeat(256 * 1024 + 1))).status).toBe(413); + expect(await storedLimits()).toEqual([]); + }); + + it("answers /ingest/limits 403 for no token, a forged one or another app's, 405 for anything but POST, 503 while INGEST_AUD is empty", async () => { + const body = { vega: { five_hour: win(0.5, 60_000) } }; + for (const token of [null, "", "not.a.jwt", await sign(OTHER_AUD), await sign(USAGE_AUD), await sign(env.INGEST_AUD, { iss: "https://evil.cloudflareaccess.com" }), await sign(env.INGEST_AUD, { exp: 1 })]) { + expect((await pushLimits(body, { token })).status).toBe(403); + } + for (const method of ["GET", "PUT", "DELETE"]) expect((await answer(await exports.default.fetch(`${BASE}/ingest/limits`, { method }))).status, method).toBe(405); + const req = new Request(`${BASE}/ingest/limits`, { method: "POST", headers: { "Cf-Access-Jwt-Assertion": await sign(env.INGEST_AUD) }, body: JSON.stringify(body) }); + expect((await answer(await worker.fetch(req, { ...env, INGEST_AUD: "" }))).status).toBe(503); + expect(await storedLimits()).toEqual([]); + }); + + it("shows no limits on /api/state, /api/replay or the /ws first frame", async () => { + await pushLimits({ rigel: { five_hour: win(0.8765, 60_000), seven_day: win(0.4321, 60_000), event: rejected(30_000) } }); + const ws = (await exports.default.fetch(`${BASE}/ws`, { headers: { Upgrade: "websocket" } })).webSocket; + const first = new Promise((resolve) => ws.addEventListener("message", (m) => resolve(m.data))); + ws.accept(); + const seen = [(await answer(await exports.default.fetch(`${BASE}/api/state`))).text, (await answer(await exports.default.fetch(`${BASE}/api/replay?h=24`))).text, await first]; + ws.close(1000, "done"); + await evictDurableObject(hub(), { webSockets: "close" }); + for (const text of seen) { + for (const word of ["limits", "five_hour", "seven_day", "utilization", "0.8765", "0.4321", "rejected", "rate_limit_type", "resets_at", "observed_at", "rigel"]) expect(text).not.toContain(word); + } + }); + it("shows none of it on /api/state, /api/replay or the /ws first frame", async () => { await push({ buckets: [bucket("2026-10-14T15", "rigel", 7_654_321, { model: "claude-haiku-5-5", surface: "codex" })] }); const ws = (await exports.default.fetch(`${BASE}/ws`, { headers: { Upgrade: "websocket" } })).webSocket;