Repository navigation
[AI-3447] Ops activity: list every step reached with its first day - #2163
Conversation
GET /v1/ops/orgs/{org_id}/activity answered one furthest_step, so an
organization that only texts read as stuck before voice_video, and staff
could not see how many days each step took.
The answer gains steps: one item per step of activity.Steps with a
request answered from since on, in ladder order, with the UTC day of the
first. ActivityAreas, the query the endpoint already runs, reads that day
per area and client (MIN(day) FILTER (WHERE requests > refused));
activity.ReachedSteps folds it per step. furthest_step is unchanged.
Spec and clients regenerated.
Refs AI-3447
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ctivity-steps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t from since on Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
kanat
left a comment
There was a problem hiding this comment.
Round-1 review at 58fecdd08. The head already contains the base feature/ai-3330-router-usage-tracking at b1d9096cd. The PR is a draft, so every CI job was skipped; CI runs at gh pr ready.
[Should fix] R1.3 The PR body still describes the round-0 design. The code no longer matches it:
- "Store.ActivityAreas + MIN(day) FILTER ... (per area, client)" and "Reuses the query the endpoint already runs". The code now uses a new query,
Store.ActivityFirstAnswered, grouped by area, with nosincebound. - "
first_answeredis the first answered day fromsinceon, likefirst_seenandareas". The code now does the opposite: it reads every day recorded, as the field doc says. - "
areasitems keep their shape: the new store column is not sent there". There is no new store column any more. - Some test names in the body no longer exist:
TestAStepsFirstDayIsReadFromSinceOnand...ItsFirstDayFromSinceWithARequestAnswered. Also,ReachedStepsSuiteno longer tests "earliest of every client". - Most rows of the mutation table name code that was removed: the
FILTER, the zero-day skip,Before→After,!found ||andif found.
Fix: rewrite "What changed", "Tests" and "Mutations" for the current head.
Checked and sound:
- Touched suites pass on the head:
OrganizationActivitySuite(27 tests),PostureSuite(including the committed-spec check),internal/activity, and the fullinternal/storerun (222 s). - Generated files are in step.
generated.shpasses, and rerunningcmd/openapi, the Gogo generate, the Pythongenerate.pyandnpm run types -- --checkchanged nothing. - The operation is still staff-only: the diff adds no
x-client-accessible. - The fields the answer already had are unchanged. The control test asserts their exact JSON, and it kills the mutants that drop
LastRefusalCodeorFirstAt. - Query cost. I ran EXPLAIN ANALYZE on 8.76M seeded rows. The query uses the primary key
(app_id, area, client, day): 8.5 ms for one app and 117 ms cold for 20 apps with a year of rows each. - Mutations: 20 checked, 2 survived.
steps-no-breakis equivalent, because the store returns one row per area. The other survivor is R1.1.
Verdict: request changes, for R1.1-R1.3 (Should fix).
| if err != nil { | ||
| return nil, err | ||
| } | ||
| firsts, err := s.store.ActivityFirstAnswered(ctx, apps) |
There was a problem hiding this comment.
[Should fix] R1.1 Nothing pins that steps covers every app of the organization. The mutant ActivityFirstAnswered(ctx, apps[len(apps)-1:]), which reads only the last app, passes the whole OrganizationActivitySuite, because every steps test records activity for one app only. Fix: add this test to OrganizationActivitySuite. I checked it in a scratch copy: it passes on this head and fails under apps[len(apps)-1:] and under apps[:1].
// Defect: steps read off one app of the organization rather than every app, or off an app of
// another organization.
func (s *OrganizationActivitySuite) TestStepsAreTheEarliestOfEveryAppOfTheOrganization() {
second := s.appOf()
stranger := s.data.createApp()
today := time.Now().UTC().Truncate(24 * time.Hour)
ago := func(n int) time.Time { return today.Add(-time.Duration(n) * 24 * time.Hour) }
other := ago(9).Add(time.Hour)
s.Require().NoError(s.store.RecordActivity(context.Background(), []store.ActivityDelta{
s.answeredOn(activity.AreaAgentSetup, store.AuditSourceCLI, ago(6), 1, 0),
s.answeredOn(activity.AreaConversations, store.AuditSourceSDK, ago(2), 1, 0),
{OrganizationID: s.app.organization.ID, AppID: second, Area: activity.AreaConversations,
Client: store.AuditSourceSDK, Day: ago(5), Requests: 1, FirstAt: ago(5).Add(time.Hour), LastAt: ago(5).Add(time.Hour)},
{OrganizationID: s.app.organization.ID, AppID: second, Area: activity.AreaWorkers,
Client: store.AuditSourceSDK, Day: ago(3), Requests: 1, FirstAt: ago(3).Add(time.Hour), LastAt: ago(3).Add(time.Hour)},
{OrganizationID: stranger.organization.ID, AppID: stranger.app.ID, Area: activity.AreaDashboardReads,
Client: store.AuditSourceDashboard, Day: ago(9), Requests: 1, FirstAt: other, LastAt: other},
}))
s.Equal([]ActivityStepDay{
{Step: activity.AreaAgentSetup, FirstAnswered: ago(6).Format(time.DateOnly)},
{Step: activity.AreaWorkers, FirstAnswered: ago(3).Format(time.DateOnly)},
{Step: activity.AreaConversations, FirstAnswered: ago(5).Format(time.DateOnly)},
}, s.read(s.app.organization.ID, time.Time{}).Steps)
}There was a problem hiding this comment.
Fixed in 6aca7a1: TestStepsAreTheEarliestOfEveryAppOfTheOrganization. Mutants apps[len(apps)-1:] and apps[:1] at the ActivityFirstAnswered call both fail it.
| } | ||
| for _, area := range areas { | ||
| answer.Areas = append(answer.Areas, ActivityArea(area)) | ||
| answer.Areas = append(answer.Areas, ActivityArea{ |
There was a problem hiding this comment.
[Should fix] R1.2 This field-by-field copy is left over from round 0, when store.ActivityArea gained FirstAnswered. The fix removed that field again, so store.ActivityArea is now the same as on the base, and the copy only adds diff. It also loses a check: with the base line, answer.Areas = append(answer.Areas, ActivityArea(area)), the build fails when the two structs drift apart. With the copy, a field added to both structs is dropped from the answer without any error. Fix: go back to ActivityArea(area). The control test TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged still covers it.
There was a problem hiding this comment.
Fixed in 4531a59: back to ActivityArea(area), as the base. No new test: a pure revert to base code; the unchanged-fields control test (TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged) still covers LastRefusalCode and FirstAt.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
kanat
left a comment
There was a problem hiding this comment.
Verdict: GO (delta round 2, head 6aca7a1, base c800c0d already contained).
Prior findings:
- R1.1 (Should fix): fixed. TestStepsAreTheEarliestOfEveryAppOfTheOrganization passes on head; mutants
apps[len(apps)-1:]andapps[:1]at the call site are killed. - R1.2 (Should fix): fixed. org_activity.go:210 is
ActivityArea(area), the same line as base. Explicit-copy mutants that drop LastRefusalCode or FirstAt are killed by TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged. - R1.3 (Should fix, PR body): fixed. The body describes the current design; no round-0 phrases remain. Its test names and mutation rows match the code.
New findings: none.
Suites on 6aca7a1: gofmt and vet (with and without the integration tag) clean; ./internal/activity ok; OrganizationActivitySuite and PostureSuite all pass. go run ./cmd/openapi shows no diff; npm run types -- --check in sdks/js is in step with the spec.
Mutations: 4 checked, 0 survived.
CI: draft, so the jobs are skipped; GitGuardian and labeler pass.
Details: AI-3447/reviewer-2.md under the ship-task ledger.
12747cc
into
feature/ai-3330-router-usage-tracking
…-to (#2165) ## Problem `customer_activity` and `activity_digests` keep rows forever, with no written policy (GDPR Art. 5(1)(e), storage limitation). Staff also have no written way to list an org's refused requests in ClickHouse or to check the funnel order. The README's digest line and ops-endpoint text predate the "Stuck" / "First conversation" sections (#2158) and the `steps` field (#2163). ## What changed ``` cmd/router/main.go:872 costAnomalies(...).Start() (every router with Postgres; Slack settings only turn the posts on) └─ Anomalies.check, hourly: its goroutine tries the cost anomaly lock each hour (Hold); the holder keeps the lock connection and runs the rest ├─ postDigest (unchanged; only with Slack) ├─ prune ← new: once per UTC day (prunedDay, set only after a delete succeeds), with or without Slack │ └─ store.PruneActivity(now) │ DELETE FROM customer_activity WHERE day < (UTC today - 400 days) │ DELETE FROM activity_digests WHERE day < (UTC today - 400 days) └─ note / unposted (unchanged; only with Slack) ``` Without `ROUTER_LIMITS_SLACK_TOKEN` and `_CHANNEL` the check still runs: it holds the lock, deletes, and posts nothing (`cmd/router/limits.go`). Before, it started only with Slack settings, so a deployment without them never deleted. New cost on such a router: one goroutine, one lock attempt per hour, and on the holder one kept Postgres connection. | Piece | Where | | --- | --- | | `activityKeepDays = 400` with its reason (13 months + margin for year-over-year; GDPR Art. 5(1)(e)) | `internal/store/activity_digest.go` | | `PruneActivity`, copied from `PruneChannelInbound` (bun `NewDelete`, rows affected) | `internal/store/activity_digest.go` | | `Anomalies.prune`, called from `check` | `internal/limitnotice/anomaly.go:162` | | README: digest sections, `steps`, "Usage tracking for staff" (where the data lives, 400-day policy, ClickHouse query, funnel SQL) | `acceleration/README.md` | A day 400 days back stays; 401 goes. A router that takes the lock later the same day deletes again and finds nothing (the cutoff is the same). `customer_activity` has no index on `day` alone, so the daily delete scans the table once a day. Nothing is deleted on a shared env until rows are 400 days old (the table started in Nov 2026). A step's or an org's "first day" now means the first in the last 400 days; the README says so. ## Tests `DigestSuite` (pinned clock 10 Mar 2026 15:00 UTC, own database): - `TestActivityPastFourHundredDaysIsDeletedOnceADay`: rows 401/400/399/0 days back and digest rows 401/400 days back. After the first check: 401 gone, 400/399/today kept, the 400-day digest kept. A 401-day row written afterwards survives a second check 8 h later the same day (no second delete), and goes the next day with the row that became 401. - `TestTheKeepCountsFromTheUTCDayWhateverTheClocksZone`: 01:00 on 11 Mar in UTC+14 is 10 Mar UTC, so the 400-day row stays. - `TestOnlyTheRouterHoldingTheLockDeletesTheActivity`: another router holds the lock, so a 401-day row stays; after it lets go, the holder's next check deletes it. - `TestADeleteThatFailedIsTriedAgainTheSameDay`: a delete that fails (table renamed in the suite's own database) is tried again at the next check the same day. - `TestActivityIsDeletedWithoutASlackAndNothingIsPosted`, `TestTheCheckRunsWithoutSlackAndPostsNothing`: no Slack settings, the old rows go and nothing is posted. `go test -tags integration -p 1 -count=1 ./internal/limitnotice/...` pass; `go test ./internal/store/ ./internal/limitnotice/` pass; `go vet ./...` and `go vet -tags integration ./...` clean. ## Mutations (one batched script, 11/11 killed) | Mutant | Killed by | | --- | --- | | once-a-day guard off; `prunedDay` never set; day not truncated (`today := now`) | `...DeletedOnceADay` (second check same day) | | `a.prune` call removed | `...DeletedOnceADay` | | keep 401; keep 399; `day <=` | `...DeletedOnceADay` (edge rows) | | `activity_digests` not pruned; `customer_activity` not pruned | `...DeletedOnceADay` | | `now.UTC()` dropped in the store | `...WhateverTheClocksZone` | | count only the last table | `...WhateverTheClocksZone` (`pruned == 1`) | Also killed: prune above the lock check (`TestOnlyTheRouterHoldingTheLockDeletesTheActivity`); `prunedDay` set before the delete (`TestADeleteThatFailedIsTriedAgainTheSameDay`); the Slack-only gate restored or the digest/poster built without Slack (the two no-Slack tests). The `ctx.Err()` and `pruned > 0` lines only choose a log line. ## README queries, run for real **Funnel SQL**, on a local test database seeded with 4 orgs (seed: dashboard → setup → conversations at 60/59/58 days back plus activity 20 days back; one that stalls; one 10 days old; one that skips setup), output matches the hand count: ``` step | reached | reached_30_days_ago | active_30_days_later dashboard_reads | 4 | 3 | 2 agent_setup | 3 | 2 | 1 conversations | 2 | 2 | 2 ``` **ClickHouse query**: run read-only on staging ClickHouse (`logs.agent_access_logs`, `k8s_namespace = 'accelerate-staging'`, one staging test app, 7 days). It returned rows grouped by `error_code` and operation (output not pasted: public repo). That run used the first version of the query (`response >= 400` only). The corrected filter, `AND (response >= 400 OR error_code != '')`, was not re-run on staging. A local ClickHouse probe on the table's schema returns 4 of 4 refused lines with it, and 2 of 4 with the old filter. Columns: `agent_access_logs` from the ClickHouse chart's `init-schema-job.yaml` (GetStream/chat), `app_id` = the access line's `customer`, `response` = `status`, `error_code` as logged; kept `appLogs.ttlDays` = 14 days. A hook replayed to the call's router is logged on both nodes, so the query counts it twice; the README says so. ## Gates This is a data delete on shared envs: name it in #2063's gates. It deletes nothing before rows are 400 days old. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Problem
GET /v1/ops/orgs/{org_id}/activityanswers onefurthest_step. An organization that only texts stops atconversations, and reads as stuck beforevoice_video. Staff also cannot see how many days each step took.What changed
furthest_stepconversationsconversations(unchanged)steps[{dashboard_reads, D}, {agent_setup, D+1}, {conversations, D+6}]first_answeredis the first answered day ever recorded for the step, over every app of the organization and every client. It is not limited tosince, so it is true for an organization older than the window.stepslists every step ever answered; a step whose every request was refused is absent (same rule asfurthest_step).furthest_step,areas,first_seen,last_seenandmeterskeep theirsincewindow and their shape.store.ActivityAreasandActivityAreaare untouched.Store.FirstAnswers([AI-3445] Digest: list stuck orgs and time to first conversation #2158), which lists every organization first answered on one given day (the digest's shape).(app_id, area, client, day). EXPLAIN on a scratch table of 8.76M rows (500 apps x 12 areas x 4 clients x 365 days): 1 app 8.5 ms, 20 apps 117 ms cold. Retention is AI-3448.Tests
OrganizationActivitySuite.TestStepsListEveryStepAnsweredWithItsFirstDayInLadderOrder: the ticket's case (D, D+1, D+6, voice_video only refused) plus a refused-only earlier day, a later answered day, a second client and a non-step area.TestAStepsFirstDayAndTheStepsListedReachBeyondSince: a day 45 days back is the first day althoughsinceis 30 days, whilefurthest_stepandareasstill follow the window.TestStepsAreTheEarliestOfEveryAppOfTheOrganization: two apps of the organization each hold a different step day, an app of another organization holds an earlier one.TestAnOrganizationWithNothingRecordedHasEmptyListsAndNoStepalso assertssteps: [].StoreSuite.TestTheFirstAnsweredDayOfAnAreaIsItsFirstDayEverWithARequestAnswered(runs in both store suites).ReachedStepsSuite(unit): ladder order, non-step areas left out, nothing recorded.TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchangedasserts the base answer's keys and the exact JSON ofareas,meters,first_seen,last_seenandfurthest_step.Suites:
internal/api(OrganizationActivity, Posture),internal/activity: pass.Mutations
Each mutant was applied alone and a test failed, except the one marked equivalent.
requests > refused>to>=MIN(day)toMAX(day)since(30 days)TestAStepsFirstDayAndTheStepsListedReachBeyondSinceStepsStepsnull or unsetTestStepsAreTheEarliestOfEveryAppOfTheOrganizationtime.RFC3339instead of the dateLastRefusalCodeorFirstAtfromareasEquivalent, not killed: removing the
breakinReachedSteps, because the query returns one row per area.🤖 Generated with Claude Code