Skip to content

[AI-3447] Ops activity: list every step reached with its first day - #2163

Merged
kanat merged 7 commits into
feature/ai-3330-router-usage-trackingfrom
ai-3447-ops-activity-steps
Nov 20, 2026
Merged

kanat merged 7 commits into
feature/ai-3330-router-usage-trackingfrom
ai-3447-ops-activity-steps

Conversation

@kanat

@kanat kanat commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

GET /v1/ops/orgs/{org_id}/activity answers one furthest_step. An organization that only texts stops at conversations, and reads as stuck before voice_video. Staff also cannot see how many days each step took.

What changed

customer_activity (shared DB)
  └─ Store.ActivityFirstAnswered   MIN(day) per area over every day recorded, answered rows only (requests > refused)
       └─ activity.ReachedSteps    the areas that are steps, in activity.Steps order
            └─ getOrganizationActivity   steps: [{step, first_answered: "YYYY-MM-DD"}]
Field Before After
furthest_step conversations conversations (unchanged)
steps absent [{dashboard_reads, D}, {agent_setup, D+1}, {conversations, D+6}]
  • first_answered is the first answered day ever recorded for the step, over every app of the organization and every client. It is not limited to since, so it is true for an organization older than the window.
  • steps lists every step ever answered; a step whose every request was refused is absent (same rule as furthest_step).
  • furthest_step, areas, first_seen, last_seen and meters keep their since window and their shape. store.ActivityAreas and ActivityArea are untouched.
  • A new store query rather than Store.FirstAnswers ([AI-3445] Digest: list stuck orgs and time to first conversation #2158), which lists every organization first answered on one given day (the digest's shape).
  • The query has no day bound: it reads the organization's apps' rows from the primary key (app_id, area, client, day). EXPLAIN on a scratch table of 8.76M rows (500 apps x 12 areas x 4 clients x 365 days): 1 app 8.5 ms, 20 apps 117 ms cold. Retention is AI-3448.
  • Spec and clients regenerated: Python, Go, JS, .NET, PHP, Ruby, Rust. Kotlin, Dart and Swift are unchanged (staff-only operation, not in their client-accessible filter).

Tests

  • OrganizationActivitySuite.TestStepsListEveryStepAnsweredWithItsFirstDayInLadderOrder: the ticket's case (D, D+1, D+6, voice_video only refused) plus a refused-only earlier day, a later answered day, a second client and a non-step area.
  • TestAStepsFirstDayAndTheStepsListedReachBeyondSince: a day 45 days back is the first day although since is 30 days, while furthest_step and areas still follow the window.
  • TestStepsAreTheEarliestOfEveryAppOfTheOrganization: two apps of the organization each hold a different step day, an app of another organization holds an earlier one.
  • TestAnOrganizationWithNothingRecordedHasEmptyListsAndNoStep also asserts steps: [].
  • StoreSuite.TestTheFirstAnsweredDayOfAnAreaIsItsFirstDayEverWithARequestAnswered (runs in both store suites).
  • ReachedStepsSuite (unit): ladder order, non-step areas left out, nothing recorded.
  • Control: TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged asserts the base answer's keys and the exact JSON of areas, meters, first_seen, last_seen and furthest_step.

Suites: internal/api (OrganizationActivity, Posture), internal/activity: pass.

Mutations

Each mutant was applied alone and a test failed, except the one marked equivalent.

Mutation Killed by
store: drop requests > refused store test; API ladder test
store: > to >= store test
store: MIN(day) to MAX(day) store test
store: drop the app filter store test
store: group by client too store test
store: bound the query to since (30 days) TestAStepsFirstDayAndTheStepsListedReachBeyondSince
steps: iterate areas in name order, not Steps ladder-order unit and API tests
steps: return nil unit tests
API: no append, Steps null or unset API ladder and nothing-recorded tests
API call site: nil apps, org id, first app only API tests
API call site: last app only TestStepsAreTheEarliestOfEveryAppOfTheOrganization
API: time.RFC3339 instead of the date API ladder test
API: drop LastRefusalCode or FirstAt from areas the unchanged-fields control test

Equivalent, not killed: removing the break in ReachedSteps, because the query returns one row per area.

🤖 Generated with Claude Code

kanat and others added 2 commits November 20, 2026 17:17
GET /v1/ops/orgs/{org_id}/activity answered one furthest_step, so an
organization that only texts read as stuck before voice_video, and staff
could not see how many days each step took.

The answer gains steps: one item per step of activity.Steps with a
request answered from since on, in ladder order, with the UTC day of the
first. ActivityAreas, the query the endpoint already runs, reads that day
per area and client (MIN(day) FILTER (WHERE requests > refused));
activity.ReachedSteps folds it per step. furthest_step is unchanged.

Spec and clients regenerated.

Refs AI-3447

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ctivity-steps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kanat and others added 2 commits November 20, 2026 17:23
…t from since on

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round-1 review at 58fecdd08. The head already contains the base feature/ai-3330-router-usage-tracking at b1d9096cd. The PR is a draft, so every CI job was skipped; CI runs at gh pr ready.

[Should fix] R1.3 The PR body still describes the round-0 design. The code no longer matches it:

  • "Store.ActivityAreas + MIN(day) FILTER ... (per area, client)" and "Reuses the query the endpoint already runs". The code now uses a new query, Store.ActivityFirstAnswered, grouped by area, with no since bound.
  • "first_answered is the first answered day from since on, like first_seen and areas". The code now does the opposite: it reads every day recorded, as the field doc says.
  • "areas items keep their shape: the new store column is not sent there". There is no new store column any more.
  • Some test names in the body no longer exist: TestAStepsFirstDayIsReadFromSinceOn and ...ItsFirstDayFromSinceWithARequestAnswered. Also, ReachedStepsSuite no longer tests "earliest of every client".
  • Most rows of the mutation table name code that was removed: the FILTER, the zero-day skip, Before → After, !found || and if found.

Fix: rewrite "What changed", "Tests" and "Mutations" for the current head.

Checked and sound:

  • Touched suites pass on the head: OrganizationActivitySuite (27 tests), PostureSuite (including the committed-spec check), internal/activity, and the full internal/store run (222 s).
  • Generated files are in step. generated.sh passes, and rerunning cmd/openapi, the Go go generate, the Python generate.py and npm run types -- --check changed nothing.
  • The operation is still staff-only: the diff adds no x-client-accessible.
  • The fields the answer already had are unchanged. The control test asserts their exact JSON, and it kills the mutants that drop LastRefusalCode or FirstAt.
  • Query cost. I ran EXPLAIN ANALYZE on 8.76M seeded rows. The query uses the primary key (app_id, area, client, day): 8.5 ms for one app and 117 ms cold for 20 apps with a year of rows each.
  • Mutations: 20 checked, 2 survived. steps-no-break is equivalent, because the store returns one row per area. The other survivor is R1.1.

Verdict: request changes, for R1.1-R1.3 (Should fix).

if err != nil {
return nil, err
}
firsts, err := s.store.ActivityFirstAnswered(ctx, apps)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.1 Nothing pins that steps covers every app of the organization. The mutant ActivityFirstAnswered(ctx, apps[len(apps)-1:]), which reads only the last app, passes the whole OrganizationActivitySuite, because every steps test records activity for one app only. Fix: add this test to OrganizationActivitySuite. I checked it in a scratch copy: it passes on this head and fails under apps[len(apps)-1:] and under apps[:1].

// Defect: steps read off one app of the organization rather than every app, or off an app of
// another organization.
func (s *OrganizationActivitySuite) TestStepsAreTheEarliestOfEveryAppOfTheOrganization() {
	second := s.appOf()
	stranger := s.data.createApp()
	today := time.Now().UTC().Truncate(24 * time.Hour)
	ago := func(n int) time.Time { return today.Add(-time.Duration(n) * 24 * time.Hour) }
	other := ago(9).Add(time.Hour)
	s.Require().NoError(s.store.RecordActivity(context.Background(), []store.ActivityDelta{
		s.answeredOn(activity.AreaAgentSetup, store.AuditSourceCLI, ago(6), 1, 0),
		s.answeredOn(activity.AreaConversations, store.AuditSourceSDK, ago(2), 1, 0),
		{OrganizationID: s.app.organization.ID, AppID: second, Area: activity.AreaConversations,
			Client: store.AuditSourceSDK, Day: ago(5), Requests: 1, FirstAt: ago(5).Add(time.Hour), LastAt: ago(5).Add(time.Hour)},
		{OrganizationID: s.app.organization.ID, AppID: second, Area: activity.AreaWorkers,
			Client: store.AuditSourceSDK, Day: ago(3), Requests: 1, FirstAt: ago(3).Add(time.Hour), LastAt: ago(3).Add(time.Hour)},
		{OrganizationID: stranger.organization.ID, AppID: stranger.app.ID, Area: activity.AreaDashboardReads,
			Client: store.AuditSourceDashboard, Day: ago(9), Requests: 1, FirstAt: other, LastAt: other},
	}))

	s.Equal([]ActivityStepDay{
		{Step: activity.AreaAgentSetup, FirstAnswered: ago(6).Format(time.DateOnly)},
		{Step: activity.AreaWorkers, FirstAnswered: ago(3).Format(time.DateOnly)},
		{Step: activity.AreaConversations, FirstAnswered: ago(5).Format(time.DateOnly)},
	}, s.read(s.app.organization.ID, time.Time{}).Steps)
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6aca7a1: TestStepsAreTheEarliestOfEveryAppOfTheOrganization. Mutants apps[len(apps)-1:] and apps[:1] at the ActivityFirstAnswered call both fail it.

}
for _, area := range areas {
answer.Areas = append(answer.Areas, ActivityArea(area))
answer.Areas = append(answer.Areas, ActivityArea{

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.2 This field-by-field copy is left over from round 0, when store.ActivityArea gained FirstAnswered. The fix removed that field again, so store.ActivityArea is now the same as on the base, and the copy only adds diff. It also loses a check: with the base line, answer.Areas = append(answer.Areas, ActivityArea(area)), the build fails when the two structs drift apart. With the copy, a field added to both structs is dropped from the answer without any error. Fix: go back to ActivityArea(area). The control test TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged still covers it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4531a59: back to ActivityArea(area), as the base. No new test: a pure revert to base code; the unchanged-fields control test (TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged) still covers LastRefusalCode and FirstAt.

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: GO (delta round 2, head 6aca7a1, base c800c0d already contained).

Prior findings:

  • R1.1 (Should fix): fixed. TestStepsAreTheEarliestOfEveryAppOfTheOrganization passes on head; mutants apps[len(apps)-1:] and apps[:1] at the call site are killed.
  • R1.2 (Should fix): fixed. org_activity.go:210 is ActivityArea(area), the same line as base. Explicit-copy mutants that drop LastRefusalCode or FirstAt are killed by TestTheFieldsTheAnswerHadBeforeSessionsAndLatencyAreUnchanged.
  • R1.3 (Should fix, PR body): fixed. The body describes the current design; no round-0 phrases remain. Its test names and mutation rows match the code.

New findings: none.

Suites on 6aca7a1: gofmt and vet (with and without the integration tag) clean; ./internal/activity ok; OrganizationActivitySuite and PostureSuite all pass. go run ./cmd/openapi shows no diff; npm run types -- --check in sdks/js is in step with the spec.

Mutations: 4 checked, 0 survived.

CI: draft, so the jobs are skipped; GitGuardian and labeler pass.

Details: AI-3447/reviewer-2.md under the ship-task ledger.

@kanat
kanat marked this pull request as ready for review November 20, 2026 16:57
@kanat
kanat merged commit 12747cc into feature/ai-3330-router-usage-tracking Nov 20, 2026
20 of 21 checks passed
@kanat
kanat deleted the ai-3447-ops-activity-steps branch November 20, 2026 16:57
kanat added a commit that referenced this pull request Nov 20, 2026
…-to (#2165)

## Problem

`customer_activity` and `activity_digests` keep rows forever, with no
written policy (GDPR Art. 5(1)(e), storage limitation). Staff also have
no written way to list an org's refused requests in ClickHouse or to
check the funnel order. The README's digest line and ops-endpoint text
predate the "Stuck" / "First conversation" sections (#2158) and the
`steps` field (#2163).

## What changed

```
cmd/router/main.go:872  costAnomalies(...).Start()   (every router with Postgres; Slack settings only turn the posts on)
  └─ Anomalies.check, hourly: its goroutine tries the cost anomaly lock each hour (Hold); the holder
     keeps the lock connection and runs the rest
       ├─ postDigest            (unchanged; only with Slack)
       ├─ prune  ← new: once per UTC day (prunedDay, set only after a delete succeeds), with or without Slack
       │    └─ store.PruneActivity(now)
       │         DELETE FROM customer_activity WHERE day < (UTC today - 400 days)
       │         DELETE FROM activity_digests  WHERE day < (UTC today - 400 days)
       └─ note / unposted       (unchanged; only with Slack)
```

Without `ROUTER_LIMITS_SLACK_TOKEN` and `_CHANNEL` the check still runs:
it holds the lock, deletes, and posts nothing (`cmd/router/limits.go`).
Before, it started only with Slack settings, so a deployment without
them never deleted. New cost on such a router: one goroutine, one lock
attempt per hour, and on the holder one kept Postgres connection.

| Piece | Where |
| --- | --- |
| `activityKeepDays = 400` with its reason (13 months + margin for
year-over-year; GDPR Art. 5(1)(e)) | `internal/store/activity_digest.go`
|
| `PruneActivity`, copied from `PruneChannelInbound` (bun `NewDelete`,
rows affected) | `internal/store/activity_digest.go` |
| `Anomalies.prune`, called from `check` |
`internal/limitnotice/anomaly.go:162` |
| README: digest sections, `steps`, "Usage tracking for staff" (where
the data lives, 400-day policy, ClickHouse query, funnel SQL) |
`acceleration/README.md` |

A day 400 days back stays; 401 goes. A router that takes the lock later
the same day deletes again and finds nothing (the cutoff is the same).
`customer_activity` has no index on `day` alone, so the daily delete
scans the table once a day. Nothing is deleted on a shared env until
rows are 400 days old (the table started in Nov 2026). A step's or an
org's "first day" now means the first in the last 400 days; the README
says so.

## Tests

`DigestSuite` (pinned clock 10 Mar 2026 15:00 UTC, own database):

- `TestActivityPastFourHundredDaysIsDeletedOnceADay`: rows 401/400/399/0
days back and digest rows 401/400 days back. After the first check: 401
gone, 400/399/today kept, the 400-day digest kept. A 401-day row written
afterwards survives a second check 8 h later the same day (no second
delete), and goes the next day with the row that became 401.
- `TestTheKeepCountsFromTheUTCDayWhateverTheClocksZone`: 01:00 on 11 Mar
in UTC+14 is 10 Mar UTC, so the 400-day row stays.
- `TestOnlyTheRouterHoldingTheLockDeletesTheActivity`: another router
holds the lock, so a 401-day row stays; after it lets go, the holder's
next check deletes it.
- `TestADeleteThatFailedIsTriedAgainTheSameDay`: a delete that fails
(table renamed in the suite's own database) is tried again at the next
check the same day.
- `TestActivityIsDeletedWithoutASlackAndNothingIsPosted`,
`TestTheCheckRunsWithoutSlackAndPostsNothing`: no Slack settings, the
old rows go and nothing is posted.

`go test -tags integration -p 1 -count=1 ./internal/limitnotice/...`
pass; `go test ./internal/store/ ./internal/limitnotice/` pass; `go vet
./...` and `go vet -tags integration ./...` clean.

## Mutations (one batched script, 11/11 killed)

| Mutant | Killed by |
| --- | --- |
| once-a-day guard off; `prunedDay` never set; day not truncated (`today
:= now`) | `...DeletedOnceADay` (second check same day) |
| `a.prune` call removed | `...DeletedOnceADay` |
| keep 401; keep 399; `day <=` | `...DeletedOnceADay` (edge rows) |
| `activity_digests` not pruned; `customer_activity` not pruned |
`...DeletedOnceADay` |
| `now.UTC()` dropped in the store | `...WhateverTheClocksZone` |
| count only the last table | `...WhateverTheClocksZone` (`pruned == 1`)
|

Also killed: prune above the lock check
(`TestOnlyTheRouterHoldingTheLockDeletesTheActivity`); `prunedDay` set
before the delete (`TestADeleteThatFailedIsTriedAgainTheSameDay`); the
Slack-only gate restored or the digest/poster built without Slack (the
two no-Slack tests). The `ctx.Err()` and `pruned > 0` lines only choose
a log line.

## README queries, run for real

**Funnel SQL**, on a local test database seeded with 4 orgs (seed:
dashboard → setup → conversations at 60/59/58 days back plus activity 20
days back; one that stalls; one 10 days old; one that skips setup),
output matches the hand count:

```
      step       | reached | reached_30_days_ago | active_30_days_later
 dashboard_reads |       4 |                   3 |                    2
 agent_setup     |       3 |                   2 |                    1
 conversations   |       2 |                   2 |                    2
```

**ClickHouse query**: run read-only on staging ClickHouse
(`logs.agent_access_logs`, `k8s_namespace = 'accelerate-staging'`, one
staging test app, 7 days). It returned rows grouped by `error_code` and
operation (output not pasted: public repo). That run used the first
version of the query (`response >= 400` only). The corrected filter,
`AND (response >= 400 OR error_code != '')`, was not re-run on staging.
A local ClickHouse probe on the table's schema returns 4 of 4 refused
lines with it, and 2 of 4 with the old filter.

Columns: `agent_access_logs` from the ClickHouse chart's
`init-schema-job.yaml` (GetStream/chat), `app_id` = the access line's
`customer`, `response` = `status`, `error_code` as logged; kept
`appLogs.ttlDays` = 14 days. A hook replayed to the call's router is
logged on both nodes, so the query counts it twice; the README says so.

## Gates

This is a data delete on shared envs: name it in #2063's gates. It
deletes nothing before rows are 400 days old.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant