Skip to content

[AI-3448] Delete activity older than 400 days and write the staff how-to - #2165

Merged
kanat merged 11 commits into
feature/ai-3330-router-usage-trackingfrom
ai-3448-activity-retention
Nov 20, 2026
Merged

kanat merged 11 commits into
feature/ai-3330-router-usage-trackingfrom
ai-3448-activity-retention

Conversation

@kanat

@kanat kanat commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

customer_activity and activity_digests keep rows forever, with no written policy (GDPR Art. 5(1)(e), storage limitation). Staff also have no written way to list an org's refused requests in ClickHouse or to check the funnel order. The README's digest line and ops-endpoint text predate the "Stuck" / "First conversation" sections (#2158) and the steps field (#2163).

What changed

cmd/router/main.go:872  costAnomalies(...).Start()   (every router with Postgres; Slack settings only turn the posts on)
  └─ Anomalies.check, hourly: its goroutine tries the cost anomaly lock each hour (Hold); the holder
     keeps the lock connection and runs the rest
       ├─ postDigest            (unchanged; only with Slack)
       ├─ prune  ← new: once per UTC day (prunedDay, set only after a delete succeeds), with or without Slack
       │    └─ store.PruneActivity(now)
       │         DELETE FROM customer_activity WHERE day < (UTC today - 400 days)
       │         DELETE FROM activity_digests  WHERE day < (UTC today - 400 days)
       └─ note / unposted       (unchanged; only with Slack)

Without ROUTER_LIMITS_SLACK_TOKEN and _CHANNEL the check still runs: it holds the lock, deletes, and posts nothing (cmd/router/limits.go). Before, it started only with Slack settings, so a deployment without them never deleted. New cost on such a router: one goroutine, one lock attempt per hour, and on the holder one kept Postgres connection.

Piece Where
activityKeepDays = 400 with its reason (13 months + margin for year-over-year; GDPR Art. 5(1)(e)) internal/store/activity_digest.go
PruneActivity, copied from PruneChannelInbound (bun NewDelete, rows affected) internal/store/activity_digest.go
Anomalies.prune, called from check internal/limitnotice/anomaly.go:162
README: digest sections, steps, "Usage tracking for staff" (where the data lives, 400-day policy, ClickHouse query, funnel SQL) acceleration/README.md

A day 400 days back stays; 401 goes. A router that takes the lock later the same day deletes again and finds nothing (the cutoff is the same). customer_activity has no index on day alone, so the daily delete scans the table once a day. Nothing is deleted on a shared env until rows are 400 days old (the table started in Nov 2026). A step's or an org's "first day" now means the first in the last 400 days; the README says so.

Tests

DigestSuite (pinned clock 10 Mar 2026 15:00 UTC, own database):

  • TestActivityPastFourHundredDaysIsDeletedOnceADay: rows 401/400/399/0 days back and digest rows 401/400 days back. After the first check: 401 gone, 400/399/today kept, the 400-day digest kept. A 401-day row written afterwards survives a second check 8 h later the same day (no second delete), and goes the next day with the row that became 401.
  • TestTheKeepCountsFromTheUTCDayWhateverTheClocksZone: 01:00 on 11 Mar in UTC+14 is 10 Mar UTC, so the 400-day row stays.
  • TestOnlyTheRouterHoldingTheLockDeletesTheActivity: another router holds the lock, so a 401-day row stays; after it lets go, the holder's next check deletes it.
  • TestADeleteThatFailedIsTriedAgainTheSameDay: a delete that fails (table renamed in the suite's own database) is tried again at the next check the same day.
  • TestActivityIsDeletedWithoutASlackAndNothingIsPosted, TestTheCheckRunsWithoutSlackAndPostsNothing: no Slack settings, the old rows go and nothing is posted.

go test -tags integration -p 1 -count=1 ./internal/limitnotice/... pass; go test ./internal/store/ ./internal/limitnotice/ pass; go vet ./... and go vet -tags integration ./... clean.

Mutations (one batched script, 11/11 killed)

Mutant Killed by
once-a-day guard off; prunedDay never set; day not truncated (today := now) ...DeletedOnceADay (second check same day)
a.prune call removed ...DeletedOnceADay
keep 401; keep 399; day <= ...DeletedOnceADay (edge rows)
activity_digests not pruned; customer_activity not pruned ...DeletedOnceADay
now.UTC() dropped in the store ...WhateverTheClocksZone
count only the last table ...WhateverTheClocksZone (pruned == 1)

Also killed: prune above the lock check (TestOnlyTheRouterHoldingTheLockDeletesTheActivity); prunedDay set before the delete (TestADeleteThatFailedIsTriedAgainTheSameDay); the Slack-only gate restored or the digest/poster built without Slack (the two no-Slack tests). The ctx.Err() and pruned > 0 lines only choose a log line.

README queries, run for real

Funnel SQL, on a local test database seeded with 4 orgs (seed: dashboard → setup → conversations at 60/59/58 days back plus activity 20 days back; one that stalls; one 10 days old; one that skips setup), output matches the hand count:

      step       | reached | reached_30_days_ago | active_30_days_later
 dashboard_reads |       4 |                   3 |                    2
 agent_setup     |       3 |                   2 |                    1
 conversations   |       2 |                   2 |                    2

ClickHouse query: run read-only on staging ClickHouse (logs.agent_access_logs, k8s_namespace = 'accelerate-staging', one staging test app, 7 days). It returned rows grouped by error_code and operation (output not pasted: public repo). That run used the first version of the query (response >= 400 only). The corrected filter, AND (response >= 400 OR error_code != ''), was not re-run on staging. A local ClickHouse probe on the table's schema returns 4 of 4 refused lines with it, and 2 of 4 with the old filter.

Columns: agent_access_logs from the ClickHouse chart's init-schema-job.yaml (GetStream/chat), app_id = the access line's customer, response = status, error_code as logged; kept appLogs.ttlDays = 14 days. A hook replayed to the call's router is logged on both nodes, so the query counts it twice; the README says so.

Gates

This is a data delete on shared envs: name it in #2063's gates. It deletes nothing before rows are 400 days old.

🤖 Generated with Claude Code

kanat and others added 5 commits November 20, 2026 17:37
The cost anomaly check's lock holder deletes customer_activity and
activity_digests rows of the UTC days more than 400 days back, once
per UTC day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r not

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed #2165 at 3374bbd merged onto feature/ai-3330-router-usage-tracking (c800c0d). CI: draft, go and integration skipped; they run at gh pr ready. Locally the limitnotice, cmd/router and store suites pass (unit and integration), and so do gofmt and vet.

[Should fix] R1.3 PR body: it went stale after the no-Slack fix. The diagram still says costAnomalies(...).Start() runs "(only with ROUTER_LIMITS_SLACK_TOKEN + _CHANNEL)". Tests leaves out TestActivityIsDeletedWithoutASlackAndNothingIsPosted and TestTheCheckRunsWithoutSlackAndPostsNothing. Nothing says that every router with Postgres now runs the check (a goroutine, an hourly lock try, and the holder's kept connection) whether or not Slack is set. Fix: update the body.

[Nit] R1.4 acceleration/cmd/router/main.go:870-871: the comment still says the check only reads spend and posts the digest. It does not say the check also deletes once a day, or that it runs without Slack.

Checked and sound:

  • With Slack set, the options are the same as on base, and the order is digest, then prune, then note.
  • On 4.86M seeded customer_activity rows, the daily delete takes 126 ms (Bitmap Index Scan on customer_activity_organization_day).
  • With no Poster, nothing is posted and no cost_anomaly_notices/activity_digests row is written. Mutants that drop the nil guards panic or fail the new tests.
  • The UTC day is used on both sides.
  • The funnel SQL, run on a seeded DB, matches the hand count.
  • steps text matches ai-3447-ops-activity-steps (6aca7a1). Merge #2163 first, or the README describes a field the base lacks.

Verdict: request changes, for R1.1 (the README query misses refusals answered 200/101) and R1.2 (two surviving mutants).

Comment thread acceleration/README.md Outdated
FROM logs.agent_access_logs
WHERE k8s_namespace = 'accelerate-prod' -- accelerate-staging on staging
AND app_id IN (<app ids>)
AND response >= 400

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.1 response >= 400 (and line 2230's "from status 400, error_code") misses refusals that answer 200 or 101:

  • A refused arriving call is answered 200 with its code noted: callhooks.go:232,241,255,291 and telnyxhook.go:104-127 (telephony suspended, money refusal, no sessions).
  • A refused socket is a 101: socketws.go:108,115.

accessFields (server.go:1124) logs error_code whenever a refusal is noted, and callhooks_test.go:214 already asserts status=200 … error_code=unavailable. customer_activity counts these lines as refused (activity.go:141), so for the phone orgs the README query shows fewer refusals than the table does.

Probe: clickhouse local with the agent_access_logs columns from the chart, three refused lines (404, 200, 101). This query returns 1 of them; AND (response >= 400 OR error_code != '') returns all 3.

Fix: use that WHERE, and say that error_code is on every refused request, including a hook or socket refused with 200/101.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 990ad06: the query now reads AND (response >= 400 OR error_code != ''), and the sentence says error_code is on every refused request including a hook or socket refused with 200 or 101. Not re-run on staging ClickHouse (the author's access command is not recorded); the local probe from the review shows 3 of 3 lines.

if a.digest && a.poster != nil {
a.postDigest(checking, now)
}
a.prune(checking, now)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.2 Two mutants of the new rules survive every test in the PR:

  1. MA: a.prune(checking, now) moved above if !held { return }, so every router deletes. That breaks "by one router".
  2. MB: a.prunedDay = today moved before PruneActivity (line 200), so a failed delete is not tried again that day. The DigestSuite has its own database, so DDL there is fine.

Ready rows, both checked to pass on head and fail under their mutant. Both go in digest_test.go (DigestSuite):

  • Only the lock holder deletes: write rows 401 and 0 days back. Hold s.db.CostAnomalyLock() from another lock, run s.router().Start() and s.checked(0). Expect s.days("customer_activity") == ["2025-02-02","2026-03-10"]. Then Release(), and Eventually it is ["2026-03-10"]. Kills MA.
  • A failed delete is retried the same day: use NewAnomalies with no Poster and Every 20 ms. ALTER TABLE activity_digests RENAME TO activity_digests_probe, then Start and checked(0). Rename it back and INSERT INTO activity_digests (day) VALUES ('2025-02-01'). Eventually activity_digests is empty. Kills MB.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 386d1a3: DigestSuite TestOnlyTheRouterHoldingTheLockDeletesTheActivity (prune above the lock check fails it) and TestADeleteThatFailedIsTriedAgainTheSameDay (prunedDay set before the delete fails it); both mutants seen failing.

kanat and others added 4 commits November 20, 2026 18:15
… 200 or 101

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s retried the same day

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[reviewer R2] GO

Delta review of #2165 from 3374bbd to b3b9e3d (three commits, base merge 7b0b925 is a clean merge). CI: draft, so go and integration are skipped; labeler and GitGuardian pass.

Prior findings

  • R1.1 fixed (990ad06): the README refusal query now reads (response >= 400 OR error_code != ''). Hooks refused with 200 and sockets refused with 101 carry error_code (server.go:1124-1125, callhooks.go:232-291, socketws.go:108-115). A local ClickHouse probe on the table's columns returns all refused lines with the new filter and misses the 200 and 101 ones with the old. The corrected query was not re-run on staging.
  • R1.2 fixed (386d1a3): both surviving mutants are now killed by the new tests. Prune above the lock check fails TestOnlyTheRouterHoldingTheLockDeletesTheActivity. prunedDay set before the delete fails TestADeleteThatFailedIsTriedAgainTheSameDay.
  • R1.3 fixed (PR body): the diagram, the no-Slack behaviour and the test list match the code.
  • R1.4 fixed (b3b9e3d): the comment at cmd/router/main.go:870-872 is accurate.

New findings

  • Nit PR body, "Anomalies.prune, called from check": the call is at internal/limitnotice/anomaly.go:162, not :161.
  • Should fix (unchanged text, routed to a ticket by the default policy) PR body: the staging ClickHouse output table and the funnel output table are probe output in PR text, which the public-repo rule excludes. Keep the commands and one line of result, and drop the tables.
  • Nit internal/limitnotice/digest_test.go, TestOnlyTheRouterHoldingTheLockDeletesTheActivity: the lock is released only on the success path, because s.checked(0) can fail the test before elsewhere.Release(). Fix: s.T().Cleanup(elsewhere.Release) after Hold. Release is idempotent (internal/store/shipping.go:272-288).

Checks

  • Mutants MA and MB re-applied on the new head: both killed. The eight round-1 mutants (MC to MH) were re-run on the head as well: all killed, none survived.
  • limitnotice integration (-p 1) passes. limitnotice and cmd/router unit suites pass. The two new tests pass at -count=50 -cpu 1 and under -race -count=5.
  • gofmt and go vet clean on the touched packages.

Unverified: the corrected query on staging, and the body's claim that the table started in Nov 2026.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kanat
kanat marked this pull request as ready for review November 20, 2026 17:29
@kanat
kanat merged commit 0d93fc1 into feature/ai-3330-router-usage-tracking Nov 20, 2026
11 checks passed
@kanat
kanat deleted the ai-3448-activity-retention branch November 20, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant