Repository navigation
[AI-3448] Delete activity older than 400 days and write the staff how-to - #2165
Conversation
The cost anomaly check's lock holder deletes customer_activity and activity_digests rows of the UTC days more than 400 days back, once per UTC day. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r not Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
kanat
left a comment
There was a problem hiding this comment.
Reviewed #2165 at 3374bbd merged onto feature/ai-3330-router-usage-tracking (c800c0d). CI: draft, go and integration skipped; they run at gh pr ready. Locally the limitnotice, cmd/router and store suites pass (unit and integration), and so do gofmt and vet.
[Should fix] R1.3 PR body: it went stale after the no-Slack fix. The diagram still says costAnomalies(...).Start() runs "(only with ROUTER_LIMITS_SLACK_TOKEN + _CHANNEL)". Tests leaves out TestActivityIsDeletedWithoutASlackAndNothingIsPosted and TestTheCheckRunsWithoutSlackAndPostsNothing. Nothing says that every router with Postgres now runs the check (a goroutine, an hourly lock try, and the holder's kept connection) whether or not Slack is set. Fix: update the body.
[Nit] R1.4 acceleration/cmd/router/main.go:870-871: the comment still says the check only reads spend and posts the digest. It does not say the check also deletes once a day, or that it runs without Slack.
Checked and sound:
- With Slack set, the options are the same as on base, and the order is digest, then prune, then note.
- On 4.86M seeded
customer_activityrows, the daily delete takes 126 ms (Bitmap Index Scan oncustomer_activity_organization_day). - With no Poster, nothing is posted and no
cost_anomaly_notices/activity_digestsrow is written. Mutants that drop the nil guards panic or fail the new tests. - The UTC day is used on both sides.
- The funnel SQL, run on a seeded DB, matches the hand count.
stepstext matchesai-3447-ops-activity-steps(6aca7a1). Merge #2163 first, or the README describes a field the base lacks.
Verdict: request changes, for R1.1 (the README query misses refusals answered 200/101) and R1.2 (two surviving mutants).
| FROM logs.agent_access_logs | ||
| WHERE k8s_namespace = 'accelerate-prod' -- accelerate-staging on staging | ||
| AND app_id IN (<app ids>) | ||
| AND response >= 400 |
There was a problem hiding this comment.
[Should fix] R1.1 response >= 400 (and line 2230's "from status 400, error_code") misses refusals that answer 200 or 101:
- A refused arriving call is answered 200 with its code noted:
callhooks.go:232,241,255,291andtelnyxhook.go:104-127(telephony suspended, money refusal, no sessions). - A refused socket is a 101:
socketws.go:108,115.
accessFields (server.go:1124) logs error_code whenever a refusal is noted, and callhooks_test.go:214 already asserts status=200 … error_code=unavailable. customer_activity counts these lines as refused (activity.go:141), so for the phone orgs the README query shows fewer refusals than the table does.
Probe: clickhouse local with the agent_access_logs columns from the chart, three refused lines (404, 200, 101). This query returns 1 of them; AND (response >= 400 OR error_code != '') returns all 3.
Fix: use that WHERE, and say that error_code is on every refused request, including a hook or socket refused with 200/101.
There was a problem hiding this comment.
Fixed in 990ad06: the query now reads AND (response >= 400 OR error_code != ''), and the sentence says error_code is on every refused request including a hook or socket refused with 200 or 101. Not re-run on staging ClickHouse (the author's access command is not recorded); the local probe from the review shows 3 of 3 lines.
| if a.digest && a.poster != nil { | ||
| a.postDigest(checking, now) | ||
| } | ||
| a.prune(checking, now) |
There was a problem hiding this comment.
[Should fix] R1.2 Two mutants of the new rules survive every test in the PR:
- MA:
a.prune(checking, now)moved aboveif !held { return }, so every router deletes. That breaks "by one router". - MB:
a.prunedDay = todaymoved beforePruneActivity(line 200), so a failed delete is not tried again that day. The DigestSuite has its own database, so DDL there is fine.
Ready rows, both checked to pass on head and fail under their mutant. Both go in digest_test.go (DigestSuite):
- Only the lock holder deletes: write rows 401 and 0 days back. Hold
s.db.CostAnomalyLock()from another lock, runs.router().Start()ands.checked(0). Expects.days("customer_activity") == ["2025-02-02","2026-03-10"]. ThenRelease(), andEventuallyit is["2026-03-10"]. Kills MA. - A failed delete is retried the same day: use
NewAnomalieswith no Poster andEvery20 ms.ALTER TABLE activity_digests RENAME TO activity_digests_probe, then Start andchecked(0). Rename it back andINSERT INTO activity_digests (day) VALUES ('2025-02-01').Eventuallyactivity_digestsis empty. Kills MB.
There was a problem hiding this comment.
Fixed in 386d1a3: DigestSuite TestOnlyTheRouterHoldingTheLockDeletesTheActivity (prune above the lock check fails it) and TestADeleteThatFailedIsTriedAgainTheSameDay (prunedDay set before the delete fails it); both mutants seen failing.
…cking' into ai-3448-activity-retention
… 200 or 101 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s retried the same day Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
kanat
left a comment
There was a problem hiding this comment.
[reviewer R2] GO
Delta review of #2165 from 3374bbd to b3b9e3d (three commits, base merge 7b0b925 is a clean merge). CI: draft, so go and integration are skipped; labeler and GitGuardian pass.
Prior findings
- R1.1 fixed (
990ad06): the README refusal query now reads(response >= 400 OR error_code != ''). Hooks refused with 200 and sockets refused with 101 carryerror_code(server.go:1124-1125,callhooks.go:232-291,socketws.go:108-115). A local ClickHouse probe on the table's columns returns all refused lines with the new filter and misses the 200 and 101 ones with the old. The corrected query was not re-run on staging. - R1.2 fixed (
386d1a3): both surviving mutants are now killed by the new tests. Prune above the lock check failsTestOnlyTheRouterHoldingTheLockDeletesTheActivity.prunedDayset before the delete failsTestADeleteThatFailedIsTriedAgainTheSameDay. - R1.3 fixed (PR body): the diagram, the no-Slack behaviour and the test list match the code.
- R1.4 fixed (
b3b9e3d): the comment atcmd/router/main.go:870-872is accurate.
New findings
- Nit PR body, "Anomalies.prune, called from check": the call is at
internal/limitnotice/anomaly.go:162, not:161. - Should fix (unchanged text, routed to a ticket by the default policy) PR body: the staging ClickHouse output table and the funnel output table are probe output in PR text, which the public-repo rule excludes. Keep the commands and one line of result, and drop the tables.
- Nit
internal/limitnotice/digest_test.go,TestOnlyTheRouterHoldingTheLockDeletesTheActivity: the lock is released only on the success path, becauses.checked(0)can fail the test beforeelsewhere.Release(). Fix:s.T().Cleanup(elsewhere.Release)afterHold.Releaseis idempotent (internal/store/shipping.go:272-288).
Checks
- Mutants MA and MB re-applied on the new head: both killed. The eight round-1 mutants (MC to MH) were re-run on the head as well: all killed, none survived.
limitnoticeintegration (-p 1) passes.limitnoticeandcmd/routerunit suites pass. The two new tests pass at-count=50 -cpu 1and under-race -count=5.gofmtandgo vetclean on the touched packages.
Unverified: the corrected query on staging, and the body's claim that the table started in Nov 2026.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0d93fc1
into
feature/ai-3330-router-usage-tracking
Problem
customer_activityandactivity_digestskeep rows forever, with no written policy (GDPR Art. 5(1)(e), storage limitation). Staff also have no written way to list an org's refused requests in ClickHouse or to check the funnel order. The README's digest line and ops-endpoint text predate the "Stuck" / "First conversation" sections (#2158) and thestepsfield (#2163).What changed
Without
ROUTER_LIMITS_SLACK_TOKENand_CHANNELthe check still runs: it holds the lock, deletes, and posts nothing (cmd/router/limits.go). Before, it started only with Slack settings, so a deployment without them never deleted. New cost on such a router: one goroutine, one lock attempt per hour, and on the holder one kept Postgres connection.activityKeepDays = 400with its reason (13 months + margin for year-over-year; GDPR Art. 5(1)(e))internal/store/activity_digest.goPruneActivity, copied fromPruneChannelInbound(bunNewDelete, rows affected)internal/store/activity_digest.goAnomalies.prune, called fromcheckinternal/limitnotice/anomaly.go:162steps, "Usage tracking for staff" (where the data lives, 400-day policy, ClickHouse query, funnel SQL)acceleration/README.mdA day 400 days back stays; 401 goes. A router that takes the lock later the same day deletes again and finds nothing (the cutoff is the same).
customer_activityhas no index ondayalone, so the daily delete scans the table once a day. Nothing is deleted on a shared env until rows are 400 days old (the table started in Nov 2026). A step's or an org's "first day" now means the first in the last 400 days; the README says so.Tests
DigestSuite(pinned clock 10 Mar 2026 15:00 UTC, own database):TestActivityPastFourHundredDaysIsDeletedOnceADay: rows 401/400/399/0 days back and digest rows 401/400 days back. After the first check: 401 gone, 400/399/today kept, the 400-day digest kept. A 401-day row written afterwards survives a second check 8 h later the same day (no second delete), and goes the next day with the row that became 401.TestTheKeepCountsFromTheUTCDayWhateverTheClocksZone: 01:00 on 11 Mar in UTC+14 is 10 Mar UTC, so the 400-day row stays.TestOnlyTheRouterHoldingTheLockDeletesTheActivity: another router holds the lock, so a 401-day row stays; after it lets go, the holder's next check deletes it.TestADeleteThatFailedIsTriedAgainTheSameDay: a delete that fails (table renamed in the suite's own database) is tried again at the next check the same day.TestActivityIsDeletedWithoutASlackAndNothingIsPosted,TestTheCheckRunsWithoutSlackAndPostsNothing: no Slack settings, the old rows go and nothing is posted.go test -tags integration -p 1 -count=1 ./internal/limitnotice/...pass;go test ./internal/store/ ./internal/limitnotice/pass;go vet ./...andgo vet -tags integration ./...clean.Mutations (one batched script, 11/11 killed)
prunedDaynever set; day not truncated (today := now)...DeletedOnceADay(second check same day)a.prunecall removed...DeletedOnceADayday <=...DeletedOnceADay(edge rows)activity_digestsnot pruned;customer_activitynot pruned...DeletedOnceADaynow.UTC()dropped in the store...WhateverTheClocksZone...WhateverTheClocksZone(pruned == 1)Also killed: prune above the lock check (
TestOnlyTheRouterHoldingTheLockDeletesTheActivity);prunedDayset before the delete (TestADeleteThatFailedIsTriedAgainTheSameDay); the Slack-only gate restored or the digest/poster built without Slack (the two no-Slack tests). Thectx.Err()andpruned > 0lines only choose a log line.README queries, run for real
Funnel SQL, on a local test database seeded with 4 orgs (seed: dashboard → setup → conversations at 60/59/58 days back plus activity 20 days back; one that stalls; one 10 days old; one that skips setup), output matches the hand count:
ClickHouse query: run read-only on staging ClickHouse (
logs.agent_access_logs,k8s_namespace = 'accelerate-staging', one staging test app, 7 days). It returned rows grouped byerror_codeand operation (output not pasted: public repo). That run used the first version of the query (response >= 400only). The corrected filter,AND (response >= 400 OR error_code != ''), was not re-run on staging. A local ClickHouse probe on the table's schema returns 4 of 4 refused lines with it, and 2 of 4 with the old filter.Columns:
agent_access_logsfrom the ClickHouse chart'sinit-schema-job.yaml(GetStream/chat),app_id= the access line'scustomer,response=status,error_codeas logged; keptappLogs.ttlDays= 14 days. A hook replayed to the call's router is logged on both nodes, so the query counts it twice; the README says so.Gates
This is a data delete on shared envs: name it in #2063's gates. It deletes nothing before rows are 400 days old.
🤖 Generated with Claude Code