Repository navigation
infra(hub): latency alarm from the hub log; maintenance window after the backup run (D-196) - #155
Merged
Merged
Conversation
…ance window after the backup run (D-196) The latency-p99 alarm on the ALB's TargetResponseTime was in ALARM from the first day: the 25 s sync long-poll and the WebSocket upgrade dominate the p99. A log metric filter now publishes elapsed_ms of every other request as RequestLatencyMs and the alarm watches that. Every Sunday's AWS Backup job failed because the 06:00 UTC run was too close to the sun:06:30 RDS maintenance window; the window moves to sun:09:00. The us-west-2 region policy exception is narrowed to listed actions (recorded in D-072's notes). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
blove
enabled auto-merge
November 9, 2026 22:33
Terraform plan: testPlan: 1 to add, 2 to change, 0 to destroy (a replacement counts as both an add and a destroy).
The full plan is in the workflow run log. |
Terraform plan: prodPlan: 3 to add, 3 to change, 0 to destroy (a replacement counts as both an add and a destroy).
The full plan is in the workflow run log. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
latency-p99was in ALARM in both accounts since 2026-11-02: the ALB p99 is the 25 s sync long-poll. A CloudWatch Logs metric filter on/b4hub/<env>/hubpublisheselapsed_msfor everyrequest completeline except/v1/changesand/v1/devices/self/link; the alarm is p99 > 2000 ms for ten minutes on that metric.sun:06:30tosun:09:00UTC.us-west-2policy exception was narrowed today).Module tests: 53 pass;
tflint,terraform fmtclean. Expected plan per account: 1 metric filter to add, the latency alarm replaced (namespace change), the RDS instance modified in place (window;apply_immediately = false, so it takes effect at the next window).🤖 Generated with Claude Code