Repository navigation
perf: reduce duplicate push audit storage - #1795
Conversation
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1795 +/- ##
==========================================
+ Coverage 87.05% 87.95% +0.90%
==========================================
Files 114 122 +8
Lines 6502 6848 +346
Branches 1266 1302 +36
==========================================
+ Hits 5660 6023 +363
+ Misses 537 515 -22
- Partials 305 310 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
jescalada
left a comment
There was a problem hiding this comment.
Looks good so far, just wondering if we could simplify the PR description/title a bit to make it easier to follow.
Since we're technically fixing the audit problem but also doing the push count recalculation stuff, it might even be better to have separate PRs for each 🤔
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
|
Split as suggested: repository activity rollups, the backfill and the dirty-refresh path moved to #1801, so the repoActivity threads here are outdated. The list projection no longer includes author, message, protocol or errorMessage, the AttestationInfo/RejectionInfo changes are no longer in this PR, and the title and description were simplified. Ready for another look. |
jescalada
left a comment
There was a problem hiding this comment.
Thanks for simplifying this 🙂
Description
Large pushes store the diff repeatedly in processor logs and content, and the final step is duplicated in
lastStep. Store each diff once and replace an identical final step with an internal reference in the file, MongoDB, and PostgreSQL sinks. Detail reads restore the complete API response; existing records remain readable.Push lists and profile activity use the same metadata allowlist across all three sinks. Unused top-level author, message, protocol, and errorMessage fields are excluded. Commit/tag metadata needed by the UI is preserved; full diffs and processor results remain available on the detail endpoint.
Related Issue
Storage portion of #1691 and PostgreSQL parity in #1794, following the merge of #1687. Repository activity summaries and the related UI refresh fixes are now in #1801. Both PRs are needed to complete the agreed scope. The Action.diff refactor in #1693 and #1701 remains separate.
Validation
A constructed push whose final step held a 2 MiB diff serialized to 8,389,308 bytes before compaction and 2,097,722 bytes afterward. Historical records are preserved rather than rewritten.
Checklist
No configuration schema changes.