Skip to content

perf: reduce duplicate push audit storage - #1795

Merged
jescalada merged 24 commits into
finos:mainfrom
dcoric:perf/push-storage-rollups
Nov 13, 2026
Merged

jescalada merged 24 commits into
finos:mainfrom
dcoric:perf/push-storage-rollups

Conversation

@dcoric

@dcoric dcoric commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Large pushes store the diff repeatedly in processor logs and content, and the final step is duplicated in lastStep. Store each diff once and replace an identical final step with an internal reference in the file, MongoDB, and PostgreSQL sinks. Detail reads restore the complete API response; existing records remain readable.

Push lists and profile activity use the same metadata allowlist across all three sinks. Unused top-level author, message, protocol, and errorMessage fields are excluded. Commit/tag metadata needed by the UI is preserved; full diffs and processor results remain available on the detail endpoint.

Related Issue

Storage portion of #1691 and PostgreSQL parity in #1794, following the merge of #1687. Repository activity summaries and the related UI refresh fixes are now in #1801. Both PRs are needed to complete the agreed scope. The Action.diff refactor in #1693 and #1701 remains separate.

Validation

  • 1,705 unit tests passed; 91.62% line coverage.
  • 67 MongoDB and 87 PostgreSQL integration tests passed, including the shared storage contract and legacy record compatibility.
  • Six real Git push/fetch scenarios and all 18 current Cypress tests passed against PostgreSQL in the combined verification build.
  • Manually verified metadata-only list responses, complete detail diffs, restored lastStep, and rendered push details.
  • Production build, TypeScript, formatting, and lint passed, with the existing tagPush Cypress warning.
  • Combined with perf: cache repository activity summaries #1801 locally: 729 unit/service/processor tests, 95 MongoDB integration tests, and 101 PostgreSQL integration tests passed. The sink adapter overlap has been resolved and verified locally for when the second PR is updated after the first merges.

A constructed push whose final step held a 2 MiB diff serialized to 8,389,308 bytes before compaction and 2,097,722 bytes afterward. Historical records are preserved rather than rewritten.

Checklist

  • CONTRIBUTING.md reviewed; Conventional Commits used
  • FINOS CLA on file
  • Tests and processor documentation updated
  • Unit/integration tests, lint, formatting, type checks, and build passed

No configuration schema changes.

@dcoric
dcoric requested a review from a team as a code owner October 4, 2026 14:37
@netlify

netlify Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Name Link
🔨 Latest commit 76372a2
🔍 Latest deploy log https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/6af6bd523bbb5f92faccb20f
😎 Deploy Preview https://deploy-preview-1795.git-proxy.preview.finos.org
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.95%. Comparing base (6baafbc) to head (76372a2).
⚠️ Report is 31 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1795      +/-   ##
==========================================
+ Coverage   87.05%   87.95%   +0.90%     
==========================================
  Files         114      122       +8     
  Lines        6502     6848     +346     
  Branches     1266     1302      +36     
==========================================
+ Hits         5660     6023     +363     
+ Misses        537      515      -22     
- Partials      305      310       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dcoric added 2 commits October 4, 2026 18:15
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>

@jescalada jescalada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far, just wondering if we could simplify the PR description/title a bit to make it easier to follow.

Since we're technically fixing the audit problem but also doing the push count recalculation stuff, it might even be better to have separate PRs for each 🤔

Comment thread src/db/pushProjection.ts Resolved
Comment thread src/db/mongo/repoActivity.ts Resolved
Comment thread src/db/mongo/repoActivity.ts Resolved
Comment thread src/db/mongo/repoActivity.ts Resolved
Comment thread src/db/mongo/repoActivity.ts Resolved
@kriswest kriswest mentioned this pull request Oct 5, 2026
17 of 24 tasks
dcoric added 4 commits October 6, 2026 12:02
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
@dcoric dcoric mentioned this pull request Oct 6, 2026
4 tasks done
@dcoric dcoric changed the title perf: reduce push audit storage and repository rollup scans perf: reduce duplicate push audit storage Oct 6, 2026
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Comment thread src/db/pushStorage.ts Resolved
@kriswest kriswest modified the milestone: 2.4.0 Oct 19, 2026
@dcoric

dcoric commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Split as suggested: repository activity rollups, the backfill and the dirty-refresh path moved to #1801, so the repoActivity threads here are outdated. The list projection no longer includes author, message, protocol or errorMessage, the AttestationInfo/RejectionInfo changes are no longer in this PR, and the title and description were simplified. Ready for another look.

@dcoric
dcoric requested review from jescalada and kriswest November 7, 2026 22:16

@jescalada jescalada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for simplifying this 🙂

@jescalada
jescalada merged commit 166c719 into finos:main Nov 13, 2026
24 checks passed
@jescalada jescalada linked an issue Nov 17, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complete PostgreSQL push storage and activity parity Reduce stored push document size and rollup scan cost across sink backends

3 participants