Skip to content

perf: cache repository activity summaries - #1801

Merged
jescalada merged 15 commits into
finos:mainfrom
dcoric:perf/repository-activity-summaries
Nov 16, 2026
Merged

jescalada merged 15 commits into
finos:mainfrom
dcoric:perf/repository-activity-summaries

Conversation

@dcoric

@dcoric dcoric commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Repository pages no longer scan the full push history for push counts and recent activity. The file, MongoDB and PostgreSQL sinks keep per-repository activity summaries, updated on every push write and backfilled by a startup migration. On a 10,000-push PostgreSQL history, warm reads dropped from 11.27 ms to 2.44 ms.

Upgrade notes (MongoDB duplicate push IDs, PostgreSQL with autoMigrate: false, rebuilding summaries) are in the new website/docs/migrations.mdx. Review actions also refresh the relevant UI caches and approval/rejection timestamps are parsed consistently.

Validation and implementation details: #1801 (comment)

Related Issue

Activity portion of #1691, with PostgreSQL parity for #1794. Audit storage changes are in #1795; both are needed for the agreed scope.

Checklist

  • CONTRIBUTING.md reviewed; Conventional Commits used
  • FINOS CLA on file
  • Tests and documentation updated
  • Unit/integration tests, lint, formatting, type checks, and build passed

No configuration schema changes.

dcoric added 5 commits October 6, 2026 12:23
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
@dcoric
dcoric requested a review from a team as a code owner October 6, 2026 10:30
@netlify

netlify Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit 758ee04
🔍 Latest deploy log https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/6afafb5373444f9b27e7f714
😎 Deploy Preview https://deploy-preview-1801.git-proxy.preview.finos.org
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.83382% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.87%. Comparing base (a6bce7d) to head (758ee04).

Files with missing lines Patch % Lines
src/db/mongo/repoActivity.ts 97.00% 0 Missing and 3 partials ⚠️
src/db/repoActivity.ts 98.27% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1801      +/-   ##
==========================================
+ Coverage   88.01%   88.87%   +0.86%     
==========================================
  Files         122      127       +5     
  Lines        6857     7092     +235     
  Branches     1304     1318      +14     
==========================================
+ Hits         6035     6303     +268     
+ Misses        510      493      -17     
+ Partials      312      296      -16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
@kriswest kriswest mentioned this pull request Oct 20, 2026
14 of 21 tasks
Refresh against finos/main while retaining the repository activity summaries
and current push diff/list behavior. Add shared sink regression coverage for
detail payload preservation and list projections across review actions and
activity rebuilds.

Signed-off-by: Denis Ćorić <denis85@gmail.com>
@dcoric

dcoric commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@kriswest, a bit of context: this PR covers the repository activity optimisation from #1691. I split it out of #1795 following Juan’s suggestion to review it separately from the audit-storage changes. #1795 reduces duplicate audit data, while this PR maintains activity summaries so repository-page loads don’t need to recalculate them from the full push history.

@kriswest kriswest added this to the 2.4.0 milestone Nov 2, 2026

@jescalada jescalada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after fixing the documentation 👍🏼

Another thing: I feel the PR description isn't easy for humans to digest - do you mind condensing the PR description into a few sentences? The "validation" section can go in a separate comment so we aren't greeted with a wall of text when revisiting the PR in the future 🙂

Comment thread website/docs/deployment.mdx Resolved
@dcoric

dcoric commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Validation

Moved here from the PR description.

  • 1,698 unit tests passed; 90.86% line coverage.
  • 85 MongoDB and 89 PostgreSQL integration tests passed, including duplicate-ID migration recovery, concurrent readers, stale refresh protection, and shared rebuild behavior.
  • Six real Git push/fetch scenarios and all 18 current Cypress tests passed against PostgreSQL.
  • Manually checked approval/rejection timestamps, full diffs, and immediate count updates after rejection. A real MongoDB startup with duplicate IDs failed before either server listened, preserved both audits, and succeeded after reconciliation.
  • Verified the documented rebuild command using the installed package API.
  • Production build, TypeScript, formatting, and lint passed, with the existing tagPush Cypress warning.
  • Combined with perf: reduce duplicate push audit storage #1795 locally: 729 unit/service/processor tests, 95 MongoDB integration tests, 101 PostgreSQL integration tests, six Git push/fetch scenarios, and 18 Cypress tests passed. The sink adapter overlap has been resolved and verified locally for when the second PR is updated after the first merges.

On a local synthetic PostgreSQL history of 10,000 pushes across 10 repositories, five-run median warm reads were 2.44 ms for summaries versus 11.27 ms for the previous history query. Initial backfill took 178.51 ms. These are workload examples, not general performance guarantees.

Implementation notes

  • File summaries update in memory. MongoDB persists revisions and retries interrupted refreshes; concurrent dashboard reads share one refresh per process, and revision checks stop stale work from other instances overwriting newer summaries.
  • PostgreSQL migration 9 records audit changes transactionally via a trigger and refreshes only the affected summaries.
  • Startup migration 20261013-repository-activity backfills summaries before either server accepts requests. MongoDB checks for duplicate push IDs without deleting audits.
  • Rebuilding is available through the shared database API for all three sinks.

@jescalada jescalada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🚀

@jescalada
jescalada enabled auto-merge November 16, 2026 12:47
@jescalada
jescalada merged commit 6c444cb into finos:main Nov 16, 2026
28 checks passed
@jescalada jescalada linked an issue Nov 17, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complete PostgreSQL push storage and activity parity

3 participants