Skip to content

ci: Compile benchmarked code before CodSpeed measures it - #315

Merged
emyller merged 1 commit into
mainfrom
ci/codspeed-always-sparkplug
Nov 2, 2026
Merged

emyller merged 1 commit into
mainfrom
ci/codspeed-always-sparkplug

Conversation

@emyller

@emyller emyller commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

CodSpeed's simulation mode runs each benchmark 7 times to warm up, then measures a single call with V8's optimising compilers disabled. V8 compiles a function to baseline (Sparkplug) code once it has run enough times, which can happen inside that measured call. Results then depend on when the compile lands, not on the SDK's code. In #314 this shows up as a regression in small environment::getEnvironmentFlags, even though the call is ~3× faster in steady state.

Changes

  • Run benchmarks with --always-sparkplug, so V8 compiles baseline code on a function's first call, before the measured call.

How did you test this code?

Reproduced CodSpeed's measurement locally with its V8 flags (7 warm-ups, GC, one timed call), 5 runs per branch, for small environment::getEnvironmentFlags:

main #314
CodSpeed's flags ~105µs ~185µs
plus --always-sparkplug ~97µs ~92µs

Also checked that test.execArgv is passed alongside the flags the CodSpeed plugin sets.

🤖 Generated with Claude Code

CodSpeed's simulation measures a single call after a fixed warm-up, so V8 compiling baseline code within that call skewed results.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@emyller
emyller requested a review from a team as a code owner November 2, 2026 15:29
@emyller
emyller requested review from khvn26 and removed request for a team November 2, 2026 15:29
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 6502768a-d890-4e39-80e4-1aa78b416f82
Commits

Reviewing files that changed from the base of the PR and between c386862 and 2dcb801.

Files selected for processing: 1
  • vitest.config.benchmark.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

The benchmark test configuration now passes --always-sparkplug through execArgv.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: Minimal · up to 2dcb8

The change is limited to benchmark execution; no merge-blocking issue is identified. Merge after normal checks pass.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@emyller
emyller merged commit 7b2bd9a into main Nov 2, 2026
6 of 7 checks passed
@emyller
emyller deleted the ci/codspeed-always-sparkplug branch November 2, 2026 15:31
@codspeed

codspeed Bot commented Nov 2, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 15 untouched benchmarks


Comparing ci/codspeed-always-sparkplug (2dcb801) with main (c386862)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants