Repository navigation
Support the JVM annotations on JUnit 5 dynamic tests - #298
Merged
jeanbisutti merged 1 commit intoNov 17, 2026
Merged
jeanbisutti merged 1 commit into
jeanbisutti merged 1 commit into
Conversation
jeanbisutti
force-pushed
the
support-jvm-annotations-on-junit5-dynamic-tests
branch
from
November 17, 2026 13:31
9a92653 to
bba517a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR title
Support the JVM annotations on JUnit 5 dynamic tests
PR description
Fixes #115.
What's new
The QuickPerf annotations that need a specific JVM now work on JUnit 5 dynamic tests. You can put them on a
@TestFactorymethod, on its class, or in your global annotations:@HeapSize,@Xms,@Xmx,@JvmOptions,@UseGC,@EnableGcLogging@MeasureHeapAllocation,@ExpectMaxHeapAllocation,@ExpectNoHeapAllocation@MeasureRSS,@ExpectMaxRSS@ProfileJvm,@ExpectNoJvmIssueUntil now, each dynamic test of such a test factory failed with
RuntimeException: Cannot run a dynamic test on a forked JVM. QuickPerf now executes each dynamic test in its own JVM, as it already does for a test method, and reports the result on each dynamic test. This also works for the dynamic tests of dynamic containers, and for the test factories of@Nestedclasses.QuickPerf now also measures each dynamic test on its own, whatever the QuickPerf annotation: see "Behavior changes for existing tests" below.
Example
QuickPerf starts one JVM per dynamic test.
ArrayList of size 1000000fails, because it allocates more than 1 MB. The two other dynamic tests pass.How it works
For a test factory providing N dynamic tests:
@BeforeEachmethods and the test factory method, as usual.@BeforeAlland@BeforeEachmethods and the test factory method again, then only the dynamic test at the same position, then the@AfterEachand@AfterAllmethods. QuickPerf measures the dynamic test only. The test JVM reports the measures and the failures of the new JVM on the dynamic test.@AfterEachmethods, as usual.So the test factory method and its
@BeforeEachand@AfterEachmethods are executed N + 1 times.Writing a test factory with a JVM annotation
Each new JVM executes the test factory method again, and finds the dynamic test by its position, not by its name: the dynamic test
#2is the second one that the test factory method provides. So:Provide the same dynamic tests, in the same order, in every JVM.
Set.of(...),Map.of(...)List.of(...),Stream.of(...),Arrays.asList(...),LinkedHashMap,TreeMapHashSetor aHashMapof enum constants, or of objects not overridinghashCode()Fruit.values(),EnumSet,LinkedHashSet,TreeSet, or.sorted(...)Class.getDeclaredMethods()orClass.getMethods(), unsorted.sorted(Comparator.comparing(Method::toString)): sorting byMethod::getNameleaves the overloaded methods in any orderFiles.list(dir), a query withoutORDER BYORDER BY@JvmOptions("-Dname=value")@BeforeAllor@BeforeEachmethodA
HashSet<String>or aHashSet<Integer>keeps the same order in every JVM.When the new JVM does not provide the same dynamic tests:
In the specific JVM where QuickPerf executes ..., the display name of the dynamic test is "...", and not "...", followed by the possible causes;JUnit started 0 tests instead of 1 in the specific JVM where QuickPerf executes ..., followed by the possible causes;"dynamic test " + indexover aHashSetfor example);Give each dynamic test a display name that does not change from one JVM to another. QuickPerf compares the display names to detect another order, so the dynamic test fails even if the right dynamic test is executed. Avoid an
Object.toString()that is not overridden, a time, a random value, a@TempDirpath, and a format depending on a system property of the test JVM,user.languagefor example: useString.format(Locale.ROOT, ...).Do not make a dynamic test depend on the previous ones. The new JVM executes one dynamic test only: what the previous dynamic tests did, a warm-up for example, is not there.
Remember that the
@BeforeEachand@AfterEachmethods are executed in the test JVM and in each new JVM.@BeforeEachmethod acquires in the test JVM, a lock, a port or a file for example, is held until the@AfterEachmethods, so while the new JVMs are executed. A@BeforeEachmethod acquiring the same exclusive resource fails in the new JVM, or blocks the build if it waits for it, because the test JVM waits for the new JVM without timeout.MockitoExtension, a stub created by a@BeforeEachmethod fails the test factory in the test JVM with anUnnecessaryStubbingException. In the JVM of a dynamic test, the stubs that only the other dynamic tests use fail the dynamic test. Uselenient().when(...)or@MockitoSettings(strictness = Strictness.LENIENT), and move theverify(...)calls into the dynamic tests.@AfterEachmethod checking that the dynamic tests released their resources for example, fails the dynamic test withJUnit reported a failure of dynamic_tests() after the dynamic test in the specific JVM where QuickPerf executes ....With the JUnit parallel execution, protect the resources that the JVMs of the dynamic tests share. These JVMs can be executed at the same time, and each one executes the
@BeforeAll,@BeforeEach,@AfterEachand@AfterAllmethods. If they use a shared resource, a file of the working directory, a port or a database table for example, annotate the test factory with@Execution(ExecutionMode.SAME_THREAD).Expect one JVM per dynamic test. Starting a JVM takes time, and each one leaves a temporary working folder, as for the other tests executed in a specific JVM.