Skip to content

Support the JVM annotations on JUnit 5 dynamic tests - #298

Merged
jeanbisutti merged 1 commit into
quick-perf:masterfrom
jeanbisutti:support-jvm-annotations-on-junit5-dynamic-tests
Nov 17, 2026
Merged

jeanbisutti merged 1 commit into
quick-perf:masterfrom
jeanbisutti:support-jvm-annotations-on-junit5-dynamic-tests

Conversation

@jeanbisutti

Copy link
Copy Markdown
Collaborator

PR title

Support the JVM annotations on JUnit 5 dynamic tests

PR description

Fixes #115.

What's new

The QuickPerf annotations that need a specific JVM now work on JUnit 5 dynamic tests. You can put them on a @TestFactory method, on its class, or in your global annotations:

  • JVM configuration: @HeapSize, @Xms, @Xmx, @JvmOptions, @UseGC, @EnableGcLogging
  • heap allocation: @MeasureHeapAllocation, @ExpectMaxHeapAllocation, @ExpectNoHeapAllocation
  • resident set size: @MeasureRSS, @ExpectMaxRSS
  • JDK Flight Recorder: @ProfileJvm, @ExpectNoJvmIssue

Until now, each dynamic test of such a test factory failed with RuntimeException: Cannot run a dynamic test on a forked JVM. QuickPerf now executes each dynamic test in its own JVM, as it already does for a test method, and reports the result on each dynamic test. This also works for the dynamic tests of dynamic containers, and for the test factories of @Nested classes.

QuickPerf now also measures each dynamic test on its own, whatever the QuickPerf annotation: see "Behavior changes for existing tests" below.

Example

@QuickPerfTest
class ArrayListTest {

    @ExpectMaxHeapAllocation(value = 1, unit = AllocationUnit.MEGA_BYTE)
    @TestFactory
    Stream<DynamicTest> array_lists() {
        return Stream.of(1_000, 10_000, 1_000_000)
                     .map(size -> dynamicTest("ArrayList of size " + size, () -> new ArrayList<>(size)));
    }

}

QuickPerf starts one JVM per dynamic test. ArrayList of size 1000000 fails, because it allocates more than 1 MB. The two other dynamic tests pass.

How it works

For a test factory providing N dynamic tests:

  1. The test JVM executes the @BeforeEach methods and the test factory method, as usual.
  2. For each dynamic test, QuickPerf starts a new JVM, with the JVM options of the annotations, instead of executing the dynamic test in the test JVM. The new JVM executes the @BeforeAll and @BeforeEach methods and the test factory method again, then only the dynamic test at the same position, then the @AfterEach and @AfterAll methods. QuickPerf measures the dynamic test only. The test JVM reports the measures and the failures of the new JVM on the dynamic test.
  3. The test JVM executes the @AfterEach methods, as usual.

So the test factory method and its @BeforeEach and @AfterEach methods are executed N + 1 times.

Writing a test factory with a JVM annotation

Each new JVM executes the test factory method again, and finds the dynamic test by its position, not by its name: the dynamic test #2 is the second one that the test factory method provides. So:

  1. Provide the same dynamic tests, in the same order, in every JVM.

    Avoid Because Use instead
    Set.of(...), Map.of(...) Java 9+ changes their iteration order at each JVM start List.of(...), Stream.of(...), Arrays.asList(...), LinkedHashMap, TreeMap
    A HashSet or a HashMap of enum constants, or of objects not overriding hashCode() their hash codes change at each JVM start Fruit.values(), EnumSet, LinkedHashSet, TreeSet, or .sorted(...)
    The methods of Class.getDeclaredMethods() or Class.getMethods(), unsorted they come in an order that changes at each JVM start .sorted(Comparator.comparing(Method::toString)): sorting by Method::getName leaves the overloaded methods in any order
    Random values, the current time, Files.list(dir), a query without ORDER BY they change from one JVM to another A fixed seed or date, sorted files, an ORDER BY
    A system property set in the test JVM only the new JVM does not get it @JvmOptions("-Dname=value")
    A static state filled by the previous tests the new JVM starts empty Build it in the test factory method, or in a @BeforeAll or @BeforeEach method

    A HashSet<String> or a HashSet<Integer> keeps the same order in every JVM.

    When the new JVM does not provide the same dynamic tests:

    • another dynamic test at this position: the dynamic test fails with In the specific JVM where QuickPerf executes ..., the display name of the dynamic test is "...", and not "...", followed by the possible causes;
    • no dynamic test at this position: the dynamic test fails with JUnit started 0 tests instead of 1 in the specific JVM where QuickPerf executes ..., followed by the possible causes;
    • another dynamic test having the same display name: QuickPerf cannot see it, and reports the measures of this other dynamic test without any message. This happens when several dynamic tests have the same display name, or when the display name does not depend on what the dynamic test does ("dynamic test " + index over a HashSet for example);
    • additional dynamic tests at the end: they are not executed, so they do no harm.
  2. Give each dynamic test a display name that does not change from one JVM to another. QuickPerf compares the display names to detect another order, so the dynamic test fails even if the right dynamic test is executed. Avoid an Object.toString() that is not overridden, a time, a random value, a @TempDir path, and a format depending on a system property of the test JVM, user.language for example: use String.format(Locale.ROOT, ...).

  3. Do not make a dynamic test depend on the previous ones. The new JVM executes one dynamic test only: what the previous dynamic tests did, a warm-up for example, is not there.

  4. Remember that the @BeforeEach and @AfterEach methods are executed in the test JVM and in each new JVM.

    • What a @BeforeEach method acquires in the test JVM, a lock, a port or a file for example, is held until the @AfterEach methods, so while the new JVMs are executed. A @BeforeEach method acquiring the same exclusive resource fails in the new JVM, or blocks the build if it waits for it, because the test JVM waits for the new JVM without timeout.
    • A check made after the dynamic tests sees none of them in the test JVM, and only one in each new JVM. For example, with the strict stubs of Mockito, the default of MockitoExtension, a stub created by a @BeforeEach method fails the test factory in the test JVM with an UnnecessaryStubbingException. In the JVM of a dynamic test, the stubs that only the other dynamic tests use fail the dynamic test. Use lenient().when(...) or @MockitoSettings(strictness = Strictness.LENIENT), and move the verify(...) calls into the dynamic tests.
    • A failure in the JVM of a dynamic test after the dynamic test, of an @AfterEach method checking that the dynamic tests released their resources for example, fails the dynamic test with JUnit reported a failure of dynamic_tests() after the dynamic test in the specific JVM where QuickPerf executes ....
  5. With the JUnit parallel execution, protect the resources that the JVMs of the dynamic tests share. These JVMs can be executed at the same time, and each one executes the @BeforeAll, @BeforeEach, @AfterEach and @AfterAll methods. If they use a shared resource, a file of the working directory, a port or a database table for example, annotate the test factory with @Execution(ExecutionMode.SAME_THREAD).

  6. Expect one JVM per dynamic test. Starting a JVM takes time, and each one leaves a temporary working folder, as for the other tests executed in a specific JVM.

@jeanbisutti
jeanbisutti force-pushed the support-jvm-annotations-on-junit5-dynamic-tests branch from 9a92653 to bba517a Compare November 17, 2026 13:31
@jeanbisutti
jeanbisutti merged commit 385e763 into quick-perf:master Nov 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support JVM annotations for dynamic tests

1 participant