Skip to content

Fail a JUnit 5 invocation executed with other arguments in its specific JVM - #300

Merged
jeanbisutti merged 1 commit into
quick-perf:masterfrom
jeanbisutti:fail-junit5-invocation-with-other-arguments
Nov 17, 2026
Merged

jeanbisutti merged 1 commit into
quick-perf:masterfrom
jeanbisutti:fail-junit5-invocation-with-other-arguments

Conversation

@jeanbisutti

Copy link
Copy Markdown
Collaborator

PR title

Fail a JUnit 5 invocation executed with other arguments in its specific JVM

PR description

What's new

Since #288, QuickPerf executes each invocation of a @ParameterizedTest, a @RepeatedTest or a @ParameterizedClass in its own JVM, when the test has an annotation needing a specific JVM: @HeapSize, @JvmOptions, @MeasureHeapAllocation, @ExpectMaxHeapAllocation, @ProfileJvm... The new JVM calls the arguments source again, and executes the invocation at the same position.

Until now, when the arguments source provided other arguments in the new JVM, or the same arguments in another order, QuickPerf reported the result and the measures of another invocation, without any message: a test could pass or fail because of another argument, and some arguments were never tested.

An invocation of a @ParameterizedTest, or a test of an invocation of a @ParameterizedClass, now fails with an explicit message in this case, as #298 does for the dynamic tests. QuickPerf detects it by comparing the display names of the invocation in both JVMs: so an invocation whose display name changes from one JVM to another now fails too. See "Writing a parameterized test with a JVM annotation" below.

Example

static Set<Integer> allocations() {
    return Set.of(1, 3);
}

@ExpectMaxHeapAllocation(value = 2.5, unit = AllocationUnit.MEGA_BYTE)
@ParameterizedTest(name = "allocating {0} MB")
@MethodSource("allocations")
public void allocating(int allocationInMegaBytes) {
    byte[] allocatedBytes = new byte[allocationInMegaBytes * 1024 * 1024];
}

Set.of(1, 3) iterates as 1, 3 in some JVM processes, and as 3, 1 in others. When it iterates as 1, 3 in the test JVM, and as 3, 1 in the new JVMs:

Invocation Argument in the test JVM Argument in the new JVM Before this PR With this PR
#1 1 3 allocating 1 MB fails: 3 MB measured Fails: another invocation executed
#2 3 1 allocating 3 MB passes: 1 MB measured Fails: another invocation executed

allocating 1 MB fails with:

In the specific JVM where QuickPerf executes [engine:junit-jupiter]/[class:org.example.AllocationTest]/[test-template:allocating(int)]/[test-template-invocation:#1], the display name of the invocation is "allocating 3 MB", and not "allocating 1 MB": QuickPerf cannot report the measures and the failures of this JVM for "allocating 1 MB".
Possible causes:
- The arguments source provides other arguments, or the same arguments in another order, in the specific JVM. It has to provide the same arguments, in the same order, in every JVM: do not use Set.of(...), Map.of(...), a HashSet or random values. If the arguments come from the methods of a class, sort these methods, for example by Method.toString(): Class.getDeclaredMethods() and Class.getMethods() return them in an order that changes from one JVM to another.
- The display name of the invocation changes from one JVM to another, for example because an argument has an Object.toString() that is not overridden, or contains a time, a random value or a @TempDir path. Give the invocation a display name that does not change, for example with @ParameterizedTest(name = "..."), @ParameterizedClass(name = "...") or Named.of("...", argument).
- The display name of the invocation depends on a system property of the test JVM, user.language or user.timezone for example: the specific JVM does not get the system properties of the test JVM. Give the invocation a display name that does not depend on them, or set the same system property with @JvmOptions("-Dname=value").

For a test of a @ParameterizedClass, the message gives the display names of the class invocation, for example [1] 2 and [1] 1.

How it works

  1. The test JVM saves the display names of the invocations of a @ParameterizedTest or of a @ParameterizedClass of the test and of its ancestors. By default, JUnit builds them from the arguments, for example [2] b.
  2. The new JVM fails the test when one of these invocations has another display name there. QuickPerf does not report the measures and the failures of the new JVM, which may be the ones of another invocation.
  3. The new JVM now gets the configuration parameters of the test JVM that change the display names: junit.jupiter.displayname.generator.default, junit.jupiter.params.displayname.default and junit.jupiter.params.displayname.argument.maxlength. Given by Maven Surefire, Gradle, a system property or the @ConfigurationParameter of a @Suite, they did not reach the new JVM: the invocations had other display names there, and TestInfo.getDisplayName() returned another value than in the test JVM.

The invocations of a @RepeatedTest, or of a custom TestTemplateInvocationContextProvider or ClassTemplateInvocationContextProvider, are not checked: the first ones have no arguments, and a custom provider can name its invocations with a counter, which the new JVM does not increment for the invocations that it does not create.

Writing a parameterized test with a JVM annotation

As for a test factory (#298), the new JVM finds the invocation by its position, not by its arguments: the invocation #2 is the one having the second arguments of the arguments source. So:

  1. Provide the same arguments, in the same order, in every JVM. The advice of Support the JVM annotations on JUnit 5 dynamic tests #298 for the dynamic tests applies to the arguments: avoid Set.of(...), Map.of(...), a HashSet or a HashMap of enum constants or of objects not overriding hashCode(), the unsorted methods of Class.getDeclaredMethods() or Class.getMethods(), random values, the current time, Files.list(dir), a query without ORDER BY, a system property set in the test JVM only, and a static state filled by the previous tests.

  2. Give each invocation a display name that does not change from one JVM to another. QuickPerf compares the display names to detect other arguments, so the invocation fails even if it is executed with the right arguments. By default, the display name contains the toString() of the arguments: avoid an Object.toString() that is not overridden, of an object, a lambda or a mock for example, a time, a random value, a @TempDir path, a set or a map whose order changes from one JVM to another, and a format depending on a system property of the test JVM, user.language or user.timezone for example. Name the invocation with @ParameterizedTest(name = "..."), @ParameterizedClass(name = "...") or Named.of("...", argument) (JUnit 5.8+), or set the system property with @JvmOptions("-Dname=value").

  3. Let the display name tell the arguments apart. QuickPerf cannot see other arguments when the display name does not contain them, for example with @ParameterizedTest(name = "{index}"), or when they differ only after the place where JUnit 5.10+ cuts a long argument. The new JVM then executes the invocation at the same position without any message, as before this PR.

@jeanbisutti
jeanbisutti merged commit 05dbab2 into quick-perf:master Nov 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant