Repository navigation
Fail a JUnit 5 invocation executed with other arguments in its specific JVM - #300
Merged
jeanbisutti merged 1 commit intoNov 17, 2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR title
Fail a JUnit 5 invocation executed with other arguments in its specific JVM
PR description
What's new
Since #288, QuickPerf executes each invocation of a
@ParameterizedTest, a@RepeatedTestor a@ParameterizedClassin its own JVM, when the test has an annotation needing a specific JVM:@HeapSize,@JvmOptions,@MeasureHeapAllocation,@ExpectMaxHeapAllocation,@ProfileJvm... The new JVM calls the arguments source again, and executes the invocation at the same position.Until now, when the arguments source provided other arguments in the new JVM, or the same arguments in another order, QuickPerf reported the result and the measures of another invocation, without any message: a test could pass or fail because of another argument, and some arguments were never tested.
An invocation of a
@ParameterizedTest, or a test of an invocation of a@ParameterizedClass, now fails with an explicit message in this case, as #298 does for the dynamic tests. QuickPerf detects it by comparing the display names of the invocation in both JVMs: so an invocation whose display name changes from one JVM to another now fails too. See "Writing a parameterized test with a JVM annotation" below.Example
Set.of(1, 3)iterates as1, 3in some JVM processes, and as3, 1in others. When it iterates as1, 3in the test JVM, and as3, 1in the new JVMs:#113allocating 1 MBfails: 3 MB measured#231allocating 3 MBpasses: 1 MB measuredallocating 1 MBfails with:For a test of a
@ParameterizedClass, the message gives the display names of the class invocation, for example[1] 2and[1] 1.How it works
@ParameterizedTestor of a@ParameterizedClassof the test and of its ancestors. By default, JUnit builds them from the arguments, for example[2] b.junit.jupiter.displayname.generator.default,junit.jupiter.params.displayname.defaultandjunit.jupiter.params.displayname.argument.maxlength. Given by Maven Surefire, Gradle, a system property or the@ConfigurationParameterof a@Suite, they did not reach the new JVM: the invocations had other display names there, andTestInfo.getDisplayName()returned another value than in the test JVM.The invocations of a
@RepeatedTest, or of a customTestTemplateInvocationContextProviderorClassTemplateInvocationContextProvider, are not checked: the first ones have no arguments, and a custom provider can name its invocations with a counter, which the new JVM does not increment for the invocations that it does not create.Writing a parameterized test with a JVM annotation
As for a test factory (#298), the new JVM finds the invocation by its position, not by its arguments: the invocation
#2is the one having the second arguments of the arguments source. So:Provide the same arguments, in the same order, in every JVM. The advice of Support the JVM annotations on JUnit 5 dynamic tests #298 for the dynamic tests applies to the arguments: avoid
Set.of(...),Map.of(...), aHashSetor aHashMapof enum constants or of objects not overridinghashCode(), the unsorted methods ofClass.getDeclaredMethods()orClass.getMethods(), random values, the current time,Files.list(dir), a query withoutORDER BY, a system property set in the test JVM only, and a static state filled by the previous tests.Give each invocation a display name that does not change from one JVM to another. QuickPerf compares the display names to detect other arguments, so the invocation fails even if it is executed with the right arguments. By default, the display name contains the
toString()of the arguments: avoid anObject.toString()that is not overridden, of an object, a lambda or a mock for example, a time, a random value, a@TempDirpath, a set or a map whose order changes from one JVM to another, and a format depending on a system property of the test JVM,user.languageoruser.timezonefor example. Name the invocation with@ParameterizedTest(name = "..."),@ParameterizedClass(name = "...")orNamed.of("...", argument)(JUnit 5.8+), or set the system property with@JvmOptions("-Dname=value").Let the display name tell the arguments apart. QuickPerf cannot see other arguments when the display name does not contain them, for example with
@ParameterizedTest(name = "{index}"), or when they differ only after the place where JUnit 5.10+ cuts a long argument. The new JVM then executes the invocation at the same position without any message, as before this PR.