Writing Parallel Parameterized Tests with Kotlin and JUnit 5

I'm a longtime Go developer, and if there's one thing Go developers love, it's a parameterized (or "table-driven") test.

Table-driven tests allow you to write a single test function that consumes a collection of parameters containing various inputs and their expected outputs. For functions with simple inputs and outputs, this dramatically reduces the boilerplate required to write exhaustive tests. Think of it as the poor man's fuzzing.

Go's no-magic approach makes this easy. Whack your test parameters into a slice and iterate, calling the test function for each one. Better still, the standard library testing package has first-class support for parallelism, so parallelising your tests is trivial.

Lately, I've been getting up to speed with Kotlin, and I found myself wondering how I'd do this with JUnit, the premier testing framework on the JVM. Here's the most effective way I've found, and some pitfalls to avoid.

Scenario

Test the Point Class

Imagine you want to test a class like Point, below. You've overriden the + operator, and are keen to ensure it behaves as expected.

data class Point(val x: Int, val y: Int)

operator fun Point.plus(other: Point): Point {
    return Point(x + other.x, y + other.y)
}

Make the Tests Fast

You know that by avoiding shared mutable state in your tests, you can run them in parallel, slashing your test suite's runtime. You dream of what you could achieve with the cumulative time savings.

Solution

Here's the finished code, with some test cases omitted for brevity. Check the numbered annotations to understand the benefits and trade-offs of each decision.

// build.gradle.kts

// ...
dependencies {
    testImplementation(kotlin("test"))

    val junitVersion = "5.9.2" // your preferred version here
    testImplementation("org.junit.jupiter:junit-jupiter-params:$junitVersion") // for parameterized tests
}
// ...

# src/test/resources/junit-platform.properties

junit.jupiter.testinstance.lifecycle.default = per_class # (1)
junit.jupiter.execution.parallel.enabled = true # (2)

# Read the walkthrough to learn why you __DON'T__ need the following lines.
~~junit.jupiter.execution.parallel.mode.default = concurrent~~
~~junit.jupiter.execution.parallel.mode.classes.default = concurrent~~

// src/test/kotlin/PointTest.kt

import org.junit.jupiter.api.Assertions.*
import org.junit.jupiter.api.Named
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.Nested
import org.junit.jupiter.api.parallel.Execution
import org.junit.jupiter.api.parallel.ExecutionMode
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.MethodSource
import java.util.stream.Stream

@Execution(ExecutionMode.CONCURRENT) // (2)
internal class PointTest {
    @Nested // (3)
    @Execution(ExecutionMode.CONCURRENT)
    inner class Plus { // (3)
        inner class Params(val lhs: Point, val rhs: Point, val expected: Point)

        private fun getParams(): Stream<Named<Params>> { // (1)
            val lhs = Point(1, 1)
            return Stream.of(
                Named.of("RHS is positive", Params(lhs, Point(1, 1), Point(2, 2))),
                Named.of("RHS is negative", Params(lhs, Point(-1, -1), Point(0, 0))),
                Named.of("RHS is zero", Params(lhs, Point(0, 0), lhs))
            )
        }

        @ParameterizedTest // (4)
        @MethodSource("getParams") // (1)
        fun `returns the expected point`(params: Params) {
            assertEquals(params.expected, params.lhs + params.rhs)
        }
    }
}

(1) Test Class Lifecycle

junit.jupiter.testinstance.lifecycle.default = per_class tells JUnit to create a single instance of the test classes PointTest and Plus and run all test methods on these instances.

This is necessary to avoid having to declare MethodSource methods as static, which is a pain in Kotlin. Since Kotlin doesn't support static methods directly, you have to declare all such methods in a companion object and annotate them with @JvmStatic.

This gets ugly. You can only have one companion object per class, which means test cases for all your test methods must be declared in the same companion object, separate from the methods that consume them. Maintenance nightmare.

As a result of using per_class, you MUST avoid shared mutable state in test classes, but this is good practice anyway.

(2) Parallel Execution

junit.jupiter.execution.parallel.enabled = true tells JUnit to run tests in parallel where instructed to with the annotation @Execution(ExecutionMode.CONCURRENT).

Gotcha: the options junit.jupiter.execution.parallel.mode.default and junit.jupiter.execution.parallel.mode.classes.default are incompatible with junit.jupiter.testinstance.lifecycle.default = per_class, and do nothing while this setting is active.

This is for our own safety: test class instances are shared, so we must signal to other developers that our classes are required to be thread-safe with @Execution(ExecutionMode.CONCURRENT).

(3) Nested Class Per Method

@Nested tells JUnit to nest the test class Plus inside PointTest. This is a nice way to group related tests together. The test parameter class, Params, the method source, getParams, and the test method, returns the expected point, are all nested inside Plus.

Note that nested classes must be declared inner class, since JUnit requires instances of nested test classes to hold a reference to their parent instance.

(4) @ParameterizedTest

@ParameterizedTest tells JUnit to run the test method multiple times, consuming the test cases provided by the method source. This is where the magic happens, but steps 1-3 are essential to take us from boilerplate hell to parallel, parameterized heaven.