“Mutation is random; natural selection is the very opposite of random.”
RICHARD DAWKINS
When I first heard about mutation testing, I was – to put it kindly – very confused. The idea that we change our code to see which tests fail doesn’t look particularly promising at first glance. But if you take a closer look at the approach, you realize the potential it holds.
Mutation testing is a technique used in software development to evaluate the quality of test cases by making small changes, known as mutations, to the source code. These mutations are designed to simulate errors or bugs in the code, such as changing operators or altering conditional statements. After these mutations have been introduced, the test run is restarted to check whether some of the tests do not recognize the changes. If a test case continues to succeed after a mutation, this indicates that the test case is not effective in detecting potential errors in the code.
The goal of mutation testing is to measure the effectiveness of the test suite in detecting bugs in the code. By introducing mutations and analyzing the behavior of the test cases, developers can identify weaknesses in their testing approach and improve the quality of their tests. This process helps to ensure that the test suite is robust and reliable to detect potential errors, resulting in a more reliable and high-quality software product.
There can be many reasons for weaknesses in your test suite. Edge cases were not considered, additions were not validated with additional tests and, of course, tests for pure code coverage without any checks.
If you want to start with mutation testing in Java, you first need a code base and the corresponding unit tests. You can’t do it without the unit tests, but then there is already an obvious problem in the project. Because nobody wants to change their own code base manually in order to run unit tests on it, we use the PIT framework. The framework provides a Maven plugin, which greatly simplifies integration into your own build pipeline.
Mutation tests always take many times longer than unit tests. This is in the nature of things, because the existing unit tests are called many times for the mutations. However, mutation tests are not used for every build because they are not used for quality assurance of the application, but for quality assurance of the test suite.
If you are wondering how the Java sources are modified in PIT. In fact, it is not the sources that are modified, but the compiled bytecode. This is also considerably faster because the sources do not have to be recompiled after each mutation. Some standard mutations are used without special configuration. These include changing the return value, reversing conditions and inverting the sign.
I use the FreshMarker project as the test object for our mutation tests. First, the pom.xml receives the following additional plugin.
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.15.8</version>
<dependencies>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
<configuration>
<targetClasses>
<class>org.freshmarker.core.*</class>
</targetClasses>
<targetTests>
<test>org.freshmarker.core.*</test>
</targetTests>
</configuration>
</plugin>
It is interesting to note that pitest-junit5-plugin is required as an additional dependency. By now you should be able to expect that JUnit 5 is used as standard and not any earlier versions. Additionally, in the configuration, the analysis space has been reduced to the packages below org.freshmarker.core. Due to the long runtimes, care should always be taken to minimize the number of visited classes and tests.
After a test run with mvn test-compile org.pitest:pitest-maven:mutationCoverage, the results are displayed in the form of a report.
The test run with 976 mutations takes about 2.5 minutes for the entire project. If all mutations are activated, the test run takes 7.5 minutes with a total of 4097 mutations. For the package org.freshmarker.core.providers, the following report shows up after the 2.5 minutes run.
The mutation coverage shows that 13 mutations were created and 12 of them did not pass the tests. One mutation in the BeanTemplateObjectProvider class was not recognized. The detailed analysis for this class then also shows the problem.
The provide method checks whether the class of the parameter o may be used as a bean. However, there is no test that checks whether the call to ModelSecurityGateway#check throws an UnsupportedDataTypeException. At this point, the test suite can be improved by inserting an additional test.
@Test
void illegalBeanAccess() throws ParseException {
Template template = configuration.getTemplate("test", "${bean}");
Map<String, Object> data = Map.of("bean", UUID.randomUUID());
ProcessException processException = assertThrows(ProcessException.class, () -> template.process(data));
assertEquals("unsupported system class: class java.util.UUID at test:1:1 '${bean}'", processException.getMessage());
}
The illegalBeanAccess test inserts a UUID from the java.util package into the model. This class is not one of the classes supported by default and is also located in a package that is prohibited by default. If you want to use the UUID class, you can meanwhile add it as a custom string type with configuration.registerSimpleMapping(UUID.class).
The new unit test ensures that none of the 13 mutations in the package survive and our test suite has become a little more stable again.