“Rather than apply minutes of suspect reasoning, we can just ask the computer by making the change and running the tests.”
Kent Beck
The article Bug Pattern in Eigenbau showed how easily the Error Prone catalogue can be expanded with your own Bug Patterns. However, it failed to mention what unit tests for such Bug Patterns look like.
There are two different types of Error Prone tests. The first variant tests whether the Bug Pattern actually recognises a bug and the second variant tests the correct replacement of the bug.
The ExceptionAsSurprise bug pattern from the Error Prone Bug Patterns project is used as an example to illustrate some of the possibilities. The ExceptionAsSurprise Bug Pattern generates a compiler warning if the else branch of an if statement only contains a throw statement.
if (i == 42) {
// a lot of code
// ...
// ...
// ...
// ...
} else {
throw new IllegalArgumentException();
}
In this case, the source is more readable if the if statement is inverted and the redundant else is removed.
if (i != 42) {
throw new IllegalArgumentException();
}
// a lot of code
// ...
// ...
// ...
// ...
In the corrected version, the if statement serves as a guard clause for the following code.
Error Prone provides some helper classes to write unit tests. This classes work with JUnit 4 and JUnit 5 and other frameworks. A CompilationTestHelper is required to check whether a Bug Pattern has been recognised by Error Prone.
private final CompilationTestHelper compilationHelper = CompilationTestHelper.newInstance(ExceptionAsSurprise.class, getClass());
With this CompilationTestHelper the simplest Error Prone unit test can now be formulated. The noBugPattern test checks that this Bug Pattern is not recognized in the source.
@Test
void noBugPattern() {
compilationHelper
.addSourceLines("Test.java", """
public class Test {
public void test(int i) {
if (i != 42) {
throw new IllegalArgumentException();
} else {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
}
}
}
""")
.doTest();
}
With the addSourceLines, the source to be tested is passed to the CompilationTestHelper in the form of a text block. When doTest is called, the source is compiled and the Bug Pattern is checked. Since the IllegalArgumentException is not thrown in the else branch in this test, the Bug Pattern cannot take effect and the test is successful.
Another test in which no Bug Pattern should be detected is an else branch with more than one line that ends with a throw statement.
@Test
void multipleLinesBugPattern() {
compilationHelper
.addSourceLines("Test.java", """
public class Test {
public void test(int i) {
if (i == 42) {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
} else {
System.out.println("Not the Answer to the Ultimate Question of Life, the Universe, and Everything");
throw new IllegalArgumentException();
}
}
}
""")
.doTest();
}
The next test checks the detection of the Bug Pattern. The throw statement is now in the else branch. To ensure that the association between compiler message and error is clear, a marker must be inserted into the source. The comment line // BUG: Diagnostic matches: X is used for this purpose.
private final CompilationTestHelper compilationHelper =
CompilationTestHelper.newInstance(ExceptionAsSurprise.class, getClass());
@Test
void singleLineBugPattern() {
compilationHelper
.addSourceLines("Test.java", """
public class Test {
public void test(int i) {
// BUG: Diagnostic matches: X
if (i == 42) {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
} else {
throw new IllegalArgumentException();
}
}
}
""")
.expectErrorMessage("X", Predicates.containsPattern("Throwing an Exception in the else statement is much harder to read"))
.doTest();
}
This test expects an error and checks it with the expectErrorMessage statement. The specified compiler message must belong to the line following the marker with key X.
Error Prone is able to correct detected Bug Patterns. To do this, the implementation of the respective Bug Pattern must be able to generate a replacement. A replacement can be checked using the BugCheckerRefactoringTestHelper. It is instantiated like the CompilationTestHelper.
private final BugCheckerRefactoringTestHelper testHelper = BugCheckerRefactoringTestHelper.newInstance(ExceptionAsSurprise.class, getClass());
The BugCheckerRefactoringTestHelper receives the sources to be checked using the addInputLines method and these are checked in the test against the expected output in addOutputLines.
@Test
void singleLineBugPatternRefactoring() {
testHelper
.addInputLines("Test.java", """
public class Test {
public void test(int i) {
if (i == 42) {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
} else {
throw new IllegalArgumentException();
}
}
}
""")
.addOutputLines("Test.java", """
public class Test {
public void test(int i) {
if (i != 42) {
throw new IllegalArgumentException();
}
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
}
}
""")
.doTest();
}
The singleLineBugPatternRefactoring test checks the correct replacement of the Bug Pattern found. In this test, the doTest method does not check the text of the source, but rather the AST generated from it. This is flattering for the ExceptionAsSurprise Bug Pattern because the replacement doesn’t look that nice.
Error Prone offers the possibility to configure Bug Patterns via the command lines. This is also testable and is presented with a small extension of the ExceptionAsSurprise Bug Pattern. So far, only else branches that consisted of a single throw statement were replaced.
It is now possible for else branches to be recognized by the Bug Pattern with up to three lines by default. The number of lines is also configurable using the ExceptionAsSurprise:Lines command line flag.
public ExceptionAsSurprise(ErrorProneFlags flags) {
maximumThrowBlockLength = flags.getInteger("ExceptionAsSurprise:Lines").orElse(3);
}
public ExceptionAsSurprise() {
maximumThrowBlockLength = 3;
}
The default constructor is replaced by the two constructors above. The parameterless constructor is required by the ServiceLocator mechanism, the other one receives the command line flags passed as an instance of the ErrorProneFlags class. The length of the else branch is checked elsewhere using the maximumThrowBlockLength attribute.
In order to check the implementation, the multipleLinesBugPattern test, which is now failing, is corrected. Since the else branch consists of two lines, we now generate a compiler message.
@Test
void multipleLinesBugPattern() {
compilationHelper
.addSourceLines("Test.java", """
public class Test {
public void test(int i) {
// BUG: Diagnostic matches: X
if (i == 42) {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
} else {
System.out.println("Not the Answer to the Ultimate Question of Life, the Universe, and Everything");
throw new IllegalArgumentException();
}
}
}
""")
.expectErrorMessage("X", Predicates.containsPattern("Throwing an Exception in the else statement is much harder to read"))
.doTest();
}
Another test is necessary to check the configurable number of lines. The multipleLinesExceedingLimitBugPattern test uses the same source as the previous test, but the bug pattern is configured via the line .setArgs("-XepOpt:ExceptionAsSurprise:Lines=1").
@Test
void multipleLinesExceedingLimitBugPattern() {
compilationHelper.addSourceLines("Test.java", """
public class Test {
public void test(int i) {
if (i == 42) {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
} else {
System.out.println("Not the Answer to the Ultimate Question of Life, the Universe, and Everything");
throw new IllegalArgumentException();
}
}
}
""")
.setArgs("-XepOpt:ExceptionAsSurprise:Lines=1")
.expectNoDiagnostics()
.doTest();
}
Since only one line is now allowed in the else branch, as was originally the case, the Bug Pattern is not recognized and the test is successful.
A final, but not necessary, test checks the possibility of suppressing the Bug Pattern detection.
@Test
void suppressSingleLineBugPattern() {
compilationHelper
.addSourceLines("Test.java", """
public class Test {
@SuppressWarnings("ExceptionAsSurprise")
public void test(int i) {
if (i == 42) {
System.out.println("Answer to the Ultimate Question of Life, the Universe, and Everything");
} else {
throw new IllegalArgumentException();
}
}
}
""")
.doTest();
}
A @SuppressWarnings annotation with the name of the Bug Pattern suppresses detection. The test is not necessary because Error Prone is tested here and not the ExceptionAsSurprise Bug Pattern. It’s not necessary, but it’s nice to see everything working.