Unit Tests: Understanding False Positives and False Negatives

by

“The truth isn’t always beauty, but the hunger for it is.”

― Nadine Gordimer

As a software developer, unit tests are an indispensable part of our daily work. They help us verify the correctness of our code and detect bugs early in the development cycle. However, like any testing system, unit tests can be flawed and lead to incorrect conclusions. In this article, I’ll explain the concepts of False Positives and False Negatives in the context of unit tests and illustrate these with truth tables.

What are False Positives and False Negatives?

A False Positive occurs when a test incorrectly fails even though the tested code is correct. The test claims to find an error where none exists. False Positive also include the so-called flutter tests, which sometimes report an error and sometimes not.

public class Calculator {
    public static int addNumbers(int a, int b) {
        return a + b;
    }
}

@Test
public void testAddNumbers() {
    int result = Calculator.addNumbers(2, 3);

    assertEquals(6, result);
}

The example test is False Positive because it fails even though the addNumbers method is correct. The error lies in the test method, which assumes that 2 + 3 results in the value 6.

A False Negative occurs when a test incorrectly passes even though the tested code contains a bug. The test misses an existing defect.

public class Calculator {
    public static int divideNumbers(int a, int b) {
        return a / b;
    }
}

@Test
public void testDivideNumbers() {
    double result = Calculator.divideNumbers(10, 2);

    assertEquals(5, result);
}

In this example, the test is not sufficient to cover all scenarios of the divideNumbers method. A False Negative exists here because the division by zero generates an ArithmeticException.

Truth Tables for Test Scenarios

Let’s examine the different combinations of code state and test results:

The desired cases are True Positive and True Negative. The tests should all run without errors if there are no errors in the application code and tests should fail if there are errors in the code.

False Positives are a nuisance because the code of the application is correct but errors in the tests cause an alarm. Then unnecessary time has to be wasted to find out what is wrong with the tests. False Positives can lead to developers becoming careless. Then a True Positive is not checked because it is mistaken for a True Negative. False Positives have one advantage: they raise the alarm and can therefore be found.

False Negatives cannot be detected as easily as False Positives. They hide from the developer and use the boundaries between the equivalence classes and cavort at the edges of extreme values. To exclude them, the developer must not only check the best case, but all possible variants.

How to Avoid False Positives and False Negatives

False Positives can be avoid with short, clear tests. The less complexity there is in a test, the fewer errors can occur during the development of a test. Well-chosen assertions and test data are also important so that every test has a clear defined hypothesis.

False negatives can be avoid by consistently testing edge cases and error scenarios. It is also helpful to check the code coverage to identify gaps in the tests. However, this approach does not work so well if there are tests for pure code coverage or if there are many alternative paths through a piece of code. This is where mutation testing, which checks the quality of the test suite, is helpful.

Test Driven Development

A very good method for avoiding False Positives and false negatives from the outset is Test Driven Development (TDD). With TDD, tests are written first and then the implementation is developed until the tests are successful.

The following expanded diagram shows why consistent TDD is so successful.

The developer starts with a first test, which tests a hypothesis against a empty implementation. The test suite created so far is in the True Negative field because the test fails. If the test does not fail, the developer must correct the test. The implementation is then completed until the test is successful. The test suite then moves to the True Positive field. To extend the implementation, new tests are written, all of which fail. The test suite then moves back to the True Negative field. With TDD, the developer never leaves the two fields True Negative and True Positive and False Positive and False Negative do not occur.

Conclusion

Understanding False Positives and False Negatives is crucial for developing reliable test suites. While False Positives primarily impact development velocity by causing unnecessary debugging, False Negatives can lead to serious production failures by allowing bugs to slip through. To consistently avoid False Positives and False Negatives, developers should work with Test Driven Development.

Leave a Comment