“Until the contract is signed, nothing is real.”
Glenn Danzig
Anyone who has ever worked with Web APIs, and who hasn’t, will be aware of the difficulties of testing in an API environment. When testing an API, there are two sides to consider, that of the producer and that of the consumer. If both do not work hand in hand, then problems are inevitable. Consumer contract tests are designed to avoid some of the difficulties in API testing.
In testing, functional tests, integration tests, regression tests, load tests, stress tests and security tests are a set that is hopefully considered in its entirety. Nothing is more annoying than an API that fails in one of these categories.
Consumer contract tests are a special group of regression tests and therefore nothing new. What makes them interesting, however, is the perspective they provide on the API.
A server team developing a new API produces a large list of tests that test all aspects of the API. These tests are used to ensure the functionality and quality of the API.
A client team in turn uses the API description to develop a client that accesses the API. Unit tests and integration tests are used to ensure the functionality of the client.
Despite all the tests, a large test gap opens up here if both teams do not have a common test infrastructure. If a modified API implementation is published, the client may experience problems if the behavior of the API has changed intentionally or unintentionally.
The behavior of an API should not change because the description of the API defines a contract between the producer and its consumers. This is why this is also referred to as the producer contract. And this contract must be strictly adhered to within a version of an API.
If the behavior of an API changes despite this, there can usually only be two reasons for this. The producer has a test gap and the change has not been registered or the consumer has an incorrect assumption about the API contract.
So why not formulate the consumer’s assumption about the API in a separate contract, the consumer contract and verify it against the API? The special thing about such a consumer contract is that it can be different for each consumer and usually only covers a fraction of the entire API. And all the tests that validate this consumer contract are the consumer contract tests.
So far, the topic of consumer contract testing has not provided much information and some of the examples in the wild do not really help. A consumer contract is a part of the existing producer contract and the tests are a subset of the regression tests.
I recently saw an example that used an OpenAPI description to execute a test and check the postcondition in the response. While this is certainly also possible as part of an automated consumer contract test, it is actually a simple producer contract test that checks the API in the form of a functional test or a regression test.
To better understand the area of consumer contract testing, we need to look at what false assumptions a consumer may have about an API.
Two known sources are lack of knowledge about the valid parameter combinations and wrong assumptions about the state of the service after a call.
With some APIs, for example the Keycloak API, the developer has to guess which parameters are passed in a request, or worse, what effects they may have. With several optional parameters, it is also not always clear in which combinations these parameters can be used. The ideas of producers and consumers often diverge widely at such points. API developers should, as far as possible, simply avoid optional values and make the behavior explicit. Parameters that are not used at all should be avoided altogether. What is not described can change and the consumer will have trouble.
The OpenAPI test mentioned above was a producer contract test because it checks formulated pre- and post-conditions. What it did not check was what change the tested system underwent and when. Such changes can often only be detected in the course of further communication via the API. Multi-stage communication, which is implicitly established via the producer contract and the use case, is referred to as a conversation.
When ordering a fiber optic connection, for example, the system acknowledges the request directly with a positive response, but access to the ONT number of the connection is still not always possible. Especially if the fiber optic expansion has not yet begun. Incorrect assumptions can therefore lead to incorrect use of the API. With such systems, there are usually asynchronous notifications about the progress of processing. Correct use of the API would therefore be to first send the order, wait for the necessary asynchronous notification from the service and then request the ONT number. This conversation between producer and consumer can be validated via a test.
In order to formulate suitable consumer contract tests, the consumer and producer must work together in some way, as the tests must be formulated from the consumer’s perspective and executed on the producer’s API. This sounds like a lot of work for the developers of the API for little value. The existing tests already attest to the correctness of the API.
The consumer contract tests provide the developers of the API with interesting additional information. All consumer contracts of the API form the consumer-driven contract of the API. Like the consumer contracts, this is generally only a subset of the provider contract. This means that the service only generates its value via this part of the API. All other components of the API are either not used at all or only used internally. In the further evolution of the API, the teams can focus their capacities on the API parts from the consumer-driven contract, additionally optimizing the existing conversations and removing other parts from the API altogether.