Property-based testing

3 min. read

Caution! This article is 5 years old. It may be obsolete or show old techniques. It may also still be relevant, and you may find it useful! So it has been marked as deprecated, just in case.

2019 update: It's still difficult to find information about property-based testing, but I found this talk!

With unit tests you usually you do the following cases, from the simplest to the most involved:

  • test for nothing [] -> []
  • test for one [1] -> [1]
  • test for many [1,2] -> [1,2]
  • test the unhappy path (errors, exceptions, explosions...)

This way of testing is very input-based, however, when you have a lot of combinations and you can't come up with examples of inputs, property-based testing is useful.

For this kind of testing you need a framework to generate the inputs. Since this technique is only about 5 years old at the moment of writing this, the tools are not there yet. Haskell does have tools like Quickcheck which are more mature, since they were the ones who started the property-based approach.

With this type of testing, you don't provide the input, you assert on properties. For example:

  • symmetry,
  • size,
  • multiple paths,
  • induction,
  • consistency,
  • idempotence,

It's often better to mix both approaches though. And you can write your own generator if the framework runs out of inputs.

The idea with this approach is to think of the general rules and then choose the simplest to start with.

Property based tests end up being more mathematical sounding, something like this:

it 'output has one more element than input' do
it 'last element of output equals pushed element' do
it 'input has one less element after pop' do
it 'output equals last element of input' do
it 'the order of operations does not matter' do   # --> pop push, push pop

Some tests that can be useful are those where you give them the minimun input to see which breaks the test, which makes your property invalid. These are called shrinker tests (explained here in Erlang). You can also do "arbitraries" with a generator and a shirnker, but you should consider writing your shirnker if you write your generator.

In retrospective, property-based testing may give you:

  • better documentation
  • acceptance tests to test inputs like those from APIs
  • can be exploratory or used as a sanity check for the missing cases

To learn more, there is a nice blog post here: what is property based testing?.