One question I get asked a lot is how you can automatically rerun your test on failure. This is a typical case for heavy, functional test scenarios, which are often flaky. While test flakiness and its management is crucial and extensive matter itself, in this post I want to give a shout to the extremely simple yet useful library: Spock-Retry. It introduce possibility to create retry policies for Spock tests, without any additional custom-rules implementation – just one annotation.
If you are not a fan of Spock testing framework and you prefer JUnit – stay tuned! I will post analogous bit about rerunning JUnit tests soon.
Instalation
If you are an maven user, add following dependency:
<dependency>
<groupId>com.anotherchrisberry</groupId>
<artifactId>spock-retry</artifactId>
<version>0.6.2</version>
<type>pom</type>
</dependency>
For gradle users:
compile 'com.anotherchrisberry:spock-retry:0.6.2'
Note that Spock-Retry is available from Bintray so you may have to add jcenter repository to your build definition.
Usage
As I’ve already said, library is pretty straightforward, as well as the usage is. Lets have a look at basic example, where we have a single Spock’s test case that occasionally may fail with false-positive result. We want to rerun it once on every failure:
What’s important here is the @RetryOnFailure annotation. It tells that If your test fails, it’ll be rerun once. You can specify the number of retry attempts by adding the times parameter. What’s more, you can define a delay time between one retry and another:
Spock-Retry supports also annotating tests on test class level. If you do so, only failed tests will be rerun, not a whole test class. You can also combine @RetryOnFailure annotation and Spock’s @Stepwise – in this case also only failed steps will be rerun.
If you want to annotate your whole test suite with @RetryOnFailure, you don’t have to put in in every class individually. In most cases Spock’s Specification class is extended with custom specifications. You can annotate your custom specification class, and RetryOnFailure will work in every test class that extends it:
If you have whole test suite were 1 rerun is just fine, but there is this one test that’s more flaky than others, you can also annotate whole suite in custom specification class, and then additionally put retry annotation with different configuration on this problematic test class.
Summary
Flaky tests are nothing new. The problem has been around for a while and there’re no simple solution – automatic reruns isn’t one. Although having longer test-run times (due to reruns) is still better than false-positive tests.
