TestNG is a wonderful Test framework. It provides lot of features that can help us make robust and maintainable frameworks. In this chapter we will learn How to Retry Failed Tests in TestNG.
You must have seen random failure during an automated test run. These failures might not necessarily be because of product bugs. These failure can be because of following reasons
- Random browser issues or browser becoming unresponsive
- Random machine issues
- Server issues like unexpected delay in the response from server
These failure are genuine and should be investigated but these failures are not necessarily because of a bug in the product. TestNG provides a wonderful feature using which you can retry a test case multiple times before declaring it as Failed. What it means is that, if you see a failure can just automatically rerun the test to make sure that test is consistently failing. This way it reduces false failures because of random issues and you spend more time debugging true failures.
In this chapter let us study how we can be implement retry on failure mechanism in TestNG. In order achieve this we have to first understand the org.testng.IRetryAnalyzer interface. The interface definition is
public interface IRetryAnalyzer {
/**
* Returns true if the test method has to be retried, false otherwise.
*
* @param result The result of the test method that just ran.
* @return true if the test method has to be retried, false otherwise.
*/
public boolean retry(ITestResult result);
}
This interface has only one method public boolean retry(ITestResult result);
This method will be called once a test method fails. You can get the details of the test from estResult input argument to this method, as shown in the method definition above. This method implementation should return true if you want to re-execute your failed test and false if you don't want to re-execute your test. Usually the implementation of this interface decides on how many times to retry a failed tests based on a fixed counter or a complex logic based on your requirements. A simple implementation of this interface looks like this
package Tests;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {
int counter = 0;
int retryLimit = 4;
/*
* (non-Javadoc)
* @see org.testng.IRetryAnalyzer#retry(org.testng.ITestResult)
*
* This method decides how many times a test needs to be rerun.
* TestNg will call this method every time a test fails. So we
* can put some code in here to decide when to rerun the test.
*
* Note: This method will return true if a tests needs to be retried
* and false it not.
*
*/
@Override
public boolean retry(ITestResult result) {
if(counter < retryLimit)
{
counter++;
return true;
}
return false;
}
}
We now have a simple implementation of IRetryAnalyzer. Taking a closer look at the retry method implementation, this method make sure that a failed test is retried 4 times. This is because we have specified retryLimit = 4;
Lets see how we can use it, there are two ways to include retry analyser in your tests
- By specifying retryAnalyzer value in the
@Testannotation - By adding Retry analyser during run time by implementing on the of the Listener interfaces
Specifying retryAnalyzer attribute in the @Test annotation
We can do this by simply using following syntax to @Test(retryAnalyzer="IRetryAnalyzer Implementing class"). Below is the code to do that
import org.testng.Assert;
import org.testng.annotations.Test;
public class Test001 {
@Test(retryAnalyzer = Tests.RetryAnalyzer.class)
public void Test1()
{
Assert.assertEquals(false, true);
}
@Test
public void Test2()
{
Assert.assertEquals(false, true);
}
}
Here we have two tests and one test is specifying which class to use as retry analyser. If we run this test we will get following results
You can see that Test1 was run 4 times and its been marked failed only at the last run. It was run 4 times because of the retry analyser. However, looking at Test2 we can see that it is run only once. This is obvious because we never specified a retryAnalyzer for Test2.
Lets look at the second way of adding retryAnalyzer to your test.
Specifying retryAnalyzer during runtime
In this case you would need to implement ITestAnnotationTransformer interface. ITestAnnotationTransformer interface falls under a broad category of interfaces called TestNG Listeners. You can read about them more here. This interface definition looks like this
public class AnnotationTransformer implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation,
Class testClass,
Constructor testConstructor,
Method testMethod) {
// TODO Auto-generated method stub
}
}
This interface is used to programatically add annotation to your test methods during run time. Transform method is called for every test during test run. A simple implementation of this interface can help us set the retry analyser like this
package Tests;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
public class AnnotationTransformer implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation,
Class testClass,
Constructor testConstructor,
Method testMethod) {
annotation.setRetryAnalyzer(RetryAnalyzer.class);
}
}
Once we have the implementation of IAnnotationTransformer, we just need to add it as a listener in the testng run xml. Like this
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="RetryFailedTests" verbose="1" >
<listeners>
<listener class-name="Tests.AnnotationTransformer"/>
</listeners>
<test name="RetryMulitple">
<classes>
<class name="Tests.Test001"/>
</classes>
</test>
</suite>
Now we don't have to specify the retryAnalyzer attribute in the @Test annotation. Updated tests will look like this
package Tests;
import org.testng.Assert;
import org.testng.annotations.Test;
public class Test001 {
@Test
public void Test1()
{
Assert.assertEquals(false, true);
}
@Test
public void Test2()
{
Assert.assertEquals(false, true);
}
}
You can now simply run the tests and see how both Test1 and Test2 are re-executed 4 times each by TestNG. This is pretty much it on this topic. In the next section we will see how we can use a custom Java Annotation to specify the retry limit count in the test directly.
