When ever we do test execution, it is also require to understand the out put of the execution. Whether it is Manual execution or an Automated, the output of the same has to be in format, which immediately depicts the overall results of the execution. Hence, our framework also should have the same capability to create output or generate test execution reports.

It is essential to know, how better we can generate our Cucumber test reports. As we know that Cucumber is a BDD framework, it does not have a fancy reporting mechanism. In order to achieve this, Cucumber itself has provided a nice feature to generate reports. These are very basic reports, but using the output of these reports anybody can build more detailed HTML reports, which is covered in the next chapter of Selenium Cucumber Framework series.

Also in this chapter, we will be working with Cucumber Options a lot, it is suggested to go through one of our cucumber tutorials on Cucumber Configurations / Cucumber Options.

Cucumber Reports

Now when we understand the importance of Cucumber Reports, let's learn to generate it as well.

When we execute Cucumber Scenarios, it automatically generates an output in the eclipse console. There is a default behavior associated with that output and we can also configure that output as per our needs also. So how do we modify the default behavior, let's see this now.

Pretty Report

The first plugin, we will talk about is Pretty.  This provides more verbose output. To implement this, just specify plugin = "pretty" in CucumberOptions. This is what the code looks like:

@CucumberOptions( plugin = { "pretty" } )

Complete TestRunner would look like this:

TestRunner.java

package runners;

import org.junit.runner.RunWith;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(
		features = "src/test/resources/functionalTests",
		glue= {"stepDefinitions"},
		plugin = { "pretty" },
		monochrome = true
		)
 
public class TestRunner {

}

Eclipse Console Output

Note: I am sure you are having difficulty in reading the above report, as it is not very well-formatted. It has some special charters in it. But there is a way to filter this output in readable one which is Monochrome.

Monochrome Mode Reporting

If the monochrome option is set to false, then the console output is not as readable as it should be. The output when the monochrome option is set to false is shown in the above example. It is just because, if the monochrome is not defined in Cucumber Options, it takes it as false by default. How to specify it:

@CucumberOptions( monochrome = true );

Full CucumberOption code will be like this:

Usage Report

If we are more concerned about the time taken by each Step Definition, then we should use the usage plugin. This is how we specify the same in @CucumberOptions:

@CucumberOptions( plugin = { "usage" })

Full Code

@CucumberOptions(
	features = "src/test/resources/functionalTests",
	glue= {"stepDefinitions"},
	plugin = { "usage" },
	monochrome = true
)

This is what the output looks like:

Note: This sorts the Step Definitions by their average execution time. The output from the usage plugin is useful for quickly finding slow parts in your code but it is also a great way to get an overview of your Step Definitions.

Cucumber Report Output

You must be wondering that all we have seen above is actually good for a test or for a couple of tests. But if we run a full test suite, this report is not much useful in that case. On top of that, it is difficult to keep these console output safe for future use.

Cucumber gives us the capability to generate reports as well in the form of HTML, XML, JSON & TXT. Cucumber frameworks generate very good and detailed reports, which can be shared with all stakeholders. There are multiple options available for reports which can be used depending on the requirement.

Cucumber HTML Reports

For HTML reports, add html:target/cucumber-reports to the @CucumberOptions plugin option.

@CucumberOptions(
	features = "src/test/resources/functionalTests",
	glue= {"stepDefinitions"},
	plugin = { "pretty", "html:target/cucumber-reports" },
	monochrome = true
)

Note: We have specified the path of the Cucumber report, which we want it to generate it under the target folder.

This will generate an HTML report at the location mentioned in the formatter itself.

Report Output Location

HTML Report Output

Cucumber JSON Report

For JSON reports, add json:target/cucumber-reports/Cucumber.json to the @CucumberOptions plugin option.

@CucumberOptions(
	features = "src/test/resources/functionalTests",
	glue= {"stepDefinitions"},
	plugin = { "pretty", "json:target/cucumber-reports/Cucumber.json" },
	monochrome = true
)

Note : This report contains all the information from the gherkin source in the JSON format. This report is meant to be post processed into another visual format by third-party tools, such as Cucumber Jenkins.

Report Output Location

JSON Report Output

Cucumber JUNIT XML Report

For JUNIT reports, add junit:targe/cucumber-reports/Cucumber.xml to the @CucumberOptions plugin option.

@CucumberOptions(
	features = "src/test/resources/functionalTests",
	glue= {"stepDefinitions"},
	plugin = { "pretty", "junit:target/cucumber-reports/Cucumber.xml" },
	monochrome = true
)

Note : This report generates XML files just like Apache Ant's junit report task. This XML format is understood by most continuous integration servers, who will use it to generate visual reports.

Report Output Location

XML Report Output

All Reports Together

We can even generate all reports together as well.

@CucumberOptions(
	features = "src/test/resources/functionalTests",
	glue= {"stepDefinitions"},
	plugin = { "pretty", "json:target/cucumber-reports/Cucumber.json",
			"junit:target/cucumber-reports/Cucumber.xml",
			"html:target/cucumber-reports"},
	monochrome = true
)