Technical description Maven plugin
Apache Maven is a common build automation tool used primarily for Java projects. In addition, Maven can be used to build projects written in C#, Ruby, Scala and other languages.
Meliora Testlab’s Maven Plugin is an extension to Maven which makes it possible to publish xUnit-compatible testing results and Robot Framework output to your Testlab project.
-
Using the plugin in your build
The plugin binary is available from jcenter repository. You should register this repository to your build to make the plugin available:<repository> <id>jcenter</id> <name>jcenter</name> <url>https://jcenter.bintray.com/</url> </repository>Note: JFrog Bintray has been deprecated as a package repository. We are in the process of moving the release to another repository. In the meantime, you can use a local copy by downloading the latest release of the plugin here.
Then, you should specify the version in your project’s plugin configuration:
<project> ... <build> <!-- To define the plugin version in your parent POM --> <pluginManagement> <plugins> <plugin> <groupId>fi.meliora</groupId> <artifactId>testlab-maven-plugin</artifactId> <version>1.2</version> </plugin> ... </plugins> </pluginManagement> <!-- To use the plugin goals in your POM or parent POM --> <plugins> <plugin> <groupId>fi.meliora</groupId> <artifactId>testlab-maven-plugin</artifactId> <version>1.2</version> </plugin> ... </plugins> </build> ... </project>For more information, see “Guide to Configuring Plug-ins”
-
Goals
Goals available for this plugin:
Goal Description testlab:publish Maven plugin Mojo to publish testing results to Meliora Testlab. -
testlab:publish
A goal to publish xUnit-compatible testing results generated by the build to Testlab as a test run.
Required parameters
Name Type Since Description apiKey String - Api key to use in communicating with the Testlab you want to publish the results to. An api key must be set for security on your Company settings at Testlab side. projectKey String - The key of the Testlab project you want to publish the results to. resources List - File resource patterns to find the xUnit’s .xml files to push the actual results from. <resources> <resource> <directory>target/surefire-reports</directory> <includes> <include>**/*.xml</include> </includes> <excludes> <exclude>**/junk/**</exclude> </excludes> </resource> </resources>Optional parameters
Name Type Since Description addIssueStrategy String 1.2 When to add issues for failing tests. One of: - DONOTADD: Do not add issues
- ADDPERTESTRUN: Add issues per test run: Adds a single issue per test run, and the issue’s description contains details on failed tests
- ADDPERTESTCASE: Add an issue per Testlab test case: Adds an issue for every failed Testlab test case, and the test case’s description contains details on failed tests
- ADDPERRESULT: Add an issue per test result: Adds an issue for every failed result
If true, issues will be added to the Testlab project for failing tests.
Default value is: DONOTADD.
assignIssuesToUser String - When set to Testlab’s user id value for an user who has a role in the set project the added issue(s) are automatically assigned to this user. automationSource String 1.2 Source is used to identify results, so that two test runs with the same source name should contain results for same, or similar tests. companyId String - Company ID of the hosted Testlab you want to publish the results to. If your testlab is accessed from mytestlab.melioratestlab.com your Company ID most likely is “mytestlab”. forcePublish boolean - If set, the results will be always published ignoring the possible -Dtest and -Dit.test parameters.
By default this option is “false”. This means, that if a test or subset of tests are specified by -Dtest or -Dit.test parameters on command line, the (probably partial) results will not be published to Testlab. Set this option to “true” to ignore these properties and always publish the results to Testlab.format String 1.1 Format of the result resources to publish. “junit” for standard xUnit formatted files and “robot” for Robot Framework’s output files. Defaults to “junit”. ignoreFailedTests boolean - If set, the build will not be marked as FAILURE if any failures are detected. Note that this plugin marks the build as failed only by inspecting the results of the resources configured. milestone String - Milestone in Testlab the results and issues are bound to. Milestone must exist in the project. If milestone does not exist the results and issues are added without the milestone binding.
You can enter an identifier of a milestone or a title of a milestone. The identifier is matched first.onpremiseUrl String - For on-premises installation, a full URL address (such as ‘https://testlab.mycompany.com’) of your Testlab must be set. - For on-premise installations on your own server: set onpremiseUrl
- For hosted Testlab: set Company ID
parameters Map - Test cases in Testlab can include parameters in the content as ${PARAM1} formatted tags. Normally, these tags are set and replaced with values set during the test case execution or execution planning phase. Set this configuration map to send the parameter values for test case parameters. <parameters> <UI>Desktop</UI> <BROWSER>Firefox</BROWSER> </parameters>reopenExistingIssues boolean - The plugin will create the issues with the title set to the name of the failing test class. When this setting is set the plugin tries to reopen existing issues in the project by matching the issue’s title.
When finding existing issues if- a single resolved or closed issue is found it is reopened and updated,
- a single still non-resolved issue is found it is updated,
- multiple issues are found and the latest issue is resolved or closed it is reopened and updated,
- multiple issues are found and the latest issue is still non-resolved it is updated.
Otherwise, a new issue is added.
When reopening an issue version, environment, description and assigned to -fields are updated to match the plugin sent values. When updating a found non-resolved issue only description is updated to plugin sent value.resultName String 1.2 If set, this will be used as a name for the results (file) added to Testlab. For example, this can be set as an URL (for example Jenkins job URL) or the name of the result file you are pushing the results from. robotCatenateParentKeywords boolean 1.1 If true, when the XML file containing the results is in Robot Framework format, and in the XML keyword has sub keywords, the sub-keywords are catenated to a single step in the result. For example, if the robot result has <kw name="Open site"> <kw name="Open URL"> <kw name="Navigate browser"> ... </kw> </kw> </kw> ..... the test case is added with a single step described as “Open site – Open URL – Navigate browser”. When concatenating, if a step fails it is always included as a step.
If false, each sub keyword will generate a separate step to the result.
This value defaults to true.tags List - Tags to be set for the Testlab test run.
When updating runs, if the plugin sends even one tag all existing tags in the run will be replaced. Leaving this parameter unset leaves the tags at Testlab side untouched.<tags> <tag>maven</tag> <tag>automated</tag> </tags>testEnvironment String - Test environment in Testlab the results and issues are bound to. If the environment does not exist in Testlab it is added to the project. testRun String - The name of the Testlab test run to create or update.
If a test run with the same name already exists in Testlab it will be always replaced with test results when this plugin is run. If it does not it will be created.testRunDescription String 1.2 A comment to be set for the Testlab test run to create or update.
Hint: You can use Maven’s properties in all the configuration variables and in the testRunDescription field this is especially handy. Use the project.*, settings.* and env.* variables to decorate your configuration as needed. The variables are referenced in typical format such as ${project.version}.testRunUser String - The “created by” user name of the test run created or updated. Defaults to “Maven”. version String - Version in Testlab the results and issues are bound to. If version does not exist in Testlab it is added to the project. Parameter details
When to add issues for failing tests. One of:
- DONOTADD: Do not add issues
- ADDPERTESTRUN: Add issues per test run: Adds a single issue per test run, and the issue’s description contains details on failed tests
- ADDPERTESTCASE: Add an issue per Testlab test case: Adds an issue for every failed Testlab test case, and the test case’s description contains details on failed tests
- ADDPERRESULT: Add an issue per test result: Adds an issue for every failed result
- Type: fi.meliora.testlab.ext.rest.model.TestResult$AddIssueStrategy
- Required: No
- Default: DONOTADD
API key to use in communicating with the Testlab you want to publish the results to. An API key must be set for security on your Company settings at Testlab side.- Type: java.lang.String
- Required: Yes
When set to Testlab’s user id value for an user who has a role in the set project the added issue(s) are automatically assigned to this user.- Type: java.lang.String
- Required: No
Source is used to identify results, so that two test runs with the same source name should contain results for same, or similar tests.- Type: java.lang.String
- Required: No
Company ID of the hosted Testlab you want to publish the results to. If your testlab is accessed from mytestlab.melioratestlab.com your Company ID most likely is “mytestlab”.- Type: java.lang.String
- Required: No
If set, the results will be always published ignoring the possible -Dtest and -Dit.test parameters.
By default this option is “false”. This means, that if a test or subset of tests are specified by -Dtest or -Dit.test parameters on command line, the (probably partial) results will not be published to Testlab. Set this option to “true” to ignore these properties and always publish the results to Testlab.- Type: boolean
- Required: No
Format of the result resources to publish. “junit” for standard xUnit formatted files and “robot” for Robot Framework’s output files. Defaults to “junit”.- Type: java.lang.String
- Required: No
If true, when the xml file containing the results is in Robot Framework format, and in the xml keyword has sub keywords, the sub keywords are catenated to a single step in the result. For example, if the robot result has
<kw name="Open site"> <kw name="Open URL"> <kw name="Navigate browser"> ... </kw> </kw> </kw> ..... the test case is added with a single step described as “Open site – Open URL – Navigate browser”. When concatenating, if a step fails it is always included as a step.
If false, each sub keyword will generate a separate step to the result.
This value defaults to true.- Type: boolean
- Required: No
If set, the build will not be marked as FAILURE if any failures are detected. Note that this plugin marks the build as failed only by inspecting the results of the resources configured.
If checked added issues are merged into a single issue. Checking this option results always in a maximum of a single issue added per plugin execution.- Type: boolean
- Required: No
The milestone in Testlab the results and issues are bound to. Milestone must exist in the project. If milestone does not exist the results and issues are added without the milestone binding.
You can enter an identifier of a milestone or a title of a milestone. The identifier is matched first.- Type: java.lang.String
- Required: No
For on-premises installation, a full URL address (such as ‘https://testlab.mycompany.com’) of your Testlab must be set.
- For on-premise installations on your own server: set onpremiseUrl
- For hosted Testlab: set Company ID
- Type: java.lang.String
- Required: No
Test cases in Testlab can include parameters in the content as ${PARAM1} formatted tags. Normally, these tags are set and replaced with values set during the test case execution or execution planning phase. Set this configuration map to send the parameter values for test case parameters.
<parameters> <UI>Desktop</UI> <BROWSER>Firefox</BROWSER> </parameters>- Type: java.util.Map
- Required: No
The key of the Testlab project you want to publish the results to.- Type: java.lang.String
- Required: Yes
The plugin will create the issues with the title set to the name of the failing test class. When this setting is set the plugin tries to reopen existing issues in the project by matching the issue’s title.
When finding existing issues if- a single resolved or closed issue is found it is reopened and updated,
- a single still non-resolved issue is found it is updated,
- multiple issues are found and the latest issue is resolved or closed it is reopened and updated,
- multiple issues are found and the latest issue is still non-resolved it is updated.
Otherwise, a new issue is added.
When reopening an issue version, environment, description and assigned to -fields are updated to match the plugin sent values. When updating a found non-resolved issue only description is updated to plugin sent value.- Type: boolean
- Required: No
File resource patterns to find the files to push the actual results from. Depending on the format, should be a set of xUnit formatted XML files or for Robot Framework, a single output.xml file.
<resources> <resource> <directory>target/surefire-reports</directory> <includes> <include>**/*.xml</include> </includes> <excludes> <exclude>**/junk/**</exclude> </excludes> </resource> </resources>- Type: java.util.List
- Required: Yes
If set, this will be used as a name for the results (file) added to Testlab. For example, this can be set as an URL (for example Jenkins job URL) or the name of the result file you are pushing the results from.- Type: java.lang.String
- Required: No
Name of the ruleset to apply to these results. Test result rulesets are configured in the “Test automation” UI in Testlab. If not set, a default ruleset for the project is used.- Type: java.lang.String
- Required: No
Tags to be set for the Testlab test run.
When updating runs, if the plugin sends even one tag all existing tags in the run will be replaced. Leaving this parameter unset leaves the tags at Testlab side untouched.<tags> <tag>maven</tag> <tag>automated</tag> </tags>- Type: java.util.List
- Required: No
Test environment in Testlab the results and issues are bound to. If the environment does not exist in Testlab it is added to the project.- Type: java.lang.String
- Required: No
The name of the Testlab test run to create or update.
If a test run with the same name already exists in Testlab it will be always replaced with test results when this plugin is run. If it does not it will be created.- Type: java.lang.String
- Required: Yes
A description to be set for the Testlab test run to create or update.
Hint: You can use Maven’s properties in all the configuration variables and in the testRunDescription field this is especially handy. Use the project.*, settings.* and env.* variables to decorate your configuration as needed. The variables are referenced in a typical format such as ${project.version}.- Type: java.lang.String
- Required: No
The “created by” username of the test run created or updated. Defaults to “Maven”.- Type: java.lang.String
- Required: No
The version in Testlab the results and issues are bound to. If the version does not exist in Testlab it is added to the project.- Type: java.lang.String
- Required: No
-
Usage and examples
The testlab:publish goal provided by this plugin is used to push testing results parsed from xUnit-compatible .xml files to Meliora Testlab as testing results.
Some common points must be taken into account when using this plugin:
- The results are parsed from xUnit-compatible reports (.xml-files) such as generated by the surefire plugin.
- You must configure your project in a way that the failed tests won’t halt the execution as FAILURE. Otherwise, this plugin won’t run. For example, with surefire, you can configure maven-surefire-plugin with
... <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <testFailureIgnore>true</testFailureIgnore> ... </configuration> </plugin> ... </plugins> ... - The goal can be configured with multiple directories and patterns from where the .xml files generated during the build are scanned from. This way, you can publish multiple results with a single invocation.
- The goal should be run after all the goals generating testing results have run. Typically, publish goal in verify phase might do the trick.
Example: complete configuration
A complete surefire-based configuration block for the plugin is as follows:
... <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <!-- ignore failure here to allow other plugins to run --> <testFailureIgnore>true</testFailureIgnore> </configuration> </plugin> <plugin> <groupId>fi.meliora</groupId> <artifactId>testlab-maven-plugin</artifactId> <version>1.2</version> <configuration> <companyId>mycompany</companyId> <apiKey>mytopsecretapikey</apiKey> <!-- or, if you are using Testlab on-premise --> <!-- onpremiseUrl>http://testlab.mycompany.com:8080</onpremiseUrl --> <projectKey>TLABDEMO</projectKey> <ruleset>MyRuleSet</ruleset> <automationSource>maven tests</automationSource> <testRun>Integration tests</testRun> <testRunUser>Maven</testRunUser> <testRunDescription>Integration tests run for 1.0-SNAPSHOT at ${maven.build.timestamp}</testRunDescription> <milestone>M1</milestone> <version>1.0-SNAPSHOT</version> <testEnvironment>system-testing</testEnvironment> <tags> <tag>maven</tag> <tag>automated</tag> <tag>integration</tag> </tags> <parameters> <UI>Desktop</UI> <BROWSER>Firefox</BROWSER> </parameters> <addIssueStrategy>ADDPERTESTRUN</addIssueStrategy> <assignIssuesToUser>agentsmith</assignIssuesToUser> <reopenExistingIssues>true</reopenExistingIssues> <format>junit</format> <resources> <resource> <directory>target/surefire-reports</directory> <includes> <include>**/*.xml</include> </includes> <!-- excludes> <exclude>**/junk/**</exclude> </excludes --> </resource> </resources> <forcePublish>false</forcePublish> <ignoreFailedTests>false</ignoreFailedTests> </configuration> <executions> <execution> <phase>verify</phase> <goals> <goal>publish</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build> ...The above will run all tests with surefire-plugin and publish the results
- to hosted Testlab with company identifier mycompany (mycompany.melioratestlabcom),
- using “mytopsecretapikey” as API key for authentication,
- to Testlab project TLABDEMO,
- by mapping the automated tests with ruleset “MyRuleSet” in the project,
- by setting the source of these tests as “maven tests”,
- to test run titled “Integration tests”,
- by setting the test run’s creator as “Maven”,
- by setting the test run’s comment field as “Integration tests run for [build version] at [timestamp]”,
- by targeting milestone “M1” in Testlab,
- by setting the version field in testrun to match the POM’s version,
- by setting the environment as “system-testing”,
- by tagging the test run with tags “maven”, “automated” and “integration”,
- by setting the test case parameters in Testlab as UI = Desktop and BROWSER = Firefox,
- with adding issues to Testlab for any failed tests as a single merged issue,
- by assigning all the issues to user “agentsmith”,
- by reopening existing matching issues in Testlab if found,
- by merging all testing failures to a single issue,
- by posting junit-formatted (xUnit) result files,
- by looking for the xUnit .xml files from target/surefire-reports path,
- by honoring the -Dtest and -Dit.test properties so that if either one is set the results are not published,
- by marking the build as failed if any failures or errors are present in result files configured and
- running the plugin in publish goal in verify phase.
Example: minimal configuration
A minimal surefire-based configuration block for the plugin is as follows:
... <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <testFailureIgnore>true</testFailureIgnore> </configuration> </plugin> <plugin> <groupId>fi.meliora</groupId> <artifactId>testlab-maven-plugin</artifactId> <version>1.2</version> <configuration> <companyId>mycompany</companyId> <apiKey>mytopsecretapikey</apiKey> <projectKey>TLABDEMO</projectKey> <resources> <resource> <directory>target/surefire-reports</directory> <includes> <include>**/*.xml</include> </includes> </resource> </resources> </configuration> <executions> <execution> <phase>verify</phase> <goals> <goal>publish</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build> ...The above will run all tests with surefire-plugin and publish the results
- to hosted Testlab with company identifier mycompany (mycompany.melioratestlabcom),
- using “mytopsecretapikey” as API key for authentication,
- by mapping the automated tests to Testlab test cases with the default ruleset,
- to Testlab project TLABDEMO,
- by looking for the junit / xUnit .xml files from target/surefire-reports path and
- running the plugin in publish goal in verify phase.
Example: On-premise Testlab
... <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <testFailureIgnore>true</testFailureIgnore> </configuration> </plugin> <plugin> <groupId>fi.meliora</groupId> <artifactId>testlab-maven-plugin</artifactId> <version>1.1</version> <configuration> <onpremiseUrl>http://testlab.mycompany.com:8080</onpremiseUrl> <apiKey>mytopsecretapikey</apiKey> <projectKey>TLABDEMO</projectKey> <resources> <resource> <directory>target/surefire-reports</directory> <includes> <include>**/*.xml</include> </includes> </resource> </resources> </configuration> <executions> <execution> <phase>verify</phase> <goals> <goal>publish</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build> ...The above will run all tests with surefire-plugin and publish the results
- to Testlab running at http://testlab.mycompany.com:8080,
- using “mytopsecretapikey” as API key for authentication,
- by mapping the automated tests to Testlab test cases with the default ruleset,
- to Testlab project TLABDEMO,
- by looking for the xUnit .xml files from target/surefire-reports path and
- running the plugin in publish goal in verify phase.
Example: robot framework publish
A minimal surefire-based configuration block for the plugin to publish Robot Framework’s output is as follows:
... <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <testFailureIgnore>true</testFailureIgnore> </configuration> </plugin> <plugin> <groupId>fi.meliora</groupId> <artifactId>testlab-maven-plugin</artifactId> <version>1.1</version> <configuration> <companyId>mycompany</companyId> <apiKey>mytopsecretapikey</apiKey> <projectKey>TLABDEMO</projectKey> <format>robot</format> <resources> <resource> <directory>target/robot</directory> <includes> <include>output.xml</include> </includes> </resource> </resources> </configuration> <executions> <execution> <phase>verify</phase> <goals> <goal>publish</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build> ...The above will run all tests with surefire-plugin and publish the results
- to hosted Testlab with company identifier mycompany (mycompany.melioratestlabcom),
- using “mytopsecretapikey” as API key for authentication,
- by mapping the automated tests to Testlab test cases with the default ruleset,
- to Testlab project TLABDEMO,
- by looking for Robot’s output.xml file from target/robot path and
- running the plugin in publish goal in verify phase.
-
Frequently asked questions
In short – what exactly does this do?
This plugin provides support for Apache Maven-based builds to publish the automated testing results generated to Testlab.
From where exactly do I set this up with?
In pom.xml of your build. Setting up this plugin probably requires basic knowledge on Maven and it’s concepts.
We have Maven-based jobs in our Jenkins, do I need this plugin?
No. You are probably better off with Testlab’s Jenkins plugin which does the exact same thing but provides a more user-friendly configuration UI in Jenkins.
I’m not getting any results in Testlab?
The testing results in xUnit-compatible files generated are mapped against Testlab’s test cases for results and this happens with the rules defined in your Testlab project’s ruleset. For an in-depth explanation on how this works please see Jenkins plugin’s documentation and the “Test automation” related section in the help manual of Testlab.
It should also be noted that by default if you specify -Dtest or -Dit.test options to pick a subset of tests for execution, the results will not be published. You can set the forcePublish configuration parameter to true to disable this behavior.
Surefire/Failsafe plugin fails my build before this plugin runs?
You must configure your test runner plugin to ignore the failures and let the build continue (for example setting testFailureIgnore to true e). By default, this plugin will also fail the build if any failures or errors are reported by the results published.