How to parametrize Maven surefire plugin so I can choose which TestNG suites to run

Question

I've got many test suites in TestNG. These are XML files. I want to be able to choose multiple XML suites when running integration-test from maven.

Currently I can add the suite files to pom.xml like this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <suiteXmlFiles>
      <suiteXmlFile>${pathToMySuiteFile_1}</suiteXmlFile>
      <suiteXmlFile>${pathToMySuiteFile_1}</suiteXmlFile>
    </suiteXmlFiles>
  </configuration>
</plugin>

This solution has some limitations. I can only change a path to the test suite I've got defined in pom.xml. So in my example it always has to be two files. I'm not able to run, lets say, 5 suites or just one.

Is there a way to somehow parametrize the whole section "suiteXmlFiles" in pom.xml ?

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <suiteXmlFiles>
      ${multiple_paths_ToMySuiteFiles}
    </suiteXmlFiles>
  </configuration>
</plugin>

Running everything that matches given test group is not an option for me: I don't want to load all the suites I've got and then run just the selected tests using groups in TestNG suite. The reason being that a report that gets generated after running all the test suites with group filters is different from a report when just the selected test suites were run.

Orchestrate

Answer 1

According to User Property of suiteXmlFiles You can use:

mvn test -Dsurefire.suiteXmlFiles=test1.xml,test2.xml

Elwina

Using this solution, you don't need to specify the <suiteXmlFiles> tag in the pom.xml file. Best answer. – Prendergast

with this one I get error [ERROR] Unknown lifecycle phase ".suiteXmlFiles=test1.xml,test2.xml". You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. – Zymometer

No doubt it is the best answer, I was struggling for some days with same issue. With this, it is resolved now. Thanks @Sam – Diadromous

Answer 2

We also hit this issue with our tests. The current workaround that we use now is to define a property variable in the property section and inject it in sure-fire suiteXmlFiles block.

<properties>
    <!-- Default suites -->
    <batsSuiteFile>${project.build.testOutputDirectory}/BatsTests.xml</batsSuiteFile>
    <smokeSuiteFile>${project.build.testOutputDirectory}/SmokeTests.xml</smokeSuiteFile>

    <!-- Default suite files if not being specified from mvn command line -->
    <defaultSuiteFiles>${batsSuiteFile},${smokeSuiteFile}</defaultSuiteFiles>
    <suiteFile>${defaultSuiteFiles}</suiteFile>
</properties>

Then in the plugin section...

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.12</version>
    <configuration>
       <suiteXmlFiles>
          <!-- Suite file injection parameter from command line -->
          <suiteXmlFile>${suiteFile}</suiteXmlFile>
       </suiteXmlFiles>
    </configuration>
</plugin>

If nothing is specified from the command line it will fall back and default to the 2 suites specified above in the properties section. Then if you want to kick off a specified set of suite files you can do:

mvn test -DsuiteFile=test1.xml,test2.xml

Surprisingly from the maven docs you expect that the arg suiteXmlFiles should just override this from the command line and should just accept a comma delimited list of testng xmls http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html#suiteXmlFiles

If anyone has any other better way please share.

Annates

other way is to run below set of commands one after the other mvn -D tests ${path}/test1 test , mvn -D tests ${path}/test2 test and dependency is <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.12</version> <configuration> <suiteXmlFiles> <!-- Suite file injection parameter from command line --> <suiteXmlFile>${tests}.xml</suiteXmlFile> </suiteXmlFiles> </configuration> </plugin> – Improvisatory

Use -Dsurefire.suiteXmlFiles instead. See my answer below. – Elwina

@norbertpy this example also uses that variable :) – Annates

Answer 3

You can use groups parameter or specify -Dgroups=... in the command line:

(TestNG/JUnit47 provider with JUnit4.8+ only) Groups for this test. Only classes/methods/etc decorated with one of the groups specified here will be included in test run, if specified. For JUnit, this parameter forces the use of the 4.7 provider This parameter is ignored if the suiteXmlFiles parameter is specified. .

Overcloud

I mentioned it that I don't want to use groups as it breaks test result report and is not flexible enough anyway – Orchestrate

Flexible or not, but that's your only choice. It can be as flexible as granular your groups are. Of course besides running individual tests. – Overcloud