What maven plugin is to be used for JMeter? jmeter-maven-plugin or chronos-jmeter-maven-plugin?

Question

I need to setup performance tests which are run automatically triggered by a CI system. For that I want to use JMeter due to some scripts and experience already exist and I want to combine it with Maven.

During my research for a reasonable plugin I found that two plugins are existing:

1. jmeter-maven-plugin: http://wiki.apache.org/jmeter/JMeterMavenPlugin
2. chronos-jmeter-maven-plugin: http://mojo.codehaus.org/chronos/chronos-jmeter-maven-plugin/usage.html

Which one is better to be used? Both seem to be currently maintained and under development. Is there any experience on this? Even the configuration is similar.

I would be happy to get some hints to help me descide without playing around with both plugins for some days.

Lentigo

The jmeter-maven-plugin doesn't require anything to be installed (Apart from maven obviously). It pulls down all the requirements from Maven central so it's basically a pure maven implementation. – Cambrian

Answer 1

I haven't yet used the .jmx files with maven and specifically those plugins you mention.

But I can think of a way how to do it if I needed that.

So consider this, you can execute jmeter test in no gui mode.

1. Create a shell script wrapper that will execute the jmeter test in no gui mode, example (jmeter_exe.sh):

$JMETER_HOME/bin/jmeter.sh -n -t MY_LOAD_TEST.jmx -l resultFile.jtl

So this will execute the given script and store results in the .jtl file, you can use that to display your test results maybe this post will be useful to you, it's off topic for now.

With step one done.

2.You could then create directory scripts in your project root. Than you can put this in your pom.xml :

<plugin>
  <artifactId>exec-maven-plugin</artifactId>
  <groupId>org.codehaus.mojo</groupId>
  <executions>
    <execution>
      <id>Run load Test</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <executable>${basedir}/scripts/jmeter_exe.sh</executable>
      </configuration>
    </execution>
  </executions>
</plugin>

And voila your test is executed during generate-sources phase. This might have been easier with the plugins you mentioned but I have no knowledge of those, this is what just came to my mind.

Convolution

Thanks for the answer. I and my collegues want to prepare and run the tests in JMeter. As soon as they work and are stable, we want to put them into Maven to get them automated. For debugging, we want to run these tests in JMeter again. This is best done with JMX files, I guess. – Lentigo

Answer 2

Use jmeter-maven-plugin: http://wiki.apache.org/jmeter/JMeterMavenPlugin.

It's the de-facto one and (as @Ardesco mentioned above) it doesn't require anything to be installed, which gives you abstraction on where JMeter executable is installed and all those kind of problems...

Humblebee

Answer 3

Word(s) of warning on the apache plugin (lazerycode):

• It suppresses JMeter output by default, add the following configuration settings to prevent that:

<configuration>  
    <suppressJMeterOutput>false</suppressJMeterOutput>  
    <!-- to override debug logging from the plugin (although also in jmeter.properties) -->  
    <overrideRootLogLevel>debug</overrideRootLogLevel>  
    <jmeterLogLevel>DEBUG</jmeterLogLevel>  
</configuration>

• Looking at the source (of version 1.8.1), it seems the -Xms and Xmx are limited to 512

• The plugin swallows exceptions so your tests may fail but you don't know why. It looks like they've just completed but not provided results.

• The jmeter mojo kicks off jmeter as a new java process but does not provide the capacity to provide any arguments to this execution. So if exceptions are swallowed (See above), and logging isn't sufficient (which it may not be) it's not easy to debug the process to fing out what's wrong. We (my colleague) added the debug args to the process execution and debugged the jmeter call to find out.

• you get informative output running jmeter directly for dev purposes. I'd say it's even more informative in the jmeter UI output.

I've not used chronos mind.

Spessartite

Version 1.9.0 should have removed some concerns you have. The -Xms and -Xmx are now configurable. You can also now add additional arguments to the JVM process that JMeter is started in. I'm not sure what you mean by "The plugin swallows exceptions". – Cambrian

@Ardesco, thanks for the update. We'll check it out... I'd have to review when exceptions were being swallowed when the opportunity arises. – Spessartite

Answer 4

JMeter Maven Plugin by @Ardesco is updated every time JMeter version is released. It is very well documented and works perfectly.

It is easily setup and allow easy addition of plugins like JMeter-Plugins or commercial plugins as long as required libraries.

You can read a full blog showing the setup for old version 1.1.10:

• http://www.ubik-ingenierie.com/blog/integrate-load-testing-in-build-process-with-jmeter-ubikloadpack-maven/

For more recent version 2.5.1 (as of November 2017) ensure you read documentation:

• https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/wiki

Farseeing