How can I run my independent Robotium UI tests in parallel?

Question

I'm using Jenkins for my Android continuous integration. I have some isolated, independent Robotium UI tests that currently take 12 minutes to run serially against a single emulator. Can anybody recommend a good way to run them in parallel so it will take only 6 minutes (or less)?

I know about various ways to run the full test suite in parallel on multiple devices/emulators, e.g. see the Multi-configuration (matrix) job section of the Jenkins Android Emulator Plugin, Spoon, or cloud testing companies like AppThwack.

I know how to run a specific subset of my tests, by using JUnit annotations, or apparently Spoon supports a similar function (see my question about it).

I'm now using Spoon to run my full test suite (mostly to take advantage of the lovely HTML output with screenshots). If anybody has tips on the best way to split my tests and run them in parallel, that would be great.

I assume I could achieve this by splitting the tests into two separate CI jobs, but it sounds like a pain to maintain two separate jobs and combine the results.

Dari

have a look here #11616175 – Cirrose

Answer 1

This answer is an improvement on my previous MultiJob answer.

---

The best way I've found to do this is to use a Jenkins Matrix job (a.k.a. "Multi-configuration project"). This is really convenient because you can configure everything in a single Jenkins job.

Spoon now supports a --e option which allows you to pass arguments directly to the instrumentation runner. I've updated their README file with a section on Test Sharding.

That README should give you what you need, but here are other highlights from our Jenkins job configuration, in case that helps.

The User-defined Axis sets the number of slave nodes we want to run. We have to set the label to android so our cloud provider can launch an appropriate slave.

We have a run-build.sh script which invokes Spoon with the correct parameters. We need to pass in the total node count (in this case 6), and the index of the specific slave node being run (automatically present in the node_index variable).

The post build steps shouldn't be any different to your existing ones. In future we'll probably need to add something to collect up the results onto the master node (this has been surprisingly difficult to figure out). For now you can still click through to results on the slaves.

Dari

Answer 2

Update: I've added another answer which I think gives a cleaner and more concise Jenkins configuration, and is based more directly on Spoon.

---

I've just discovered the Jenkins MultiJob Plugin which allows you to run multiple jobs in parallel within a Phase.

Below is my working, but slightly fragile approach to doing this using the Fork plugin. I use manually configured regular expressions to partition the tests (this was the main reason I tried to use Fork - it supports using regex).

The MultiJob looks like this with multiple downstream jobs in a Phase:

Main job configuration

Here's how my "Android Multi Job" is configured:

Downstream job configuration

Here's how the downstream "Android Phase N" jobs are configured (with different android.test.classes regular expressions for each):

Gotchas

• Fork currently fails to run on Gradle v1.0.0, as per fork plugin issue #6.
• If you want a Fork regex to match multiple different packages, you need to comma separate your regex. This isn't very well documented in the Fork project, but their TestClassFilter source shows you how they interpret the regex.
• Any abstract test classes need to be named Abstract*, otherwise Fork will try to run them as tests, creating annoying failures. Their TestClassScanner controls this, and issue #5 tracks changing this.
• IIRC, you need to have the Fingerprint Plugin installed for the "Aggregate downstream test results" option to work. If you don't have it installed you'll see this error: "Fingerprinting not enabled on this build. Test aggregation requires fingerprinting."

Limitations

• Test results are aggregated, but only using the JUnit XML test reports. This means you need to click through to each downstream job to view nice HTML results.
• Manually partitioning your tests based on regular expressions can be tedious and error prone. If you use this approach I recommend you still have a nightly/weekly Jenkins job to run your full test suite in a single place, to make sure you don't accidentally lose any tests.
• This MultiJob approach requires you to manually configure each downstream job, one for each slave node you want to use. We've prototyped a better approach using a Matrix job, where you only have to configure everything in a single Jenkins job). We'll try to write that up in the next couple of weeks.

Futures

We've also prototyped a way of extending Spoon (the output is more pretty than Fork) to automatically split the whole test suite across N downstream jobs. We still need to enhance it to aggregrate all those results back into a single HTML page in the upstream job, but unfortunately a bug in the Jenkins "Copy To Slave" plugin is blocking this from working at the moment.

Dari

Answer 3

You can perform this in 3 steps:

1. Create 2 nodes pointing to the single target machine (which satisfies your condition to run tests on same machine).
2. In the Job during execution use the Jenkins env variable $NODE_NAME to and assign different set of tests to each node (you may need the NodeLabel Parameter Plugin).
3. After execution you will have 2 report files, luckily on the same machine. You can either merge them into one if they are text files or create xml something similar to PerfPublisher plugin format which gives you detailed report.

It means you can actually execute 2 sets of tests on same machine (2 nodes pointing to it) using a single job. Obtaining single report would be tricky but If I get to know the format I can help.

Hope this is useful

Jutland

NodeLabel Parameter Plugin is not a must according to me. Only if you are deciding which node to pick for execution during run-time you would need it. But in your case you are already dividing the tests into subsets and you have to execute on both nodes but only that both runs different set of tests. This could be handled in a trigger script which could be "Shell" or "batch" script where you sure if %NODE_NAME% equal "Node1" then execute subset1 else if %NODE_NAME% equals "Node2" then execute subset2. – Jutland

My reports are in JUnit XML format, so if you know a way to merge the results that would be great! I'm using a "Post-build action" to "Publish JUnit test result report", so that would need to be triggered after both sets of tests have run. I'm not familiar with using nodes in Jenkins, but I'm very happy to try it if I know how to merge the two sets of reports! – Dari

You don't need to merge the reports. Xunit plugin handles multiple reports: wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin. You just have to copy the reports back to the master : wiki.jenkins-ci.org/display/JENKINS/Copy+To+Slave+Plugin – Abominate

My Jenkins instance is currently hosted in CloudBees, and I unfortunately don't have permission to add Nodes. However, I have just discovered the MultiJob Plugin, which seems to make the process more usable - I'll add a separate answer. – Dari

There is a plugin called PerfPublisher which takes XML files as inputs and gives you an aggregated report. But the only dependency is that the report has to be of a particular format.. – Jutland

Answer 4

You can use for example Jenkins MultiJob Plugin and Testdroid API to send your APKs to real devices. That's probably the easiest way to do it. Ref guide here.

Hootenanny