How to run Gradle test when all tests are UP-TO-DATE?

Question

I have my grade script set up. When I execute the Gradle build, everything is working and it runs the jUnit tests.

After that when I run the Gradle test I get the following:

C:\Users\..\..\Project>gradle test
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE

When I perform gradle clean, then Gradle build works, of course... I want to be able to reset only the tests, not build the whole project: how should I do this?

Piaffe

This seems unnecessary, based on the given information. If neither the application code nor the test code have changed, why do you need to re-run the tests? – Win

@Win Some of the tests in my code are related on 3-party inputs, I'm running my tests not only to make sure i did not put any bug inside the code, also to check if something change on the 3-party inputs that i'm getting – Piaffe

Sorry to be nit-picky, but I don't really think this is the correct way of thinking about this: if you have variable 3-party inputs isn't the correct way of dealing with this to mock these inputs in some way? Testing should actually be about testing the code you're writing. Aren't you in fairly obvious danger of getting false-positives if you're relying on 3-party input to be unacceptable? Shouldn't the strategy be to cater for problem input as part of your app code? – Assai

@mikerodent consider testing your code against a 3rd party online service. You would want to monitor possible changes in the service API to be able to respond with deployed fixes ASAP. Isn't CI tests a good way of doing that? Using a mock will only tell you your own code doesn't have regressions, but the dependencies still might have changes. using the real service would indicate that your product can actually perform the expected operations in the current environment. – Demerol

@Demerol I understand what you're saying... but 2 points: first, if your app has to cater for third party inputs changing, this versatility (including the ability to "hang up the phone" gracefully and cleanly and in a non-puzzling way during a run) needs to be part of your app code, not your testing code, and you have to anticipate in your app this input becoming incomprehensible. A secondary point is that the testing suites for your app shouldn't show fails or passes which result from "random" outside input: your tests should only be to tell you one thing: is your app a functional thing? – Assai

@mikerodent of course the app should behave gracefully on failure, and of course this good behaviour should be tested as part of regression tests and only when your own code changes. But let's agree that the need of the software deployer to be alerted of changes in external (even "random") dependencies and ship a hotfix ASAP can easily be addressed by running tests on unchanged code, tests which results does not have to appear as part of the testing suits of the app, but alert the product team in some other way. Inputs from an external source are sometimes an essential part of functionality. – Demerol

@Demerol yes, I really do see what you're saying... it's just that I'm not really sure this is part of the "testing" phase of a project in the sense of TDD. You seem to think that this will help "ship a hotfix ASAP"... but when are planning to run these 3rd party input tests? I suggest that if this 3rd party source is critical you'd want your app to be testing this 3rd party source regularly: every day/hour/whatever. Perhaps even set up a special service to do this... If it made sense you could use Gradle's Java classes as part of app code, if you like the way it handles task management. – Assai

This is also valid from an integration testing point of view where the point of the test is to validate the integration of your code with other bits of code, where it would be not appropriate to mock in dependancies – Korry

It's also valid in case of Selenium tests, where timing issues can cause a test to fail sometimes and pass other times. It's perfectly legitimate to run a test repeatedly to gain confidence that it's stable, and not passing by luck one time. – Sherburne

@mikerodent what a nonsense argument. For exactly the purpose of running tests regularly (every day/hour whatever) or even manually when it is suspected that something has changed in the 3rd party input, it is useful to be able re-run the tests if neither the application code nor the test code have change. Why not use gradle for this purpose? Re-running tests is a perfectly valid scenario, that's why the option --rerun-tasks exists, as a more helpful user points out in the accepted answer. – Protochordate

Answer 1

One option would be using the --rerun-tasks flag in the Forcing tasks to execute section. This would rerun all the the test task and all the tasks it depends on.

If you're only interested in rerunning the tests then another option would be to make gradle clean the tests results before executing the tests. This can be done using the cleanTest task.

Some background - the Java plugin defines a clean tasks to each of the other tasks. According to the Tasks documentation:

cleanTaskName - Deletes files created by specified task. cleanJar will delete the JAR file created by the jar task, and cleanTest will delete the test results created by the test task.

Therefore, all you need in order to re-run your tests is to also run the cleanTest task, i.e.:
gradle cleanTest test

Pillage

gradle cleanTest test does not rerun the tests, it cleans their output, but the test task will still get the test results from the cache - see github.com/gradle/gradle/issues/9153 – Allaallah

The comment above is right. But if you use --no-build-cache, then it'll work as expected, e.g. gradle cleanTest test --no-build-cache. – Chesty

--rerun-tasks is just the ticket. --no-build-cache even with cleaning was not functional for me. Thanks @Amnon – Liberality

gradle cleanTest test worked for me. – Ilbert

--rerun-tasks is a bad idea in a large project. It compiles everything from scratch, which can take ages, even if you just want to run a quick test in a local module. – Primordium

@AndrzejWąsowski what would you do instead? – Kreg

Answer 2

Other option would be to add following in your build.gradle:

test.outputs.upToDateWhen {false}

Indult

I used this technique for a funcTest task I created to run functional tests. – Hokeypokey

This is a much better approach than the accepted answer, as it will only be applied to the desired task. The upToDateWhen can be used in any "code-driven" way such as system properties, environment variables, project properties, etc. – Wagram

As answer https://mcmap.net/q/126087/-how-to-run-gradle-test-when-all-tests-are-up-to-date mentions, there's a useful blog post blog.gradle.org/stop-rerunning-tests which explains why this approach is not recommended as a general approach. I agree however that it may be useful and does achive what the question asks. – Dirt

Yes, this is dated answer, when I wrote this Gradle was at version 2.11 and just started to be usable, but still had many rough edges, which are polished today. – Rationale

Great answer!!! Passed it using a parameter: gradle test -Prerun-tests. Code in build.gradle: if(project.hasProperty("rerun-tests")) { test.outputs.upToDateWhen {false} } – Battleax

In Gradle 7.3+ you can use doNotTrackState("Tests should always run") to disable up-to-date checks. – Ranna

Thanks. This solution also work in combination with the gradle configuration-cache, unlike some other solutions. – Fatma

Answer 3

This was recently the topic on Gradle's blog post Stop rerunning your tests. The author shows an example using outputs.upToDateWhen { false } and explains why it is wrong:

This doesn’t actually force reruns

What the author of this snippet probably wanted to say is “Always rerun my tests”. That’s not what this snippet does though. It will only mark the task out-of-date, forcing Gradle to recreate the output. But here’s the thing, if the build cache is enabled, Gradle doesn’t need to run the task to recreate the output. It will find an entry in the cache and unpack the result into the test’s output directory.

The same is true for this snippet:

test.dependsOn cleanTest

Gradle will unpack the test results from the build cache after the output has been cleaned, so nothing will be rerun. In short, these snippets are creating a very expensive no-op.

If you’re now thinking “Okay, I’ll deactivate the cache too”, let me tell you why you shouldn’t.

Then, the author goes on to explain why rerunning some tests is a waste of time:

The vast majority of your tests should be deterministic, i.e. given the same inputs they should produce the same result.

In the few cases where you do want to rerun tests where the code has not changed, you should model them as an input. Here are both examples from the blog post that show adding an input so the task will use it during its up-to-date checks.

task randomizedTest(type: Test) {
  systemProperty "random.testing.seed", new Random().nextInt()
}

---

task systemIntegrationTest(type: Test) {
  inputs.property "integration.date", LocalDate.now()
}

I recommend reading the entire blog post.

Wagram

This sounds great for the specific use case you are talking about, but I'm writing post-deployment tests against a live external web service and just happen to be using junit and gradle to accomplish this. The code under test doesn't live in the repo, and in fact there is no 'application code' because I'm actually testing a live production system rather than the code itself. Thanks for the answer, this is very useful! Just wanted to point out that there are additional use cases that do require rerunning the tests every time even if none of the code gradle knows about is changing – Crossbill

the author is missing a basic concept of testing: nondeterministic behavior. If you are suspicious a test is OK but the code has some bugs and is making the result nondeterministic, then, with this short-minded advice you will never get a stacktrace if it happens to succeed in the first run. Test should be ran the amount of times you want. Considering bad practice running test multiple is plain wrong and goes against the test concept. – Ammonia

@Ammonia if you actually read the article, you'll see your case is covered explicitly. – Stearic

The author is also missing that since your elaborate hierarchy of gradle tasks is also code you've written, it's just as likely to contain logic errors as the rest of your code. You're only ever one tiny little bug or forgotten declared input away from your tests not running when they should. I've seen this happen so many times. – Fend

Answer 4

gradle test --rerun-tasks

Specifies that any task optimization is ignored.

Source: https://gradle.org/docs/current/userguide/gradle_command_line.html

Herminahermine

I think it will re-run all dependent tasks as well, which isn't the asker intended behavior. – Battleax

Yea this answer doesn't work at all, since it rebuilds and reruns zillions of unnecessary things. – Afterworld

Answer 5

--rerun-tasks works, but is inefficient as it reruns all tasks.

cleanTest by itself may not suffice due to build cache.

so, best way to accomplish this is:

./gradlew --no-build-cache cleanTest test

Parcae

Answer 6

Here's a solution using the "build.gradle" file, in case you don't want to modify your command line:

test {
    dependsOn 'cleanTest'
    //Your previous task details (if any)
}

And here's the output. Notice 2 changes from your previous output:

1) A new 'cleanTest' task appears in the output.

2) 'test' is always cleaned (i.e. never 'UP-TO-DATE') so it gets executed every time:

$ gradle build
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:findMainClass
:jar
:bootRepackage
:assemble
:cleanTest
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test
:check
:build

Squishy

running cleanTest before test will not rerun the tests, it cleans their output, but the test task will still get the test results from the cache - see github.com/gradle/gradle/issues/9153 – Allaallah

Is there a way to make gradle test do this, but not gradle build ? – Kreg

Answer 7

As of Gradle 7.6, it is possible to use gradle test --rerun.

A description of the --rerun feature is in the Gradle issue tracker and works for any task (not just 'test'). Basically it just forces Gradle to consider that task and only that task not up-to-date which will rerun it.

See also --rerun in the manual.

--rerun

Causes the task to be re-run even if up-to-date. Similar to --rerun-tasks, but for a specific task.

Favian

How can I add that by default (in build.gradle)? – Kreg

Answer 8

TL;DR

test.dependsOn cleanTest

Burial

According to https://mcmap.net/q/126087/-how-to-run-gradle-test-when-all-tests-are-up-to-date that won't work. – Freehold

Well, the gradle docs are a bit confusing.... Here they say that cleanTest can be used for this purpose. docs.gradle.org/current/userguide/…. And also, it works on my machine (and gradle version 4.10.3) ;) – Burial

Answer 9

None of the above methods worked for me.
What worked for me was simply removing all items from the cache directory in /Users/<username>/.gradle/caches/build-cache-1/.

Jari