Execute a for loop in parallel using CompletableFuture in Java and log the execution

Question

I have a for loop which I am trying to parallelize using CompletableFuture.

for (int i = 0; i < 10000; i++) {
    doSomething();
    doSomethingElse();
}

What I have till now is:

for (int i = 0; i < 10000; i++) {
    CompletableFuture.runAsync(() -> doSomething());
    CompletableFuture.runAsync(() -> doSomethingElse());
}

I guess this serves the purpose but there is a requirement to print log just before the start and end of all the processing. If I do this:

log("Started doing things");
for (int i = 0; i < 10000; i++) {
    CompletableFuture.runAsync(() -> doSomething());
    CompletableFuture.runAsync(() -> doSomethingElse());
}
log("Ended doing things");

Does this guarantee that the second log statement will be printed once all the for loop is over since that is executing in a separate thread? If not, is there a way to do this without blocking the main thread?

Greta

Why do you want to use CompletableFuture? (It doesn't seem you use these objects outside the loop any more, so there is no need of doing it with CompletableFutures.) – Ailment

In order to ensure the loop has finished executing without blocking the main thread – Greta

Therefore you don't need a CompletableFuture. There are other mechanisms in Java that are better suited for this purpose. – Ailment

Answer 1

You have to collect all CompletableFutures and wait for their complete:

log("Started doing things");
List<CompletableFuture> futures = new ArrayList();
for (int i = 0; i < 10000; i++) {
    futures.add(CompletableFuture.runAsync(() -> doSomething()));
    futures.add(CompletableFuture.runAsync(() -> doSomethingElse()));
}
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
                 .thenRunAsync(() -> log("Ended doing things"));

Or when you use the ExecutorService:

CompletableFuture.runAsync(() -> {
    try {
        executorService.invokeAll(tasks);
    } catch (InterruptedException) {
        e.printStackTrace();
    }
    log("Ended doing things");
});

Barbican

Is futures.foreach(CompletableFuture::join) blocking? I don't want to block the main thread. – Greta

It is, but you can execute the join operations in an other thread if you want the main Thread to keep running. – Barbican

So will adding this after the for loop work? CompletableFuture.runAsync(() -> { futures.forEach(CompletableFuture::join); log("Ended doing things"); }); – Greta

It should, but you can also directly use CompletableFuture.allOf() – Barbican

This is neat. Thanks! – Greta

Remove the second example using the ExecutorService as it doesn't work this way (invokeAll is a blocking call). – Ailment

If you want to use an explicit ExecutorService, you can specify it as second argument to CompletableFuture.runAsync(...) – Ailment

Answer 2

I suppose CompletableFuture is the wrong concept for your needs. If you want to execute an arbitrary number of similar tasks in parallel, the easiest thing is to use the method invokeAll(...) on an ExecutionService:

// First, create a list with all tasks you want to execute in parallel
List<Callable<?>> tasks = new ArrayList<>(10000);
for (int i = 0; i < 10000; ++i) {
    // we need to create Callables, so if your doSomething method returns void, we have to convert it to a Callable using helper method from class Executors
    tasks.add(Executors.callable(this::doSomething));
}

// Then, create an executor service that can execute these tasks
// There are different executors you can choose from, I take one that has a fixed pool of threads
ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());

// Last but not least, call invokeAll to execute all tasks and wait for them to complete
executorService.invokeAll(tasks);

// This method will be called when all tasks have been completed successfully:
System.out.println("done");

Ailment

Is invokeAll() a blocking call? I don't want to block the main thread. – Greta

Yes it is. But you wrote that you want the second log statement to be called when all tasks have been finished. This is exactly what blocking is. – Ailment

Maybe I could do that in a different thread itself? I was using CompletableFuture because it has thenRun() which does not block the main thread. Which would be the better way? – Greta

So you don't care about the thread that runs your second log statement? – Ailment

No, I just want it to execute immediately after the for loop without blocking the main thread. – Greta

"No"? I mean: Is it important to you, which thread invokes the final operation (second log statement), or can it be any thread, as long as it is after all other tasks? – Ailment

It can be any thread but since this snippet is a method call, I don't want the main thread to be blocked, that's all. – Greta

In this case you should go with the answer of kairaedsch – Ailment