Running tasks parallel in powershell

Question

I have a PowerShell script like this:

Foreach ($file in $files) {
    [Do something]
    [Do something]
    [Do something]
}

This way one file is treated after the other. I want to treat 4 files at the same time.

I know of the foreach -parallel loop, but that does the [do something] tasks in parallel. I basically want to run the whole foreach loop in parallel.

How can I achieve this in PowerShell?

Berliner

Answer 1

You might look into Jobs or runspaces. Here is an example of Jobs:

$block = {
    Param([string] $file)
    "[Do something]"
}
#Remove all jobs
Get-Job | Remove-Job
$MaxThreads = 4
#Start the jobs. Max 4 jobs running simultaneously.
foreach($file in $files){
    While ($(Get-Job -state running).count -ge $MaxThreads){
        Start-Sleep -Milliseconds 3
    }
    Start-Job -Scriptblock $Block -ArgumentList $file
}
#Wait for all jobs to finish.
While ($(Get-Job -State Running).count -gt 0){
    start-sleep 1
}
#Get information from each job.
foreach($job in Get-Job){
    $info= Receive-Job -Id ($job.Id)
}
#Remove all jobs created.
Get-Job | Remove-Job

In the above code I have it where each $file is running in parallel with eachother (Up to 4 running simultaneously).

EDIT: In response to the comments, here is some documentation about scriptblocks. The short reason about why you must include the parameter is because unlike PowerShell functions, scriptblocks can't specify parameters outside of the braces {}.

Simplify

Nicely done, though I would recommend using the Wait-Job cmdlet instead of all the fancy logic you include to get running jobs and wait for them to complete. Link to alternative walk-through of runspaces that I personally learned it from and like. – Ohalloran

Oh, and you should probably explain why you have to pass the $file parameter into your scriptblock. – Ohalloran

I'll look into that for future scripts. I had no idea Wait-Job existed. – Simplify

This method of using jobs is way slower for loading and filtering files. I thought it would be faster because 4 files could be loaded and filtered at the same time. But actually loading and filtering one file after another is a lot faster. How can you explain this? And is there a way to make it faster? – Berliner

Would you be willing to add an example using runspaces? I am reading up and trying to convert your Jobs example to use runspaces but can't figure it out. – Chilung

Answer 2

Powershell 7 introduces foreach-object -parallel:

https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/

Your script would then say

$files | ForEach-Object -parallel {
    [Do something]
    [Do something]
    [Do something]
}

Isa

Answer 3

According to Get-Help about_Foreach-Parallel, ForEach -Parallel... will process the entire scriptblock in parallel for each item, but the commands in the scriptblock will be processed sequentially (though presumably they will be parallelized if bracketed with Parallel {...}). However, your script must be a PowerShell workflow for this to be accepted; the Parallel and Sequence keywords are only effective within workflows.

Cristoforo

From msdn.microsoft.com/en-us/powershell/reference/5.1/psworkflow/… : Describes the ForEach -Parallel language construct in Windows PowerShell Workflow. The -Parallel parameter doesn't exist if you're not using a workflow. I didn't get the feeling the OP is working with workflows. – Mcmahon

@Mcmahon - Yes, the Parallel switch and construct are only applicable to workflows, and I did note that. The querent did mention the Foreach -parallel construct in his question, but appeared to misunderstand the effect, so I assumed that a workflow was a possibility, and answered on that basis. If he's not, the job solution that Curious One gave might be the way to go. – Cristoforo

You're absolutely right, you did mention workflows. Sorry about that! – Mcmahon

Answer 4

Leverage runSpace using Foreach-Object

Two simple comparation help to understand:

(Measure-Command {
    1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 
}).Seconds

(Measure-Command {
    1..5 | ForEach-Object { "Hello $_"; sleep 1; }
}).Seconds

A perfect blog for explain this: https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/

Jacinto