Powershell: Run multiple jobs in parralel and view streaming results from background jobs

Question

Overview

Looking to call a Powershell script that takes in an argument, runs each job in the background, and shows me the verbose output.

Problem I am running into

The script appears to run, but I want to verify this for sure by streaming the results of the background jobs as they are running.

Code

###StartServerUpdates.ps1 Script###

#get list of servers to update from text file and store in array
$servers=get-content c:\serverstoupdate.txt

#run all jobs, using multi-threading, in background
ForEach($server in $servers){
  Start-Job -FilePath c:\cefcu_it\psscripts\PSPatch.ps1 -ArgumentList $server
}

#Wait for all jobs
Get-Job | Wait-Job

#Get all job results
Get-Job | Receive-Job

What I am currently seeing:

Id              Name            State      HasMoreData     Location             Command                  
--              ----            -----      -----------     --------             -------                  
23              Job23           Running    True            localhost            #patch server ...        
25              Job25           Running    True            localhost            #patch server ...        

What I want to see:

Searching for approved updates ...

Update Found:  Security Update for Windows Server 2003 (KB2807986)
Update Found:  Windows Malicious Software Removal Tool - March 2013 (KB890830)

Download complete.  Installing updates ...

The system must be rebooted to complete installation.
cscript exited on "myServer" with error code 3.
Reboot required...
Waiting for server to reboot (35)

Searching for approved updates ...

There are no updates to install.
cscript exited on "myServer" with error code 2.
Servername "myServer" is fully patched after 2 loops

I want to be able to see the output or store that somewhere so I can refer back to be sure the script ran and see which servers rebooted, etc.

Conclusion:

In the past, I ran the script and it went through updating the servers one at a time and gave me the output I wanted, but when I started doing more servers - this task took too long, which is why I am trying to use background jobs with "Start-Job".

Can anyone help me figure this out, please?

Neotropical

Answer 1

You may take a look at the module SplitPipeline. It it specifically designed for such tasks. The working demo code is:

# import the module (not necessary in PS V3)
Import-Module SplitPipeline

# some servers (from 1 to 10 for the test)
$servers = 1..10

# process servers by parallel pipelines and output results immediately
$servers | Split-Pipeline {process{"processing server $_"; sleep 1}} -Load 1, 1

For your task replace "processing server $_"; sleep 1 (simulates a slow job) with a call to your script and use the variable $_ as input, the current server.

If each job is not processor intensive then increase the parameter Count (the default is processor count) in order to improve performance.

Granadilla

Roman, thank you for your answer. This is the closest I could get to having the output exactly as I was wanting it. I get some additional output errors, which I will troubleshoot, but this method not only runs my script, but shows me the output I desire. Thank you. – Neotropical

@talbert.houle, I am glad you find this tool useful. If you have ideas on how to make it better then you are welcome to submit them at the project site. – Granadilla

Answer 2

Between PowerShell 3 and PowerShell 5.1, PowerShell had "workflow" functionality with parallel possibilities, with less code and maybe more understandable than starting and waiting for jobs, which of course works good as well.

I have two files: TheScript.ps1 which coordinates the servers and BackgroundJob.ps1 which does some kind of check. They need to be in the same directory.

The Write-Output in the background job file writes to the same stream you see when starting TheScript.ps1.

TheScript.ps1:

workflow parallelCheckServer {
    param ($Servers)
    foreach -parallel($Server in $Servers)
    {
        Invoke-Expression -Command ".\BackgroundJob.ps1 -Server $Server"
    }
}

parallelCheckServer -Servers @("host1.com", "host2.com", "host3.com")

Write-Output "Done with all servers."

BackgroundJob.ps1 (for example):

param (
    [Parameter(Mandatory=$true)] [string] $server
)

Write-Host "[$server]`t Processing server $server"
Start-Sleep -Seconds 5

So when starting the TheScript.ps1 it will write "Processing server" 3 times but it will not wait for 15 seconds but instead 5 because they are run in parallel.

[host3.com]  Processing server host3.com
[host2.com]  Processing server host2.com
[host1.com]  Processing server host1.com
Done with all servers.

Krutz

I like that this answer does not need a 3rd party library! – Heulandite

Answer 3

In your ForEach loop you'll want to grab the output generated by the Jobs already running.

Example Not Tested

$sb = {
     "Starting Job on $($args[0])"
     #Do something
     "$($args[0]) => Do something completed successfully"
     "$($args[0]) => Now for something completely different"
     "Ending Job on $($args[0])"
}
Foreach($computer in $computers){
    Start-Job -ScriptBlock $sb -Args $computer | Out-Null
    Get-Job | Receive-Job
}

Now if you do this all your results will be mixed. You might want to put a stamp on your verbose output to tell which output came from.

Or

Foreach($computer in $computers){
    Start-Job -ScriptBlock $sb -Args $computer | Out-Null
    Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}
}
while((Get-Job -State Running).count){
    Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}
    start-sleep -seconds 1
}

It will show all the output as soon as a job is finished. Without being mixed up.

Footy

Thank you for your information. Unfortunately, I tried the first code block, and all I received on the screen was "Starting job on servername". Have waited over 20 minutes to ensure script ran on remote server, and this is all I am receiving. Will try second code block and see if anything changes. – Neotropical

Here's a link to something that may be more helpful. – Footy

Answer 4

If you're wanting to multiple jobs in-progress, you'll probably want to massage the output to help keep what output goes with which job straight on the console.

$BGList = 'Black','Green','DarkBlue','DarkCyan','Red','DarkGreen'
$JobHash = @{};$ColorHash = @{};$i=0


ForEach($server in $servers)
{
  Start-Job -FilePath c:\cefcu_it\psscripts\PSPatch.ps1 -ArgumentList $server |
   foreach {
            $ColorHash[$_.ID] = $BGList[$i++]
            $JobHash[$_.ID] = $Server
           }
}
  While ((Get-Job).State -match 'Running')
   {
     foreach ($Job in  Get-Job | where {$_.HasMoreData})
       {
         [System.Console]::BackgroundColor = $ColorHash[$Job.ID]
         Write-Host $JobHash[$Job.ID] -ForegroundColor Black -BackgroundColor White
         Receive-Job $Job
       }
    Start-Sleep -Seconds 5
   } 
 [System.Console]::BackgroundColor = 'Black'   

Hessite

When I step through this code, the script is running right over the "While" statement, and I am not receiving any output to the console. – Neotropical

The only thing I know that would cause that is for $servers to be empty/null. I tested using this scritpblock for the job action: {foreach ($i in 1..10){get-date;sleep -Seconds 2}} to create a job that runs for 20 seconds, and produces output every 2 seconds, and $servers = (1..3) as the server list. – Hessite

Answer 5

You can get the results by doing something like this after all the jobs have been received:

$array=@() Get-Job -Name * | where{$array+=$_.ChildJobs.output}

.ChildJobs.output will have anything that was returned in each job.

Tombolo