PowerShell quickly Ping Subnet with Jobs

Question

The following function will ping my subnet with PingRange 1 254 to check IP's:

function PingRange ($from, $to) {
    $from..$to | % {"192.168.0.$($_): $(Test-Connection -BufferSize 2 -TTL 5 -ComputerName 192.168.0.$($_ ) -quiet -count 1)"}
}

However, this is slow, so I was wondering if it's possible to ping them all concurrently, then collect the results. I guess that this would mean:

1. Using Start-Job on each Test-Connection (which I can do, that part is easy).

2. Waiting for all to complete.

3. Collecting only the ping success results and sorting them.

   function PingRange $from $to { $from..$to | % {Start-Job { "192.168.0.$($_): $(Test-Connection -BufferSize 2 -TTL 5 -ComputerName 192.168.0.$($_ ) -quiet -count 1)"} } Wait-Job *some test to check if all jobs are complete* Receive-Job some way to get each result, discard all failures, then sort and output to screen }

Is there a shorthand way to do a Wait-Job that will just wait for all spawned jobs to complete?

Receiving the information also seems tricky, and when I try it, I invariable get nothing back from Receive-Job (or a nasty error usually). Hopefully someone more expert on PowerShell Jobs knows how to grab these results easily?

Lector

Answer 1

Note: In Windows PowerShell the simplest solution is to use Test-Connection -AsJob, as shown in js2010's answer. -AsJob is no longer supported in PowerShell [Core] 6+, however.
This answer focuses on command-agnostic ways to achieve concurrency (not just with jobs).

---

In PowerShell v7+, you'll be able to use ForEach-Object -Parallel, which can greatly simplify your function, by running your commands in parallel, using different threads:

function PingRange ($from, $to) {
  $from..$to | ForEach-Object -Parallel {
    "192.168.0.$_`: $(Test-Connection -BufferSize 2 -TTL 5 -ComputerName 192.168.0.$_ -quiet -count 1)"
  } -ThrottleLimit ($to - $from + 1) 2>$null -ErrorVariable err | Sort-Object
}

• -ThrottleLimit defaults to 5, which means that up to 5 commands run in parallel, which additional ones queued until one threads in the pool become available again, as previous commands finish.

  • Here, I've chosen to allow all threads to run in parallel, but you'll have to test if that works in practice - it may work for network-bound tasks such as here, but it's not right choice for CPU-bound tasks; see this blog post for guidance.

• 2>$null silences error output, but -ErrorVariable err collects any errors in variable $err for later inspection:

  • Note: As of v7.0, only 2>$null works for silencing errors; the common -ErrorAction parameter is not supported (and neither are -WarningAction, -InformationAction, -PipelineVariable); note that 2>$null can trigger a script-terminating error if $ErrorActionPreference = 'Stop' happens to be in effect.

• Output from the threads will arrive in no guaranteed order, but will print as it arrives.

  • Given that your want sorted output anyway, that is not a problem here.
  • If you do need the output in input order, use the -AsJob parameter, use the resulting job object with Wait-Job to wait for all threads to finish, at which point you can call Receive-Job to receive all the outputs in input order.

---

In PowerShell v6-, you do need jobs for concurrency, but it's better to use Start-ThreadJob than Start-Job, because thread jobs have much less overheads than the standard background jobs, which are child-process-based.
Jobs of either type can be managed with the same set of other job-related cmdlets, such as Receive-Job, shown below.

Note: The implementing ThreadJob module ships with PowerShell 6+; in Windows PowerShell you can install it on demand; e.g.: Install-Module ThreadJob -Scope CurrentUser.

function PingRange ($from, $to) {
  $from..$to | ForEach-Object {
    Start-ThreadJob -ThrottleLimit ($to - $from + 1) { 
      "192.168.0.$using:_`: $(Test-Connection -BufferSize 2 -TTL 5 -ComputerName 192.168.0.$using:_ -quiet -count 1)" 
    }
  } | Receive-Job -Wait -AutoRemove -ErrorAction SilentlyContinue -ErrorVariable err |
      Sort-Object 
}

Note the need for $using:_ in order to reference the enclosing ForEach-Object script block's $_ variable.

While Start-ThreadJob uses threads (runspaces) to run its jobs, the resulting job objects can be managed with the standard job cmdlets, namely Wait-Job, Receive-Job and Remove-Job.

---

Advantages of using Start-ThreadJob over Start-Job:

• Start-ThreadJob uses threads (separate in-process PowerShell runspaces via the PowerShell SDK) for concurrency rather than the child processes Start-Job uses. Thread-based concurrency is much faster and less resource-intensive.

  • See this answer for an example of the performance gains that Start-ThreadJob brings.

• The output from thread jobs retain their original type.

  • By contrast, in Start-Job jobs input and output must cross process boundaries, necessitating the same kind of XML-based serialization and deserialization that is used in PowerShell remoting, where type fidelity is lost except for a few known types: see this answer.

The only - largely hypothetical - downside of Start-ThreadJob is that a crashing thread could crash the entire process, but note even a script-terminating error created with Throw only terminates the thread (runspace) at hand, not the caller.

In short: use Start-Job only if you need full process isolation; that is, if you need to ensure the following:

• A crashing job must not crash the caller.

• A job should not see .NET types loaded into the caller's session.

• A job should not be able to modify the caller's environment variables (in jobs of both types the caller's environment variable values are present, but in the case of background jobs they are copies).

Note that in both Start-ThreadJob and Start-Job jobs, the jobs do not see the caller's state in terms of:

• Variables, functions, aliases, or PSv5+ custom classes added to the caller's session, either interactively or via a $PROFILE file - jobs do not load $PROFILE files.

  • However, thread jobs do see .NET classes (types) loaded into the caller's session, and, unlike regular jobs, they not only see the values of the caller's environment variables, but can also modify them.

  • Both cmdlets support passing the values of regular variables from the caller's scope, via the $using: scope.

• In PowerShell 6-, the initial current directory (filesystem location) for jobs was not the same as the caller's; fortunately, this is fixed in v7+; once started, jobs maintain their own current location and changing it does not affect the caller.

Genesis

Thanks. I have read about the -Parallel option and it will be good to take advantage of that. I have never heard of Start-ThreadJob, and this is interesting, as in my tests, Start-Job can have awful performance (in the concurrent ping that I wanted to do, I found that the overhead of starting each job meant that there was no point as sequentially pinging whole subnet took almost identical time to concurrent!). Ok, this is amazing, going to test this more now. By the way, should Receive-Object be Receive-Job? I get an error from Receive-Object (on PS v5.1). – Lector

@YorSubs: Yes, it should be Receive-Job - thanks for catching that typo; I've fixed the answer. Start-Job's performance is indeed poor, and unless you need full process isolation (which is probably rare), Start-ThreadJob is the better choice, both in terms of performance and type fidelity - please see my update. – Genesis

Answer 2

Receive-Job is returning every output from the commands in the job. This means if an error is thrown inside the job - it is also displayed bei receive-job. To work around that, closely control what output your commands generate by for example piping to out-null.

A basic example:

$jobIDs = @()
for ($i = 0;$i -lt 10;$i++){
    $jobIds += (start-job -ScriptBlock {
        sleep -seconds (get-random -Maximum 10)
        return (get-random -Maximum 5)
    }).Id
}

while ((get-job -State Running).count -gt 0){
    write-host "waiting for jobs to finish... ($((get-job -state Running).count) still running)"
    sleep -Seconds 1
}

foreach ($jobID in $jobIDs){
    write-host "Job $jobID returned: $(receive-job $jobID)"
}

Cyanamide

Answer 3

You can test the whole list like this. test-connection can take an array of hosts. It will go very fast if most of them are up. The ResponseTime property will be non-null if the ip is up.

$list = 1..3 -replace '^','192.168.1.'
$result = test-connection $list -asjob -count 1 | receive-job -wait -autoremovejob
$result | where responsetime  # up hosts
$result | where { ! $_.responsetime } # down hosts

Diskin

Answer 4

You can use this adding the asJob parameter:

function PingRange ($from, $to) {
    $from..$to | % {"192.168.0.$($_): $(Test-Connection -BufferSize 2 -TTL 5 -ComputerName 192.168.0.$($_ ) -quiet -count 1 -asJob)"}
}

Ninanincompoop

Thanks for that Wasif. Starting Jobs is ok for me, I can do that (the -asJob point is useful of course though). The main question is about receiving and collating the information which I've found mostly gets me errors (i.e. if I do a receive-job on the above, I just get an error). – Lector

Why, you want to run the com.and as a job? – Ninanincompoop

I can run it as a job, that's no problem. That's the easy part, I can do that. I want to collect, filter and collate the output, that's the part where I always hit errors (see OP). – Lector

See sherweb.com/blog/cloud-server/… – Ninanincompoop

That's a great page Wasif, that's for that, exactly what I was after. I marked the other thing as the answer as that template is really perfect for me to use the PS Jobs, but this link you've given me is exactly what I was trying to build completely. Thanks. – Lector