Managing the running time of background jobs. Timing out if not completed after x seconds,

Question

I would like to time my background jobs (started with start-job) and time them out after x seconds. I find it hard however to keep track of the running time on each separate job (I am running aprox 400 jobs).

I wish there was a way to time out the job and set it to failed if not completed in X seconds, but I find no timeout-parameter.

What would be a good way to track the individual run-time of the jobs?

I guess I could create a hashtable with start-time of each job and the job-id and check against the running state and do a manual timeout, but that sounds kinda "inventing the wheel". Any ideas?

Edit Thank you everyone for a fruitful discussion and great inspiration on this topic!

Desilva

I probably would have come up with something like the HashTable solution you mentioned in the last paragraph. Seems like a good idea. – Biosphere

Answer 1

You can use a hash table of timers:

 $jobtimer = @{}

 foreach ($job in $jobs){
   start-job -name $job -ScriptBlock {scriptblock commands}
   $jobtimer[$job] = [System.Diagnostics.Stopwatch]::startnew()
   }

The running time of each job will be in $jobtimer[$job].elapsed

Sanctuary

This is perfect. I am however having problems with referencing the values directly without setting it to a variable.. How come get-job | foreach { echo $jobtimer[$_.name].elapsed.totalseconds } doesn't work but $jobtimer[4].elapsed.totalseconds does work? – Desilva

Answer 2

You can specify the timeout option of Wait-Job:

-Timeout

Determines the maximum wait time for each background job, in seconds. The default, -1, waits until the job completes, no matter how long it runs. The timing starts when you submit the Wait-Job command, not the Start-Job command.

If this time is exceeded, the wait ends and the command prompt returns, even if the job is still running. No error message is displayed.

Here's some example code:

This part just makes some test jobs:

Remove-Job -Name *
$jobs = @()
1..10 | % {
    $jobs += Start-Job -ScriptBlock {
        Start-Sleep -Seconds (Get-Random -Minimum 5 -Maximum 20)
    }
}

The variable $timedOutJobs contains jobs that timed out. You can then restart them or what have you.

$jobs | Wait-Job -Timeout 10 
$timedOutJobs = Get-Job | ? {$_.State -eq 'Running'} | Stop-Job -PassThru

Larva

This was my first thought too but the one problem with Wait-Job is that it is blocking. If the script can be completely blocked while the OP's 400 jobs either complete or time-out then this will work. There is also an implicit assumption in the use of Wait-Job Timeout that all the jobs start at more or less the same time. – Kenward

@KeithHill Sune's been developing this script with SO's help for the last couple weeks lol. One of the earlier questions dealt with how to throttle jobs. I don't think 400 will be running at the same time. – Larva

You are right @AndyArismendi! I'm throttling the jobs.. Only running 20 at the time (I've found that to be a nice sweet-spot). If I do a wait-job at the end of the script (or anywhere in the script) the script will resubmit ALL jobs that have timed out.. I need to limit this to my throttle level. I think I have to do a test against a hash-table in my foreach loop. Thanks anyway:):) – Desilva

@Desilva You could use this to do that if you make $timedOutJobs an array and add to it with every group of jobs you run. Say you run 20 and collect the jobs that timed out for that block in the array and then run 20 more and also collect the jobs that timed out for that block in the array too. By the end you'll have an array with all jobs that timed out. This would give all jobs an equal amount of time to time out. – Larva

Or, just re-submit the First one. So if you keep 20 running and have a time ordered list of jobs. Then when your Wait times out, you can just stop and re-submit one job (the firts one in the list, which is the one that has been running longest). – Washko

Answer 3

Just walk through the list of running jobs and stop any that have run past your timeout spec e.g.:

$timeout = [timespan]::FromMinutes(1)
$now = Get-Date
Get-Job | Where {$_.State -eq 'Running' -and 
                 (($now - $_.PSBeginTime) -gt $timeout)} | Stop-Job

BTW there are more properties to a job object than the default formatting shows e.g.:

3 >  $job | fl *


State         : Running
HasMoreData   : True
StatusMessage :
Location      : localhost
Command       :  Start-sleep -sec 30
JobStateInfo  : Running
Finished      : System.Threading.ManualResetEvent
InstanceId    : de370ea8-763b-4f3b-ba0e-d45f402c8bc4
Id            : 3
Name          : Job3
ChildJobs     : {Job4}
PSBeginTime   : 3/18/2012 11:07:20 AM
PSEndTime     :
PSJobType     : BackgroundJob
Output        : {}
Error         : {}
Progress      : {}
Verbose       : {}
Debug         : {}
Warning       : {}

Kenward

Psbegintime is exactly what I am looking for! But I can't find it. Are you running V2 or V3? – Desilva

Running V3 at the moment. If V2 doesn't have that field then yeah, just create a hashtable to map the $job object (or id) to a timestamp that you create when you create the job. – Kenward

I'm on Win8 beta (PSv3) and I don't seem to have the Ps* properties either :/ (Start-Job -ScriptBlock { gp } | gm -MemberType Property Ps*).Count is 0. – Larva

Hmm, I guess then I'll just say that hopefully this property makes it into the final release. :-) – Kenward

There's no PSBegintime in Powershell Beta 3 on Windows 7 either. Fingers crossed that this will make it into the fina:) – Desilva

@Desilva Looks like it Scheduling Background Jobs in Windows PowerShell 3.0 – Larva

I got it confirmed at the IPUGD yesterday as well:) – Desilva

Answer 4

For completeness, this answer combines the maximum seconds per job and the maximum concurrent jobs running. As this is what most people are after.

The example below retrieves the printer configuration for each print server. There can be over 3000 printers, so we added throttling.

$i = 0
$maxConcurrentJobs = 40
$maxSecondsPerJob = 60
$jobTimer = @{ }

$StopLongRunningJobs = {
    $jobTimer.GetEnumerator().where( {
            ($_.Value.IsRunning) -and
            ($_.Value.Elapsed.TotalSeconds -ge $maxSecondsPerJob)
        }).Foreach( {
            $_.Value.Stop()
            Write-Verbose "Stop job '$($_.Name.Name)' that ran for '$($_.Value.Elapsed.TotalSeconds)' seconds"
            Stop-Job $_.Name
        })
}

Foreach ($Computer in @($GetPrinterJobResults.Where( { $_.Data }) )) {
    foreach ($Printer in $Computer.Data) {
        do {
            & $StopLongRunningJobs
            $running = @(Get-Job -State Running)
            $Wait = $running.Count -ge $maxConcurrentJobs

            if ($Wait) {
                Write-Verbose 'Waiting for jobs to fininsh'
                $null = $running | Wait-Job -Any -Timeout 5
            }
        } while ($Wait)

        $i++
        Write-Verbose "$I $($Computer.ComputerName) Get print config '$($Printer.Name)'"
        $Job = $Printer | Get-PrintConfiguration -AsJob -EA Ignore
        $jobtimer[$Job] = [System.Diagnostics.Stopwatch]::StartNew()
    }
}

$JobResult = Get-Job | Wait-Job -Timeout $maxSecondsPerJob -EA Ignore
$JobResult = Get-Job | Stop-Job -EA Ignore # Add this line if you want to stop the Jobs that reached the max time to wait (TimeOut)
$JobResult = Get-Job | Receive-Job -EA Ignore
$JobResult.count

Oboe

Wait-Job -Timeout does not stop the job per the documentation "If this time is exceeded, the wait ends and execution continues." I had to add Get-Job | Stop-Job before using Receive-Job – Thionic