In Azure Pipelines, how can I execute a background task while other tasks run?

Question

I'm fiddling with pipelines to try and reduce the overall runtime. One of the things I'd like to do is to execute docker pull ... at the start, so that later on, when I actually need it, it's ready for me. I'd like to fire it off as a background job, and have it survive past the end of that task.

I've tried: docker pull imgname &

It does work, but the pipeline complains with this message:

The STDIO streams did not close within 10 seconds of the exit event from process '/bin/bash'. This may indicate a child process inherited the STDIO streams and has not yet exited.

I've also tried stuff like:

• docker pull imgname </dev/null &>/dev/null & disown
• docker pull imgname 0>&- 1>&- 2>&- 3>&- 4>&- 5>&- 6>&- 7>&- 8>&- 9>&- &

And a few similar tricks. Nothing helps.

This isn't a big deal, but it would be convenient to know how to make this possible!

Lilylilyan

Answer 1

Update

The STDIO streams did not close within 10 seconds of the exit event from process '/bin/bash'. This may indicate a child process inherited the STDIO streams and has not yet exited

This is not an error message which didn't write into the standard error steam and fail the task.

It should be more like an prompting message which indicate some process still run and not be clean up (expected behavior).

After enable the debug mode in the build pipeline, we could see

##[debug]The task was marked as "done", but the process has not closed after 5 seconds. Treating the task as complete.

The process should still be running as the background even though the task already marked completed.

---

According to your description, this seems not related to docker command or azure devops side.

You just need to run a powershell script (involve docker command) in background.

For example: Run Start-Job inside a PowerShell Task, that script starts to run using the Receive-Job. When the task exits the script stops.

In the PowerShell task I run the following:

Start-Job -FilePath "C:\build\BGGetFromNuGet.ps1" -ArgumentList "C:\build"

More details please take a look at this link-- Start Job

If you want that script will continue to to run in the background while the task has finished. You could try to use start-process command to launch the script. This will make sure that the launched job keeps running when the task is finished. But the job will be closed when the build is finished.

Start-Process powershell.exe -ArgumentList '-file C:\build\BGGetFromNuGet.ps1'

Koodoo

The "&" at the end of my bash command also puts the process into the background. If it wasn't a docker command, but rather just "sleep 60 &", it's the same (dunno if sleep inherits stdio though, but ..not the point) – Lilylilyan

Hi Josh, this `The STDIO streams did not close within 10 seconds of the exit event from process '/bin/bash'. This should not be an error message which more like an prompting message which indicate some process still run and not be clean up (expected behavior). With debug mode enable , we could see process should still be running as the background even the task already mark finished/completed. Please see my update reply. – Koodoo

Answer 2

Azure PowerShell task sets the context in Process scope instead of user scope.

You could switch context using:

Save-AzureRmContext -Path "$(Agent.TempDirectory)/myContext.json"
Start-Job -Job {
    Import-AzureRmContext -Path "$(Agent.TempDirectory)/myContext.json"
    Set-AzureRmContext -SubscriptionName "<sub_name>"
    # --- your powershell commands here ---
}

Source https://github.com/microsoft/azure-pipelines-tasks/wiki/Azure-PowerShell---How-to-run-background-Jobs

Both script and powershell/pwsh tasks are bound to process, but bash is not, so you could utilise bash instead, just add & at the end of command that should be running in background.

Moshe