Release Management vNext Component Deployment Bottleneck

Question

We're using Release Management 2015 with vNext release templates. We have a Powershell DSC based component deployment for each of the portions of our application, and in fact, we have two distinct applications being deployed and that are in active development and are often deployed at nearly the same time.

We very frequently get the following error during deployments:

OperationFailedException: New deployment is not allowed as an another deployment is in progress. Retry the deployment after sometime.

The full stack trace shows that the error is coming not from Powershell itself, but from Release Management's system that's in charge of execution of the powershell script on the target machine:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Microsoft.TeamFoundation.Release.Common.Helpers.OperationFailedException: New deployment is not allowed as an another deployment is in progress. Retry the deployment after sometime.
   at Microsoft.TeamFoundation.Release.EnvironmentProvider.OnPrem.Implementation.OnPremDeploymentProvider.ReadDeploymentResponse(DeploymentResponse response)
   at Microsoft.TeamFoundation.Release.EnvironmentProvider.OnPrem.Implementation.OnPremDeploymentProvider.RunScript(String scriptPath, String configurationPath, MachineSpecification machine, StorageSpecification storage, Dictionary`2 configurationVariables)
   at Microsoft.TeamFoundation.Release.MonitorServices.Dsc.OnPrem.OnPremDeploymentActions.InvokePlatform(String activityId, MachineSpecification machineSpecification, StorageSpecification storageSpecification, String scriptPath, String configurationPath, Dictionary`2 configurationVariables)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.TeamFoundation.Release.DeploymentAgent.Services.Deployer.Dsc.DscComponentInstaller.InvokeMethodByReflection(String methodArguments)

The above error causes the entire deployment to fail, and we're forced to retry the stage or the entire deployment to have it finish.

There are two scenarios that cause this:

1. Two release templates execute their powershell scripts on the same target server at the same time
2. A single release template has a parallel control flow which contains two different components that both execute scripts on the same target server

In other words, the mechanism that Release Management uses to execute a powershell script on a remote server can only seem to ever execute a single script at a time, and has no ability to wait/hold for others to complete.

This kinda/sorta makes sense if the script in question as actively modifying the server on which it's executing, but in our case, the server acts as basically a staging area to run the script. The "real" target of the script has nothing to do with the server on which the powershell happens to be executing.

Aside from having a server-per-simultaneously-deployed-component (wow), what's the work around here? It seems like a major oversight and it's seriously making me consider dropping Release Management altogether.

Lakshmi

I suggest talking to the product team @ MS. Here is a blog in which the author (Vijay M) talks about Release Management and responds (at least twice) to comments to the blog. – Incertitude

Answer 1

As I explained in another post today, MS Release Management's way of deploying is a bit counter-intuitive: Rather than just executing your Powershell deployment script against the target server using PSRemoting, it uses PSRemoting to install a Windows Service (VisualStudioRemoteDeployer.exe) on the target server. This service then runs your deployment script locally, and the MSRM server regularly polls this Windows service (see here) to see if it is finished deploying.

I suspect this strange setup has something to do with avoiding the double-hop issue - so that it allows your script to make a 2nd hop from the target server to another server, e.g. for a webservice call.

Anyway, this Windows service probably forms the bottleneck, because there can be only one such instance running per server - hence the fact that parallel deployments of components to the same server appear to collide.

I think your problems originate from the fact that you chose a setup where "the server acts as basically a staging area to run the script" - MS Release Management 2013/2015 doesn't play well in this scenario (as you found out), you should really deploy your components directly to the target servers on which they need to be installed, thus avoiding the staging area bottleneck.

The next version of MS Release Management will be using deployment agents that will serve as a staging point from which the components will be deployed to the other servers. This helps reducing the number of connections you have to allow on your firewall between the MS Release Management and the target servers (which is probably why you chose the staging area setup), while still allowing parallel (or at least queued) deployments.

Tarrsus

In many cases there is no server yo run the script from that's also the target. For instance, we deploy schema to Sql Azure instances. Where am I supposed to run these from? This limitation may be a deal breaker for us. We don't want to run a dozen "script servers" to work around this problem. – Lakshmi

Answer 2

I was having an issue was well with running Powershell scripts on remote servers. I ended up going a slightly different route. Instead, I just run a normal Powershell command with an Invoke-Command block. I believe you should be able to run this in parallel.

Function Get-PSCredential($User,$Password) {
    $SecPass = ConvertTo-SecureString -AsPlainText -String $Password -Force
    $Creds = New-Object System.Management.Automation.PSCredential -ArgumentList $User,$SecPass
    Return $Creds
}    

$credential = Get-PSCredential -User $deployUser -Password $deployPass
$session = New-PSSession YourServerName -Credential $credential

Invoke-Command -Session $session -ScriptBlock {
    # do your work here
}

If you are running as a service account that has access to the machine, you should be able to eliminate the credential stuff and just go with

$session = New-PSSession YourServerName

I only started using Release Management this week so this seemed the best way to go in that short amount of time.

Also, If you've never used Invoke-Command before, everything inside the script block is actually in its own scope so you need to pass your variables into it using -ArgumentList if you have any. Check out this article if you have any more questions about that.

Mathewmathews

I see. So you think that by explicitly creating a new session, we will avoid these issues? Should be easily enough to test. Be right back. ;) – Lakshmi

No. The issue does not appear to be powershell. It's Release Management itself. With a vNext template/component, RM uses powershell to essentially deploy an agent which then runs the component's powershell script. This vNext agent is that's failing to run, not the powershell. This is evident from the stack trace we see. I've edited the original question to show the full stack trace. – Lakshmi