Using WebDeploy (MSDeploy) to deploy to a Microsoft Azure Website target doesn't work

Question

I'm trying to deploy my web app via Visual Studio 2015's "Publish" functionality to an Azure website/app service staging slot and I receive the following errors:

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 1 of 10. Updating file (xxxxxxxxxxxxx__staging\bin\EntityFramework.dll). C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 2 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 3 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 4 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 5 of 10. Updating file (xxxxxxxxxxxxx__staging\bin\EntityFramework.dll). C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 6 of 10. Updating file (xxxxxxxxxxxxx__staging\bin\EntityFramework.dll). C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 7 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 8 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 9 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Warning : Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object sitemanifest (sourcePath). Attempt 10 of 10. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): Error : Web deployment task failed. (Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.) Publish failed to deploy.

I have searched literally everywhere for a solution. I have tried re-creating a staging slot. I have tried searching on the web for any reference to any part of the error messages; but no solutions are available.

I'm using Visual Studio 2015 Update 3. It's a .NET 4.5.1 web app.

In the event log there is also another error message: Error Code: 999 Exception Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. Exception Stack Trace: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers, SocketFlags socketFlags) at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers) --- End of inner exception stack trace ---
Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state) at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.IO.BufferedStream.Flush() at System.IO.BufferedStream.Dispose(Boolean disposing) at System.IO.Stream.Close() at System.IO.BinaryWriter.Dispose(Boolean disposing) at System.IO.BinaryWriter.Close() at Microsoft.Web.Deployment.PackageSerializer.Dispose() at Microsoft.Web.Deployment.AgentClientProvider.RemoteDestSync(DeploymentObject sourceObject, DeploymentSyncContext syncContext, Nullable1 syncPass, String syncSessionId) at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable, Nullable1 syncPassId, String syncSessionId) at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions)

However, I believe the above is a red herring; I would imagine it's a problem with:

Retrying operation 'Serialization' on object sitemanifest (sourcePath).

It appears this is retried 10 times with failure; this would probably mean the open MSDeploy connection would close, causing the second failure.

Please note that within the Publish wizard I clicked the "Validate connection" button and it came up with a green tick

Radarscope

Evil malware is interfering with your connection, usually it is the shrink-wrapped kind. Nothing that can ever be diagnosed by somebody that isn't looking over your shoulder. Just google "azure socket error (10054) occurred" for basic hits. – Spangler

Answer 1

Microsoft have pretty much confirmed it's a bug. They are struggling to reproduce the issue though.

The workaround is to put a new app setting into the Web app on the Azure Portal, called WEBSITE_WEBDEPLOY_USE_SCM and set the value to false. Deployment then works fine.

Radarscope

Thanks @Radarscope for this, took me awhile to find this post. I had to also reset and re-download my publish profile after doing this (also restarted web app first for good measure) and it worked! – Urbannai

Answer 2

We had this issue deploying to a server on-prem and after much troubleshooting our Networking team found a network issue that was intercepting these packets and dropping them. We found this by monitoring network traffic using WireShark to find where/when the issues were occurring. The Networking team researched using their tools further to find the issue. Sorry, I don't know what they actually changed to fix this.

You could try deploying from a different network (Starbucks, home, wherever) to verify this is your issue.

Yulan

Answer 3

After trying for over a week, today it worked. Nothing has changed on my side, so Microsoft must have fixed an issue in Azure.

Radarscope

And the issue is back again today. – Radarscope

Ever get to the bottom of this Kristan? – Eliaseliason

@Fetchezlavache MS pretty much confirmed it's a bug. There is a workaround, which I'll post as an answer now. – Radarscope