Transforming files with msdeploy

Question

Can I use the config transforms mechanism of MSDeploy to transform other files?

Jaunitajaunt

Answer 1

(another approach)

The msdeploy packaging is jsut invoked during an MSbuild run for your project.

TransformXml is an included task of a .csproj or .vsproj build.

Just modify your build process to invoke that task on whatever file you need.

For example, what we do is write a custom target

<Target Name="TransformFile">

    <TransformXml Source="$(DestinationPath)\$(Sourcefile)" 
       Transform="$(DestinationPath)\$(TransformFile)" 
       Destination="$(DestinationPath)\$(DestFile)" />
    </Target>

Then modify your .csproj to run this BEFORE the Publish task is invoked.

<CallTarget Targets="TransformFile" 
   Condition="'$(CustomTransforms)'=='true'" />

Brahms

Can you give samples of how you set DestinationPath, Sourcefile, TransformFile and DestFile? I can't seem to find the write MSBuild properties to set these dynamically. – Mcnutt

If you just set the target name to "AfterBuild", then it just runs, and you don't need the "CallTarget" (at least in some VS2012 projects). – Kerbela

Answer 2

The answer by Taylor didn't work for me and he didn't provide further details. So I went spelunking into the Microsoft.Web.Publishing.targets file to find a solution. The following MSBuild Target can be added to project file to transform all other config files in the root application directory. Enjoy :)

<Target Name="TransformOtherConfigs" AfterTargets="CollectWebConfigsToTransform">
<ItemGroup>
    <WebConfigsToTransform Include="@(FilesForPackagingFromProject)"
                           Condition="'%(FilesForPackagingFromProject.Extension)'=='.config'"
                           Exclude="*.$(Configuration).config;$(ProjectConfigFileName)">
    <TransformFile>%(RelativeDir)%(Filename).$(Configuration).config</TransformFile>
    <TransformOriginalFile>$(TransformWebConfigIntermediateLocation)\original\%(DestinationRelativePath)</TransformOriginalFile>
    <TransformOutputFile>$(TransformWebConfigIntermediateLocation)\transformed\%(DestinationRelativePath)</TransformOutputFile>
    <TransformScope>$([System.IO.Path]::GetFullPath($(_PackageTempDir)\%(DestinationRelativePath)))</TransformScope>
  </WebConfigsToTransform>
  <WebConfigsToTransformOuputs Include="@(WebConfigsToTransform->'%(TransformOutputFile)')" />
</ItemGroup>
</Target>

Mcnutt

Answer 3

Short answer: Yes you can. But it's "difficult".

Long answer: When we deploy sites to destinations we had the usual web.test.config, and web.prod.config. This worked fine until we introduced log4net.test.config and log4net.prod.config. MSBuild will not automatically go through and replace all of these. It will only do the web.config ones.

If you want the nitty gritty go to the last code snippet. It shows the functions to take one config and replace it with a replacement. But... it will make more sense if I describe the whole process.

The process:

1. Msbuild makes a zip file package of the site.
2. We wrote a custom .net app that will take that zip file and do the config replacements on each one of the files. Resave the zip file.
3. Execute the msdeploy command to deploy the packaged file.

MSbuild will not automatically replace all of the extra configs. What's interesting is MSBuild will remove any "extra" configs. So your log4net.test.config will be gone after it's build. So the first thing you have to do is tell msdbuild to keep those extra files in place.

You have to modify your vbProj file to include a new setting:

<AutoParameterizationWebConfigConnectionStrings>False</AutoParameterizationWebConfigConnectionStrings>

Open your vbProj file for the web application into your favorite text editor. Navigate to each deploy configuration you want this to apply too (release, prod, debug, etc.) and add that config into it. Here is an example of our "release" config.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  ...
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <DebugType>pdbonly</DebugType>
    <DefineDebug>false</DefineDebug>
    <DefineTrace>true</DefineTrace>
    <Optimize>true</Optimize>
    <OutputPath>bin\</OutputPath>
    <DocumentationFile>Documentation.xml</DocumentationFile>
    <NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022,42353,42354,42355</NoWarn>
    <CodeAnalysisRuleSet>AllRules.ruleset</CodeAnalysisRuleSet>
    <DeployIisAppPath>IISAppPath</DeployIisAppPath>
    <AutoParameterizationWebConfigConnectionStrings>False</AutoParameterizationWebConfigConnectionStrings>
  </PropertyGroup>
  ...
</Project>

So now msbduild will build the project and keep those extra files in place and not do the replacements. Now you have to manually do them.

We wrote a .net app that will watch for these new zip files. I wrote some code that will spin through the whole zip package and find any configs that match the {configname}.{env}.config. It will extract them, replace them, and put them back. To do the actual replacement we use the same DLL's that MSDeploy uses. I also use Ionic.Zip to do the zip stuff.

So add reference to:

Microsoft.Build.dll
Microsoft.Build.Engine.dll
Microsoft.Web.Publishing.Tasks (possibly, not sure if you need this or not)

Import:

Imports System.IO
Imports System.Text.RegularExpressions
Imports Microsoft.Build.BuildEngine
Imports Microsoft.Build 

Here is the code that spins through the zip file

specificpackage = "mypackagedsite.zip"
configenvironment = "DEV" 'stupid i had to pass this in, but it's the environment in web.dev.config

Directory.CreateDirectory(tempdir)

Dim fi As New FileInfo(specificpackage)

'copy zip file to temp dir   
Dim tempzip As String = tempdir & fi.Name

File.Copy(specificpackage, tempzip)

''extract configs to merge from file into temp dir
'regex for the web.config 
'regex for the web.env.config
'(?<site>\w+)\.(?<env>\w+)\.config$

Dim strMainConfigRegex As String = "/(?<configtype>\w+)\.config$"
Dim strsubconfigregex As String = "(?<site>\w+)\.(?<env>\w+)\.config$"
Dim strsubconfigregex2 As String = "(?<site>\w+)\.(?<env>\w+)\.config2$"

Dim MainConfigRegex As New Regex(strMainConfigRegex, RegexOptions.Compiled Or RegexOptions.IgnoreCase)
Dim SubConfigRegex As New Regex(strsubconfigregex, RegexOptions.Compiled Or RegexOptions.IgnoreCase)
Dim SubConfigRegex2 As New Regex(strsubconfigregex2, RegexOptions.Compiled Or RegexOptions.IgnoreCase)

Dim filetoadd As New Dictionary(Of String, String)
Dim filestoremove As New List(Of ZipEntry)
Using zip As ZipFile = ZipFile.Read(tempzip)
    For Each entry As ZipEntry In From a In zip.Entries Where a.IsDirectory = False
        For Each myMatch As Match In MainConfigRegex.Matches(entry.FileName)
            If myMatch.Success Then
                'found main config. 
                're-loop through, find any that are in the same dir as this, and also match the config name
                Dim currentdir As String = Path.GetDirectoryName(entry.FileName)
                Dim conifgmatchname As String = myMatch.Groups.Item("configtype").Value

                For Each subentry In From b In zip.Entries Where b.IsDirectory = False _
                                     And UCase(Path.GetDirectoryName(b.FileName)) = UCase(currentdir) _
                                     And (UCase(Path.GetFileName(b.FileName)) = UCase(conifgmatchname & "." & configenvironment & ".config") Or
                                          UCase(Path.GetFileName(b.FileName)) = UCase(conifgmatchname & "." & configenvironment & ".config2"))

                    entry.Extract(tempdir)
                    subentry.Extract(tempdir)

                    'Go ahead and do the transormation on these configs
                    Dim newtransform As New doTransform
                    newtransform.tempdir = tempdir
                    newtransform.filename = entry.FileName
                    newtransform.subfilename = subentry.FileName
                    Dim t1 As New Threading.Tasks.Task(AddressOf newtransform.doTransform)
                    t1.Start()
                    t1.Wait()
                    GC.Collect()
                    'sleep here because the build engine takes a while. 
                    Threading.Thread.Sleep(2000)
                    GC.Collect()

                    File.Delete(tempdir & entry.FileName)
                    File.Move(tempdir & Path.GetDirectoryName(entry.FileName) & "/transformed.config", tempdir & entry.FileName)
                    'put them back into the zip file
                    filetoadd.Add(tempdir & entry.FileName, Path.GetDirectoryName(entry.FileName))
                    filestoremove.Add(entry)
                Next
            End If
        Next
    Next

    'loop through, remove all the "extra configs"
    For Each entry As ZipEntry In From a In zip.Entries Where a.IsDirectory = False

        Dim removed As Boolean = False

        For Each myMatch As Match In SubConfigRegex.Matches(entry.FileName)
            If myMatch.Success Then
                filestoremove.Add(entry)
                removed = True
            End If
        Next
        If removed = False Then
            For Each myMatch As Match In SubConfigRegex2.Matches(entry.FileName)
                If myMatch.Success Then
                    filestoremove.Add(entry)
                End If
            Next
        End If
    Next

    'delete them
    For Each File In filestoremove
        zip.RemoveEntry(File)
    Next

    For Each f In filetoadd
        zip.AddFile(f.Key, f.Value)
    Next
    zip.Save()
End Using

Lastly but the most important is where we actually do the replacement of the web.configs.

Public Class doTransform
    Property tempdir As String
    Property filename As String
    Property subfilename As String
    Public Function doTransform()
        'do the config swap using msbuild
        Dim be As New Engine
        Dim BuildProject As New BuildEngine.Project(be)
        BuildProject.AddNewUsingTaskFromAssemblyFile("TransformXml", "$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll")
        BuildProject.Targets.AddNewTarget("null")
        BuildProject.AddNewPropertyGroup(True)
        DirectCast(BuildProject.PropertyGroups(0), Microsoft.Build.BuildEngine.BuildPropertyGroup).AddNewProperty("GenerateResourceNeverLockTypeAssemblies", "true")

        Dim bt As BuildTask
        bt = BuildProject.Targets("null").AddNewTask("TransformXml")

        bt.SetParameterValue("Source", tempdir & filename)
        bt.SetParameterValue("Transform", tempdir & subfilename)
        bt.SetParameterValue("Destination", tempdir & Path.GetDirectoryName(filename) & "/transformed.config")
        'bt.Execute()
        BuildProject.Build()
        be.Shutdown()

    End Function

End Class

Like I said... it's difficult but it can be done.

Squab

Excellent/comprehensive answer, but you can do all of this with MsBuild. See my reply. If "extra" files are a problem, you just have to modify the ItemGroups that contain input/output files that go to the Publish/Package calls – Brahms

yea, good idea. Only issue in my scenario is we have 15 sites doing this same thing all with different configs, etc. So i built this to just "do it" at deploy time. Yours is much simpler though. – Squab

Totally forgot about the main reason for that... It gives me a package with every environment config in it. So i can deploy the same exact package to dev,test,qa,stage,prod, etc. – Squab

Yeah, we do the same (one package) still using the msbuild concept. We transform all the configs and drop them in the package, and push that. Then we use the exclude operator of msdeploy in an automated fashion to push the right config at deploy time. It scales really well, we currently use for ~ 65 applications onto rack upon racks in multiple datacenters and azure. In the beginning it was a pain, but after years of growth, Im very glad we are "jsut Msbuild" – Brahms

Could you post more specifics on how you did that? I'd love to see a bit more detail. Thanks!! – Squab

Answer 4

Just to add to this awnser, in order to modify other files than the web.config in an application published with msdeploy (webdeploy) you can set the scope attribute in the parameters.xml file in the root of the project:

<parameters>
  <parameter name="MyAppSetting" defaultvalue="_defaultValue_">
    <parameterentry match="/configuration/appSettings/add[@key='MyAppSetting']/@value" scope=".exe.config$" kind="XmlFile">
    </parameterentry>
  </parameter>
</parameters>

scope is a regex that will be used to find files to apply the match xpath to. I havent experimented with this extensively but as far as I understand it simply replaces what ever the xpath matches with the value that is provided later.

There are also other values that can be used for kind that will have different behaviors than an xpath, see https://technet.microsoft.com/en-us/library/dd569084(v=ws.10).aspx for details

note: this applies to when you're using a parameters.xml, not when using the web.config.Debug/Release files

Aggarwal