Ok, Daniel I definately agree with using heat, but I think Rob pointed me in the right direction for what I ended up using.
What I ended up using was HeatDirectory.(Please note, I do not plan on doing patching. If you want patching, you may want to look at another solution, but heat/heat directory can work for patching.) Unfortunately, Rob's answer was quite limited, so I wouldn't say that he really answered my question. Just pointed me in the right direction. Thanks to both Rob and Daniel for your help.
Therefore, I came back and answered this question myself.
Hurdles...
Hurdle 1
I ran into is there is nearly no documentation on the web for how to implement heat tasks/heat directory(Heat wrapped in msbuild) from within visual studio. Please note that the wrapper to run from msbuild appears to be pretty new and I disclaim to use at your own risk, etc. Because of the lack of documentation, I sent an email to the wix user list. I received help from Brian Rogers. Thanks Brian, this solved everything for me. Take a look at his great blog entry here for a better understand how heat is going to help you automate: http://icumove.spaces.live.com/blog/cns!FB93073C6534B681!461.entry
Hurdle 2
After working with heat, I found out that I was not using components the recommended way, but I had to piece together all the reasoning. I think many people will run into this in the beginning of using Wix if you don't have prior knowledge of windows installer. Therefore I recommend some reading:
Make sure you read component rules as well if you are a novice like I was with Windows Installer.Component rules from MS: http://msdn.microsoft.com/en-us/library/aa370561.aspx. Read these from Rob Mensching for a better understanding of components: http://robmensching.com/blog/posts/2003/10/4/Windows-Installer-Components-Introduction
http://robmensching.com/blog/posts/2003/10/18/Component-Rules-101
For Extra Reading on Windows installer check these rules from the windows installer team(Rule 16: Follow Component Rules):
http://blogs.msdn.com/windows_installer_team/archive/2006/05/12/595950.aspx
Hurdle 3
You must decide whether or not you want to do patching of your files or upgrade everytime you install a new version.
I fortunately do not need to do patching, therefore for every upgrade I will just uninstall the previous version like this: How to implement WiX installer upgrade?.
This simplifies things greatly. When patching you are going to need full control over things. Something that is more difficult when you want automation/automatic building for by many developers that do not know anything about wix other than wix builds msi fields. It seems that this will work with using heat, but I am uncertain.
Hurdle 4
For me it took a little time to figure out why the include file with my preprocessor variables was not working. I finally figured it out. You must include the file in your .wxs file like this:
See here for more details:
Include a text file content into a WiX script
http://wix.sourceforge.net/manual-wix2/preprocessor.htm
Hurdle 5
Since I am using major upgrades with mix and I wanted to hook in subversion into my product number. This I thought would be quite simple and it is if you know how to do it. I had trouble here and had to get help for this as well.
Hurdle 6
With heat updating the files based on a bin folder everything works quite well. Well almost automatic, if there are any conflicts in dll references, etc then files will not be included in the installer and therefore things might not run properly. You need to have a testing process in place so as to test your program after installing. This can be critical if you miss a dll. I would also recommended that you keep track of the program's used dlls and compare against the msi file.
Now for the solution using HeatDirectory(from the help of Brian Rogers(wix team)):
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == ''">Debug</Configuration>
<OutputName>msbuild.heatfile</OutputName>
<OutputType>Package</OutputType>
<WixToolPath>..\..\YourHeatDir</WixToolPath>
<WixToolPath>$(WixToolPath)</WixToolPath>
<Cultures>en-us</Cultures>
<LinkerBaseInputPaths>..\..\data\HeatDir</LinkerBaseInputPaths>
</PropertyGroup>
<ItemGroup>
<Compile Include="product.wxs" />
<Compile Include="TestDir.wxs" />
</ItemGroup>
<Import Project="$(WixToolPath)\Wix.targets" />
<UsingTask TaskName="HeatDirectory"
AssemblyFile="$(WixToolPath)WixUtilExtension.dll" />
<Target Name="BeforeBuild">
<HeatDirectory
Directory="..\..\data\HeatDir"
DirectoryRefId="DataDir"
OutputFile="TestDir.wxs"
AutogenerateGuids="true"
ToolPath="$(WixToolPath)" />
</Target>
This needs to be pasted into your project file and appropriate settings/pathing needs to be changed. What this will do is take all the files in the path you specify
and create a TestDir.wxs file which you then can reference by component group. There are several Heatdirectory options see you need to see the heat sourcecode for details.
Just for reference: I found this post on stackoverflow after I posted my question: How to add a whole directory or project output to WiX package
Also for reference I just saw this article here: Harvesting a .csproj with heat.exe in Visual Studio 2008 and WiX(v3). and again this can be done using heat tasks instead.
Both discuss using third party tools. One called Mallow and another called Paraffin. I think paraffin looks supported and can do pretty much what heat does.
With a few additions: See here for details: http://www.wintellect.com/cs/blogs/jrobbins/archive/2007/10/18/wix-hints-for-new-users-part-1-of-3.aspx
Anyways hope this helps others.
