.NET Dependency Management and Tagging/Branching

Question

My company is having trouble figuring out the best way to manage our builds, releases, and branches... Our basic setup is we have 4 applications we maintain 2 WPF applications and 2 ASP.NET applications, all 4 of these applications share common libraries, so currently they are all in one folder /trunk/{app1, app2, app3, app4}.

This makes it very hard to branch/tag a single application because you are branching all 4 at the same time, so we would like to separate it out into something like {app1,app2,app3,app4}/{trunk,tags,branches} but then we run into the issue of where to put the shared libraries?

We can't put the shared libraries as SVN externals because then when you branch/tag the branch is still referencing the trunk shared libs instead of having them branched as well.

Any tips? Ideas?

We are currently using svn and cruisecontrol.net.

EDIT: The shared libraries are changing often as of right now, which is why we can't use them as svn externals to trunk, because we might be changing them in the branch. So we can't use them as binary references.

Its also very hard to test and debug when the libraries are statically built instead of including the source.

Jankey

Answer 1

I guess it all depends on how stable the shared libraries are. My preference would be for the shared libraries to be treated as their own project, built in CruiseControl like the others. Then the four main applications would have binary references to the shared libraries.

The primary advantage with this approach is the stability of the applications now that the shared libraries are static. A change to the libraries wouldn't affect the applications until they explicitly updated the binaries to the newer version. Branching brings the binary references with it. You won't have the situation where a seemingly innocuous change breaks the other three applications.

Zambrano

The shared libraries are changing often as of right now, which is why we can't use them as svn externals to trunk, because we might be changing them in the branch. So we can't use them as binary references. – Jankey

If you are adding to the shared libraries I would put that new code in the application in the branch, then move it to the shared libraries only when needed by one of the other apps. Do the dev in the application until fully tested, then expose it to the other apps only when needed. Probably shouldn't change the shared libraries other than bug fixes - the functionality/interface of existing classes/methods should generally stay the same. Open/closed principle - add functionality, don't change existing. – Zambrano

These shared libraries are the core of our applications, if a bug is found, or a new feature comes up, it needs to change. For example, one of these shared libraries are some asp.net user controls that are used in both applications, so recently I was re-working the credit card/payment controls to add payment plan functionality (payment plans is a new business need). I obviously didn't want to change Payment controls in trunk because that would interfere with everyone elses work. So I needed to branch until I got it working and then merge it back into trunk. – Jankey

If all the applications are so tightly coupled, then maybe they should just be one giant solution? Or maybe two solutions - web and desktop? Then you could branch the entire group. Either way, I'd recommend reevaluating your entire approach to see if there is a better fit to what you are trying to do. – Zambrano

A giant solution is often easier. Or if the "giant" solution gets too big then have many solutions that reference just enough projects for their part to compile. However remember you should still build all the projects as a single giant solution even if you work with a lot of small ones. At my work we've made a small utility that reads the small solutions and generates a giant one as a step in our automated build process. That way we don't have to maintain the giant solution whenever we add or delete a project. – Wylie

Answer 2

Can you clarify why you don't like branching all four applications at the same time?

This makes it very hard to branch/tag a single application because you are branching all 4 at the same time

I usually put all my projects directly under trunk as you are currently doing. Then when I create a release branch or a feature branch, I just ignore the other projects that get carried along. Remember, the copies are cheap, so they're not taking up space on your server.

To be specific, here's how I would lay out the source tree you've described:

• trunk
  • WPF1
  • WPF2
  • ASP.NET 1
  • ASP.NET 2
  • lib1
  • lib2
• branches
  • WPF1 v 1.0
    • WPF1
    • WPF2
    • ASP.NET 1
    • ASP.NET 2
    • lib1
    • lib2
  • WPF1 v 1.1
    • WPF1
    • WPF2
    • ASP.NET 1
    • ASP.NET 2
    • lib1
    • lib2
  • lib1 payment plan
    • WPF1
    • WPF2
    • ASP.NET 1
    • ASP.NET 2
    • lib1
    • lib2

Handshake

The copy on the server is cheap but the checkout and copy on the developers computer is not cheap, most of the devs like to keep the whole repo checked out in case they have to work on a branch with someone, they'll already have it pulled down, but this starts taking forever when you have a few branches and tags going on. But so far, this seems to be the best solution :) – Jankey

In your situation, I would check out trunk or a single branch, whatever I'm working on. Then use switch to work on another branch. I might check out two working copies so I can leave one on my main work branch and use the other to switch around when doing small jobs on other branches. Something else to consider is shallow checkouts: check out the whole repository from the root, but don't recurse that checkout down to the branches until you need them. I find this more annoying because getting rid of a branch from your checkout is a hassle. – Handshake

I'd agree this is the correct approach. The devs keeping the whole repo checked out is a behaviour that can change - there is no reason to have a sub-standard solution just to satisfy what is effectively laziness, IMHO. In addition to that, while someone is working on experimental WPF 1.1 branch, I don't want to wait to get all those updates while I am just trying to work on one specific branch. – Chenee

I've tried working on projects that used both Brians and Dons methods of separating and I think doing what Don suggests is far easier in practice and also more correct in some respect. The method Brian suggests can quickly lead to version problems if you are not on top of it ALL the time. – Wylie

Answer 3

We are kicking off an open source project to try and deal with this issue. If anyone is interested in commenting on it or contributing to it, it's at:

http://refix.codeplex.com

Santos

See the question on Maven alternatives for a list of .NET tools to manage binary dependencies. However, I think you get the most benefit using those tools to manage third-party binaries. Personally, I prefer building libraries from source if my team wrote them. – Handshake

Answer 4

I agree with @Brian Frantz. There's no reason to not treat the shared libraries as their own project that is built daily and your projects take binary dependency on the daily builds.

But even if you want to keep them as a source dependency and build them with the app, why wouldn't the SVN externals approach work for you? When you branch particular app, there's no need to branch the shared library as well, unless you need a separate copy of it for that branch. But that means, it not a shared library anymore, right?

Nickienicklaus

The shared libraries are changing often as of right now, which is why we can't use them as svn externals to trunk, because we might be changing them in the branch. So we can't use them as binary references. The reason we would branch it is because we might be making important changes that would effect the shared libraries but we don't want to break the other projects until the change is fixed and tested in the branch, and then when we merge it back into trunk we'd fix the calls to it in the other apps. – Jankey

That is a disaster recipe. What happens when you branch two of your apps and you branch the shared libraries for both of them? Merging the shared libraries changes back can (and probably will) be a logistics nightmare. (Been there, done that :-)) – Nickienicklaus

Well, obviously that is a concern but we have to rely on good communication between our developers, there should be very few times they have to work on the same exact lines and if they do touch the same lines and they conflict, we have the diff tools to figure out what went wrong, its just part of developing with a team :) I think its much better to have the code branched so everything that they change is in that branch and not affecting other developers, the only time we branch is when we are going to be making breaking changes :) – Jankey

A breaking change in the shared libraries should not be made in a branch of one of the client apps. it should be made into a shared library branch, that has stabilized versions of all the client apps, so that you have full control over what's changing in the shared libraries. – Nickienicklaus

Answer 5

I've tried solving this problem several ways over the years, and I can honestly say there is no best solution.

My team is currently in a huge development phase and everyone basically needs to be working off of the latest and greatest of the shared libs at any given time. This being the case we have a folder on everyone's C: drive called SharedLibs\Latest that is automatically synced up with the latest development release of each of our shared libraries. Every project that should be drinking from the firehose has absolute file references to this folder. As people push out new versions of the shared libs, the individual projects end up picking them up transparently.

In addition to the latest folder, we have a SharedLibs\Releases folder which has a hierarchy of folders named for each version of each shared lib. As projects mature and get towards release candidate phase, the shared lib references are pointed to these stable folders.

The biggest downside to this is that this structure needs to be in place for any project to build. If someone wants to build an app 10 years from now, they will need this structure. It is important to note that these folders need to exist on the build/CI server as well.

Previous to doing this, each solution had a lib folder that was under source control containing the binaries. Each project owner was tasked with propagating new shared dlls. Since most people owned several projects, things often fell through the cracks for the projects that were still in the non-stable phase. Additionally TFS didn't seem to track changes to binary files that well. If TFS was better at tracking dlls we probably would have used a shared libs solution / project instead of the file system approach we are taking now.

Coloration

Have a look at Apache NPanday + Maven Release + Nexus Repository Manager (I'm a committer; and a user just asked us to make NPanday more visible, because it's great :-)) – Loser

Answer 6

Apache NPanday + Apache Maven Release

... might solve your problems

It gives you dependeny management (transitive resolving), strong versioning support, and automatic tagging/branching on 14+ version control systems, including SVN.

Give me a hint, if I should elaborate more.

I think there is no way you can avoid versioning and distributing your shared libs as separate artifacts, but Maven helps you alot doing that!

And you can allways do tricks to get it all opened in one Solution :-)

A sample workflow:

1. Dev 1 build A locally using Maven
2. Checks in sources
3. Build server builds A and deploys so-called SNAPSHOT-Versions to Repository Manager (e.g. Nexus)
4. Dev 2 two loads B, NPanday will automatically resolve the A-libs from the Repository Manager (No need to get the source and build)
5. Dev 1 wants to release A: Maven Release creates a branch or a tag with your source, finalizes the Version (removing SNAPSHOT) and deploys the artifacts to a Repository Manager.
6. Dev 2 can now upgrade B to use the final release of A (change entry in xml, or use VS-addin to do so)
7. Now Dev 2 can release B, again with automatic creation of tag or branch and deployment of built artifacts.

If you want to provide zipped packages as output from your build, Maven Assembly Plugin will help you do that.

Loser

Answer 7

You can use Apache/ IVY in standalone mode.

http://ant.apache.org/ivy/history/latest-milestone/standalone.html

I need to emphasize "stand alone" mode. If you google for examples....you will find alot of (not standalone) ones.

Basically, IVY works on this premise.

You publish binaries (or any kind of file, but I'll say binaries from this point forward).....as little binary-packages.

Below is PSEUDO code, do not rely on my memory.

java.exe ivy.jar -publish MyBinaryPackageOne.xml --revision 1.2.3.4 (<< where the .xml refers to N number of files that make up the one package.))

"Package" simply means a group of files. You can include .dll and .xml and .pdb files in a package (what I do with a DotNet build of assemblies). Or whatever. IVY is file-type agnostic. If you want to put WordDocs up there you could, but sharepoint is better for documents.

As you make bug fixes to your code, you increment the revision.

java.exe ivy.jar -publish MyBinaryPackageOne.xml --revision 1.2.3.5

then later you can retrieve from IVY what you want.

java.exe ivy.jar -retrieve PackagesINeed.xml 

PackagesINeed.xml would contain information about the packages you want.

something like "I want version '1.2+ of the MyBinaryPackageOne" (defined in xml)

As you build your framework binaries...you PUBLISH to IVY. Then, as you develop and build your code...you RETRIEVE from IVY.

In a NUTSHELL, IVY is a repository for FILES (not source code).

Ivy then becomes the definitive source of your binaries.

None of the "Hey, Developer-Joe has the binaries we need" kind of bull-mess.

.......

Advantages: 1. You do NOT keep your binaries in source control. (and thus do not BLOAT your source control). 2. You have ONE definitive source for binaries. 3. Through xml configuration, you say which versions you need for a library. (In the example above, if version 2 (2.0.0.0) of MyBinaryPackageOne is published to IVY (let's assume with breaking changes from 1.2.x.y)...then you are OK, because you defined in your retrieve (xml configuration file) .. .that you only want "1.2+". Thus your project will ignore anything 2+...unless you change the configuration package.

Advanced: If you have a build machine (CruiseControl.NET for example)....you can write logic to publish your (newly built) binaries to IVY after each build. (Which is what I do).

I use the SVN revision as the last number in the build number. If my SVN revision was "3333", then I would run something like this:

java.exe ivy.jar -publish MyBinaryPackageOne.xml --revision 1.2.3.3333

Thus whenever retrieve the package for revision "1.2.3+" .... I'll get the latest build. In this case, I would get version 1.2.3.3333 of the package.

It's sad that IVY was started in 2005 (well, that's the good news)...but that NUGET didn't come out til 2010? (2011?) Microsoft was 5-6 years behind on this one, IMHO.

I would never go back to putting binaries in source control.

IVY is very good. It is time proven. It solves the problem of DEPENDENCY management.

Does it take a little bit of time to get comfortable with it? Yep.

But it is worth it in the end.

My 2 cents.

.................

But idea #2 is Learn how to use NUGET with a local (as in..local to your company) repository.

That is the about the same thing as IVY.

But having looked at NUGET, I still like IVY.

Gaucho