Mercurial subrepositories - managing more complex dependency hierarchies

Question

I have a master project that's using a fairly standard source tree approach + mercurial subrepositories.

Master
\lib - compiled binaries - things like log4net, AutoFac, etc
\source - VS solution, one folder per project, etc
\tools - stuff used during the build process

\source\contrib - contains any subrepos like so:

\source\contrib\Sub1
\source\contrib\Sub2

Master\.hgsub contains something like
source\contrib\Sub1 = https://myserver.com/Sub1

Now, it was recently determined that Sub2 needs some code from Sub1, and hence I have to adjust for this new dependency structure.

The problem of course is that if I follow the same approach as above and add Sub1 as a subrepos of Sub2, I end up with this ugly situation.

\source\contrib\Sub2\source\contrib\Sub1

Master now has 2 independent copies of Sub1!

So I know that I should be using relative paths when referencing subrepos (the RHS of the =) -- but that won't help out my scenario here as I understand it. I don't think there's any way to make the LHS live outside of the Master repository, which I think is what I really need here.

I have a couple of ideas on how to fix this, but none sits right with me, and I assume there has to be a better way. My ideal solution allows me to share the same sub-repo amongst multiple projects w/out paying the penalty of having multiple copies around. That just seems wasteful and inefficient in my scenario here (plus, I'd like to have all projects that have a dependency on Sub1 to use the same hg revision, rather than being revisioned independently)

1. Remove Sub1 as a subrepos of Master, and change any relative paths in the Master solution to reference the doubly nested Sub1. Not only is this path structure hideous, but if add a Sub3 to master that has a dependency on Sub1, I still have 2 copies of Sub1.

2. Compile a copy of Sub1 and just chuck it in the \lib directory. Sub1 is still undergoing some churn, and I'd rather build against the source versions. I don't want to pay the tax of constantly copying in new binaries to the source tree all the time (and bloating the tree).

3. Somehow break the dependency of Sub2 on Sub1. Based on the architecture of the repositories, this is probably not going to happen. Sub1 contains some very general purpose shared library code. Sub2 contains WCF service contract / interface / types that is needed in two very separate projects -- a client SDK and server implementation. At this point in time, it makes sense to keep these repositories separate.

Maybe I'm thinking about this wrong... or maybe I'm unaware of some hg trick.

Any help is appreciated.

Brickey

Answer 1

I'd say, forget about checking in binaries, that just leads to repository bloat. IMO, any output from a build should not be stored in the source repository.

How about teaching Sub2 to expect to find Sub1 at ../Sub1? Then, if you find you need to work on Sub2 independently of Sub1, create a "Sub2_standalone" repo, that pull in Sub1 and Sub2 as sub-repos.

So, when working on everything, you'd get:

Master/
Master/source/contrib/Sub1  
Master/source/contrib/Sub2

But when just working on Sub2:

Sub2_standalone/
Sub2_standalone/Sub2
Sub2_standalone/Sub1

Donatist

Well, that's not a totally terrible solution I suppose and I think that should solve the problem as long as I alway have dependent code in a 'sibling' type setup. Of course then I end up with an extra repository in Mercurial as well to keep the siblings together, but that might be the trade-off I have to make. Thanks for the idea -- I hadn't considered that. – Brickey

Not ready to mark it an answer quite yet though -- hoping someone from selenic will chime in with a 'best practice' ;0 – Brickey

An extra repository should not hurt, since it should use hardlinks for the in-repository files. – Lepidopterous

I understand it won't hurt from a disk space perspective... no worries there. I'm just thinking it could add a bit of confusion if you're looking at the list of repositories from the web view. But that's not something many users will be doing other than me at this point... – Brickey

Ok -- ended up implementing this approach and so far, so good. Solutions assume any source dependencies are in sibling folders using relative paths. I also managed to pull our folder full of build tools into its own repository as well. 'Usable' projects for pulling end up being just a wrapper around X subrepositories with a .hgsub specifying relative paths - i.e. Sub1 = ../Sub1 -- and a solution file in some cases to pull in the appropriate projects from the subs. This has the nice side effect of making subrepository remapping unnecessary should we change back end servers as well. – Brickey

Answer 2

If you need that nested structure, you could use a symlink for Sub1 below Sub2, to ensure that both versions of Sub1 are always at the same revision. Then you actually have only one version of Sub1, which seems to be what you want.

On GNU/Linux it wouly be a simple ln -s source/contrib/Sub1 source/contrib/Sub2/source/contrib/Sub1

Lepidopterous

Yeah, we're on all on Windows here. I had considered something like that, but from what I understand, symlinks don't get translated to NTFS junctions -- they translate over to some sort of file. I'd prefer to not go that route – Brickey

I've actually used this tactic within Windows to same effect via the LinkShell tool (lifehacker.com/5530931/…) – Ezmeralda