How to handle shared code with Git in this scenario?

Question

I'm currently in the process of trying to switch our version control to Git (from CVSNT). Surprisingly, it wasn't the distributedness or the concept of a staging area that I had problems with. But I am having quite a bit of trouble wrapping my head around the fact that AFAICT operations like branching, merging and tagging always are applied at the repository level rather than the file or directory level...

We reuse a lot of our code across different projects. My working area currently looks like this:

/Dev
  /Libs
    /LibA
    /LibB
    /LibC
  /Project1
  /Project2
  /Project3
  /WebDev

Now, let's say Project1 depends on LibA and LibB, Project2 depends on LibB and LibC and Project3 has no lib dependencies. Some of these libs are later compiled into DLLs (or BPLs - our primary development environment is Delphi), others are mere collections of reusable code that gets included on a file-by-file-basis into the main projects.

WebDev contains the code for our (mostly static) company web site which also contains information about Project1,2,3 and thus might have to be tagged together with them.

As I switch between projects quite a lot I usually have all of this checked out at the same time, updating the lib directories on-the-fly to the appropriate project branches where necessary.

How would I model this in Git and would it even make sense to stick to this way of working? I have read about git submodules but so far I fail to see how I would apply that here for several reasons:

• As far as I have understood, submodules would always be checked out inside their respective "superproject". However, we found managing multiple copies of (designtime) library code with Delphi to be a royal PITA, which is one of the reasons why we keep all the libs under a common directory outside the individual project trees. Additional copies are only ever checked out by build automation, never for doing any actual work on.

• I don't really want the libs to be "independent" from the projects: If I tag or branch one of the projects I always want to tag or branch the respective libs as well. When I want to go back to a specific tagged revision of the main project I want the libs to be reverted to that state as well. If possible tagging/branching/checking-out should always happen in a single step for a project and its dependencies.

I already made one attempt to put everything in a single Git repository with the library code primarily managed on the master branch, and the "projects" each on its own branch but whenever I try to merge lib changes between master and the project branches it pulls in all the files from the unrelated libs as well which is not what I want at all...

Do you guys have any ideas how best to resolve all this? I'm open to pretty much all suggestions, including a new layout for my working tree.

If anyone could point me to a real hands-on tutorial about submodules (or whatever other technique I'll need to accomplish this), that would be great, too.

Cutcliffe

Answer 1

• First: one Git repo per component (a component being a Lib or a Project, see Git Limits)
• Second: one parent project (Dev), under which you declare all your submodules.

That way, you can quickly checkout dev, and then init/update only the submodules on which you want to work on (including all of them if need be).
When tagging, the tag applied to the parent repo 'Dev' will reference precise commits of all submodules, meaning: if you revert to an older version, you will get back the exact submodules as they were in said previous state.

That organization would be a "system approach", where you can update any part of your components.

Plait

See also https://mcmap.net/q/12700/-git-submodule-update/… for more on submodules – Plait

Thanks, that sounds like an interesting approach! Would Push/Pull also be single operations applying to all components with this? – Cutcliffe

@Olivier: for fetch, if the config fetchRecurseSubmodules is set, then the fetch/pull will be recursive. Not sure for push though: see kerneltrap.com/mailarchive/git/2010/7/27/35353 – Plait

Thanks, the link to your other answer on submodules, explained some of the potential pitfalls with Push and submodules already. I'll play around with that a bit later this week. – Cutcliffe

I'm still a bit concerned about this quote "But the main idea remains: referencing specific components which: * have their own lifecycle * have their own set of tags * have their own development", none of which applies to (most of) our lib-components... that code usually never gets touched outside a project context and it doesn't have its own tags... – Cutcliffe

@Olivier: yes, a tag applies only to a given repo (not the submodules: they ignore all from the "MyApp1.0" you have applied above). That tagged repo then reference the exact commits you need. If you want a tag to be also applied on some of those components, you will need to apply said tag separately on those submodules. – Plait

@Olivier: the all vision (a tag for a git repo) is coherent with the initial vision for subproject as described in kerneltrap.com/mailarchive/git/2010/7/27/35367: strongly favor the early design decision we made back when Linus did his initial "gitlink" implementation, which is "separate project lives in a separate repository, and not having to check out any subproject should be the norm for using a superproject". – Plait

@VonC: Am I reading that right that a commit will never "recurse" into submodules? I can very well imagine cases where I would want to "bundle" changes in the lib-components with changes in the project-components that use them, e.g. when the signature of a library routine changes. While the submodule concept sure seems powerful I'm having more and more doubts that it is the right approach for what I'm trying to do... – Cutcliffe

@Olivier: correct reading. (but you can script it easily enough). The other approach would be the subtree strategy, but then you get back to one repo with everything else (which doesn't scale). Submodules is really for: I reference at all times the exact commits I need for my all project to work. If 3 of my compoents have changed, then I need to commit them individually (deciding for each one which branch and what to commit, with what message adapted to each component history). And then commit the parent project which will memorize the list of commits needed at that specific time. – Plait

@Olivier: on the subtree merge strategy: #1862720, #1084771, or #908587 – Plait

@Plait : FYI: I need a little more time to evaluate this and also the subtree merge (just in case you were waiting for the answer to be accepted). And BTW: I'm not French ;) – Cutcliffe

@Oliver: first, I am deeply sorry about your first name. I will be more careful ;) And take your time with your evaluation. If you come with a variation of that answer that better fits your need, you even can write your own answer (and accept your own answer!). I would find that very interesting. – Plait

@Plait : Sure will do. And no worries about the name: The only problem with that was that StackOverflow wasn't notifying me about your comments. – Cutcliffe