Managing a complex repository structure - module dependencies and shared code [closed]
Asked Answered
T

1

7

(The following is a kind of a "theoretical MCVE" of the kinds of complexity I'm encountering in organizing source code I'm working on. You can treat it as a concrete problem and that would be good, or you can refer to the general concerns it brings up and suggest how to address them.)

Suppose I have modules of code A, B, C and D. A depends on B,C,D; B depends on C; C, D don't depend on other modules. (I use the term "modules" loosely, so no nitpicking here please).

Additionally, in all of A,B,C,D, a few identical header files are used, and perhaps even a compiled object, and it doesn't make sense to put these together and form a fifth module because it would be too small and useless. Let's have foo.h be one of the files in that category.

While all of these modules are kept within a single monolithic code repository, all is good. There's exactly one copy of everything; no linker conflicts between objects compiled with the same functions etc.

enter image description here

The question is: How do I make each of B, C, D into a version-managed repository, so that:

  • Each of them can be built with only the presence of the modules it depends on (either as submodules/subrepositories or some other way); and
  • I do not need to make sure and manually maintain/update separate versions of the same files, or make carry-over commits from one library to the next (except perhaps changing the pointed-to revision); and
  • When everything is built together (i.e. when building A), the build does not involve qudaruple copies of foo.h and a double copy of C (once for A and once for B) - which I would probably always have to make sure and keep perfectly synchronized.

Note that when I have a bit more time I'll edit this to make the question more concrete (even though I kind of like the broad question). I will say that in my specific case the code is modern C++, CUDA, some bash scripts and CMake modules. So a Java-oriented solution would not do.

Trajan answered 2/4, 2017 at 16:9 Comment(3)
How many hours does the complete system take to compile? If it is less then 2 or 3 hours, I think you should not try to split it up into subrepos.Creosote
@IanRingrose: It takes minutes, not hours, to compile. However, some of it has been/will be published independently (more than one such part); and there's the issue of separation of work on the different parts by different people, or by myself under different "hats".Trajan
Git has submodules, which allow you to refer to a specific commit of one repository from within another. (You can see the source of the submodule, but cannot change it without jumping through hoops.) It could be worth a look and there are loads of questions on SO about git submodules.Antheridium
A
1

VERY BRIEFLY, you might want to explore an artifact repository (Artifactory) and dependency management solutions (Ivy, Maven, Gradle). Then as shared-base-module is built, you stick it in the artifact repo. When you want to build the top-modules that depend on shared-base-module; the build script simply pulls down the last version of shared-base-module and links/compiles top-module against what it pulled down .

Anagrammatize answered 4/4, 2017 at 17:29 Comment(2)
(1) Can you add a link or two to where you would recommend I would read about these things? (2) Are "artifactories" and "dependeny management solutions" complementary or contradictory? Or orthogonal? (3) Aren't Ivy, Maven and Gradle very Java-oriented?Trajan
@Trajan yes they're for Java. It's possible to use them for javascript though but C I don't think so.Quintic

© 2022 - 2024 — McMap. All rights reserved.