git-bisect, but for N repos

Question

Our software is modular and I have about 20 git repos in one project.

If a test fails, it is sometimes hard to find the matching commit since several developers work on these 20 repos.

I know the test worked yesterday and fails reproachable today.

Sometimes I use git-bisec, but this works only for one git repo.

Often changes in two git repos make a test fail.

I could write a dirty script which loops over my N git repos myself, but before doing so, I would like to know how experts would solve this.

I use Python, Django and pytest, but AFAIK this does not matter for this question.

Iver

Possible duplicate of #9712092 – Ass

@Phillip's link looks golden for this, but let me just say "this is why you use submodules. this is almost the point of submodules: record which commits go together to build a project from multiple bases". If you'd used submodules, bisect would work perfectly.". – Mock

Try to narrow the search manually: find moment when complex application was OK, and then it got broken. Is it possible to make list of commits for the whole in one line and go thought it? – Pudency

@Mock up to now we use not git submodules. I think the problem can be solved without them, too. Maybe a bit harder, but not impossible. – Iver

@Mock I agree that submodules are a good idea, but I don’t think I could convince anybody else and projects big enough to need them are also usually big enough to have multiple devs. – Pelagic

Do you use a single build to deploy your whole project? Can you please describe the directory structure. I might be able to help since I have worked for quite some time on git-bisect. – Schilling

We have all our own code in ~/src in a python virtual environment. Third party code gets installed via pip. – Iver

Answer 1

I personally prefer to use repo tool to manage complex projects. Put those 20 repos in manifest.xml and each time when build starts create patch manifest if build fails do repo diff manifests to see what was changed and where.

Sympathetic

the repo-tools looks cool. Does it work for non-android projects, too? We work on a django application. – Iver

Yes is repositories are based on .git – Sympathetic

Answer 2

There is category of QA tool for "reverse dependency" CI builds. So your higher level projects get rebuilt every time a lower level change is made. At scale it can be resource intensive.

The entire class of problem is removed if you stop dealing with repo-to-repo relationships and start following version release methodology for the subcomponents. Then you can track the versions of lower-level dependencies and know when you go to upgrade that it broke. Your CI could build against several versions of dependencies if you wanted to systematize it.

Git submodules accomplish that tracking for individual commits, so you again get to decide when to incorporate changes from lower levels. (Notably, that can also be used like released versions if you only ever update to tagged release commits.)

Gainor

Which tool of this category of QA tool for "reverse dependency" CI builds do you use? – Iver

My current shop previously did it with Jenkins scripting on our old CI server. – Gainor

We already use Jenkins. It pulls the N repos from us (all from master (agile)) and tries to test the particular project. A project is very small, mostly configuration, only few coding lines. What tool do you use now? – Iver

We switched to Travis CI with more formal SemVer releases. – Gainor