Best practice for dependency management in a project with a huge number of dependencies

Question

Our project is like an adapter/facade interface for a huge amount of different other libraries. The dependencies are somehow overlapped, sometimes in conflict or sometimes even make project breaks in silence for wrong version of dependencies provide wrong behavior of same interface. We are using Ivy and Ant to do basic dependencies management. What's the best practice to manage dependencies and detect wrong behavior early on?

Trollope

Are the other libraries written by you or are they third party? – Akins

When you have conflicting versions, and the versioning is provided by a transitive dependency management system, you're kind of stuck between a rock and a hard place, and likely need to do something like a dependency tree (I don't know how to do that with Ivy) and start figuring out the conflicts manually. Even with Maven you sort of have to do things manually, although it can tell you why a library was omitted due to conflict, e.g. maven.apache.org/plugins/maven-dependency-plugin/examples/… – Leaguer

@DaveSchweisguth They are mainly our own libs but manage by different teams or even across teams. – Trollope

If your internal libraries provide "wrong behavior" for an interface, or fail silently, no dependency management system is going to be able to do much about that. – Floruit

@Floruit It is due to wrong version provided. I was looking for some guideline for this. I understand that no system alone can do the trick... :( – Trollope

Answer 1

The important part of this question is about process, not tools.

If a project's dependencies are owned by other teams or third parties, that project must explicitly accept each new version of each new dependency. Allowing dependencies to upgrade themselves would allow them to break the depending project without warning, which is what it sounds like is happening.

Each dependency must release known versions, whether as binaries or tags in version control or whatever is appropriate to your stack. Each time a project wants to upgrade a dependency, it must test the result of the upgrade. (As usual comprehensive automated testing will be a big help.) If it fails (either because the dependency is just broken or because the dependency brings in an incompatible version of a transitive dependency), abandon the upgrade, report the problem to the owners of the dependencies, and try again after they've released a version which fixes the problem. If it succeeds, change the version of the dependency that the project uses in its build configuration.

Ideally a project will upgrade dependencies one at a time and fully test each upgrade separately. If it's necessary to upgrade more than one dependency all at once (perhaps because two dependencies both depend on a third dependency of which there can only be one version in the system) that's fine, although it's a bigger change and thus riskier. If your project has transitive dependencies like this, it will be worth the engineering effort to make them backward-compatible over as many versions as is reasonable.

Of course many tools support this process easily enough: just pin each dependency to a specific version.

Akins