GIT Nested repositories: Composer vs. SubModules vs. Subtree vs.?

Question

I've finally incorporated GitHub and Composer dependency management on my workflow. It's definitely a huge step forward, although I remain very confused about GIT managing the "nested" dependecies.

As I'm using an awesome Wordpress Stack ROOTS/BEDROCK, my simplified directory structure looks like this:

|-- /project
|   |-- /.git                    // git repository for the skeleton/stack of the project
|   |-- composer.json            // list of dependencies, most of them are my own repositories on GitHub
|   |-- /vendor
|   |   |-- /php-dependency-1    // 3rd party dependencies not directly related to Wordpress
|   |-- /web
|   |   |-- /app                 // acts as "wp-admin" folder
|   |   |   |-- /mu-plugins       
|   |   |   |   |-- /SUBREPOSITORY-1    // my own framework feature, public, GitHub
|   |   |   |   |-- /SUBREPOSITORY-2    // my own framework feature, public, GitHub
|   |   |   |-- /plugins
|   |   |   |   |-- /SUBREPOSITORY-3    // my own plugin, public, GitHub
|   |   |   |-- /themes
|   |   |   |   |-- /SUBREPOSITORY-5-PARENT-THEME    // parent theme used on my framework, public, GitHub
|   |   |   |   |-- /SUBREPOSITORY-6-CHILD-THEME     // work for client, private, BitBucket
|   |   |-- /wordpress           // Wordpress CMS
|   |   |   |-- /wp-admin
|   |   |   |-- /wp-includes

"Subrepositories" are defined in my composer.json on the root of the project and are downloaded from GitHub by composer install. So far so good.

But! I expect to tweak my parent-theme and some mu-plugins a lot, I need to be able to push/commit from each of my projects they will be included. As you know, you can't really test wordpress theme without a wordpress installation ...

So ... which way to go? There IS A LOT of posts about this topic and most of them mention SubModules, but if I get the idea of Composer correctly, they are kind of in conflict with each other.

Just use nested .git repositories seem great for my case, altough it does not seem to work - if I try to push/commit nested repo, either "everything is up to date" or I get messages such as Your branch is ahead by 1 commit. So just "nesting it" is a no a go?

Thanks in advance and sorry for a little confused tone of the question, I drowned a little in the topic. :) Any help would be much appreciated.

Gladisgladney

This question seems highly opinionated. I would still want to hear different approaches. – Lavoisier

By definition, a "question" cannot be opinionated. Only an answer. This question is straightforward (Is nesting a no-go), and the answer was informative. But to be fair, perhaps the post was also edited. – Fain

Answer 1

I have a couple of questions, and in consideration of that, the answer below is quite generic. If you answer my questions, I will gladly update it.

1. Does composer pull the repos on an update? OR reclone the repo? (Is it even doing updates?)

   • If it reclones then using it for updates will risk overwriting your working tree on those subrepos (use it for install ONLY or remove it all together)
   • If it isn't doing updates (or dependency recursion -- see below), then it is adding unecessary complexity to your project (remove it and use one of the options below)

2. Is composer actually doing any dependency management (i.e. recursing to find nested dependencies)? Or is it simply cloning the git projects as subfolders?

   • If it is, then yes, submodules may be innapropriate for your case, as they are an alternative dependency management system, but if your subprojects also manage their dependencies with submodules then doing a git clone --recursive should work for managing them as well

3. Do you want your master project to track new changes to your subprojects?

   • If yes: have a look at option #2: subrepositories
   • otherwise: try option #1: submodules
   • [there is a third option which I will link to, but I haven't used it so can't explain in detail (comments/edits appreciated)]

Option #1: Submodules

You can also manage an individual submodule by cd LOCAL_DIR_NAME_I and using normal git commands

1. Setting up:

git submodule add REMOTE_URI_1 LOCAL_DIR_NAME_1
...
...
git submodule add REMOTE_URI_N LOCAL_DIR_NAME_N
git commit -m "Add submodules..."

2. Cloning (the main project)

   git clone MAIN_URI REPO && cd REPO && git submodule update --init --recursive

   --init will copy the configuration from .gitmodules to .git/config before performing the update (if necessary), and --recursive will do that action recursively in each submodules.

   or

   git clone --recursive MAIN_URI

   --recursive tells git to update and init all submodules on cloning

3. Updating (will preserve unsaved changes)

   • Local copy has no un-pushed changes (updates all submodules by default):

   git submodule update [LOCAL_DIR_NAME_I ... LOCAL_DIR_NAME_J]

   • Local copy has un-pushed changes (updates all submodules by default):

   git submodule update --remote --rebase [LOCAL_DIR_NAME_I ... LOCAL_DIR_NAME_J]

4. Publishing/Pushing

This tries a submodule push first and if successful performs a main project push

git push --recurse-submodules=on-demand

Option #2: Subrepositories

You've said you have problems using this method, but I don't understand what they are. Please elaborate if possible.

(the git book also talks about subrepos, but I can't for the life of me find where, right now; let me know if you find it)

1. Setting up:

NOTE: the master repo will not track changes to subrepo's .git, just to the files themeselves

The slash (/) after the directory name is essential to avoid creating a submodule

git clone REMOTE_URI_1 LOCAL_DIR_NAME_1 && git add LOCAL_DIR_NAME_1/
...
...
git clone REMOTE_URI_N LOCAL_DIR_NAME_N && git add LOCAL_DIR_NAME_N/
git commit -m "Add subrepos..."

If using Composer: (and it is doing the clones for you) you can simply do the adds and commit, but maybe you can configure composer to do this as well.

2. Management

Manage an individual subrep by: `cd LOCAL_DIR_NAME_N' and use normal git commands

Remember that changes to your subrepo files will be tracked by your main repo

The biggest issue here is with cloning (if you want colaborators to be able to work on the subprojects) since your subrepo .git files aren't tracked. Including a file, remote.info in each subrepo that stores the remote should solve this. Then add a script to your main repo that does for each subdirectory cd SUBDIR && git init && cat remote.info | xargs git remote add origin. Depending on what Composer is doing (see questions above) you may want to add a composer update command to this script

Option #3: Subtree Merging

I'm not entirely confident on the subtleties of this method, so I will just link to it

Try this link for a bit of a tutorial

Option #N: Any way you want

Of course you could setup numerous other work flows that in some cases might be simpler. For example you could stick with Composer for dep management and clone your subprojects outside of your main project, creating untracked symlinks in the main project to allow easy access to those files when working on the main project. This could be automated with a script (as could a batch push of all these repos). You could probably even parse composer.json to automatically do this for new (git-based) dependencies.

Note: It seems to me that you don't need to be using Composer at all. If this assumption is incorrect, it is possible that none of the 3 options above will solve your problems.

Dementia

Dylan, that's one hell of a comprehensive answer! Thank you, whenever I will have more time on my hands, I will incorporate your suggestions to my workflow. Have a great day! – Gladisgladney

Kudos for an answer that explains options, rather than advocating for a particular solution. Well said! – Glaciate

Answer 2

@DylanYoung already provided a fantastic answer, but I just had this same question/problem come up when I started looking at how to use Bedrock with Git and think I might be able to provide some extra ideas that would be useful for consideration by people who stumble upon this post.

Like Dylan, I came to the conclusion that composer was not only an unnecessary complication, but probably also a recipe for disaster. There are various reasons for this, but the main one is how to deal with commercial plugins. Bedrock gives some options and walks through using Git repos for each commercial plugin, but they ignore how you would actually go about updating the plugins. The only possibility would be to manually download the zip file from the developer upon each update, then add it to its' git repo, only to then be re-pulled into the project via Composer. Insanity.

To clarify, Composer seems like a fantastic tool, but moreso for "proper" PHP projects, which Wordpress surely does not qualify as. Bedrock qualifies much more as "proper" PHP - via the various other tools (Acorn, Sage etc...) from Roots that integrate WordPress with Laravel. But if you aren't using those tools, then you don't really need Composer.

But if you aren't using Composer, how do you do updates!? Through WordPress' own Composer: WP Admin Updater or, for a git-oriented workflow, WP-CLI. WP-CLI allows for specifying patch, minor and major update versions, and plenty more - especially via 3rd party WP-CLI packages. Moreover, I suspect it properly handles plugin updates, which sometimes make changes to the DB (something that composer wouldn't have any knowledge of). Finally, you could surely have a script/custom package that automatically updates plugins one-by-one and commits them to git with a message that says its version number - this would allow for easy tracking of changes as well as reverting the commit/update if necessary.

For any custom plugins/code, you can either include it in the monorepo, or use a submodule as Dylan suggested in his fantastic answer. I'll likely do the latter as I might end up making some of the private plugins commercially available later. Submodules could likewise be used for any plugins that have a public git repo that you are contributing to.

I will still use Bedrock because I like how it structures everything, but other than that I'll be sticking to WP + Git.

I hope this saves some time for people who struggle with this question in the future. I wasted days trying to make sense of this until I realized that Composer was a solution looking for a problem.

edit: One extra note - many WP plugins use Composer for their own development process, and bundle it in the plugin package. That's a completely separate thing, because a plugin is a standalone project whereas the entire WP site is a project of projects.

Teno

As long as you don't need it, that's fine. But plugins that consider themselves to be the king of all crowns and not considering that all dependencies need to be managed together are just silly, however composer wordpress-plugin package type (and installer) come to the rescue here and that's likely the reason you find it documented. – Murrelet

Again, the point is that WordPress - certainly plugins, but also core - is not real php. We generally don't have a choice but to use "silly" plugins that don't track dependencies etc.. And, again, composer has no mechanism to update commercial plugins - hence wp cli is a better toll for the job. – Teno