Xcode bot: git submodules not initialized

Question

I'm trying to set up an Xcode bot with OSX server. After configuring the git repository and creating a bot, I run integrate. It fails because the repository is checked out only partially. More precisely: Of the 6 submodules configured, only 3 are initialized and checked out.

didi@mac:/Library/Server/Xcode/Data/BotRuns/Cache/c5fda8f4-4d25-4d25-c18a-eb0b16a06692/source$ git submodule status
+c6e8420aec4147641fb1ca12d9f1d31bdd804e77 libs/asi-http-request (v1.7-320-gc6e8420)
-ae64a38766b42f236bb94f0e661cddb829f9ac43 libs/kraken
-7da02b323636bbaa0bbbf5b4eb229fcc07b1e14a libs/route-me
 152f9ee5576e710705a49032253d7d5af5366f9c libs/routing (152f9ee)
 347aaf74fe0c6388785095efdbf6397851514b7f libs/rtlabel (1.0-32-g347aaf7)
-562cf6b1c879f03546f5184e012cea15c4f159db libs/skmaps

(- means not initialized)

Looking at the bot log, it seems to just ignore the missing submodules in the submodule initialize part:

...
Checking connectivity... done
Submodule 'libs/asi-http-request' (https://github.com/BikeCityGuide/asi-http-request.git) registered for path 'libs/asi-http-request'
Submodule 'libs/routing' (ssh://[email protected]/var/repos/librouting.git) registered for path 'libs/routing'
Submodule 'libs/rtlabel' (https://github.com/BikeCityGuide/RTLabel.git) registered for path 'libs/rtlabel'
Cloning into 'libs/asi-http-request'...
...

No error message here or anywhere else. No single mention of e.g. "kraken" (name of a missing submodule) anywhere in the logs.

When checking out manually (clone, submodule init, submodule update), all submodules are initialized.

.gitmodules looks like this:

$ cat .gitmodules 
[submodule "libs/asi-http-request"]
    path = libs/asi-http-request
    url = https://github.com/BikeCityGuide/asi-http-request.git
[submodule "libs/rtlabel"]
    path = libs/rtlabel
    url = https://github.com/BikeCityGuide/RTLabel.git
[submodule "libs/routing"]
    path = libs/routing
    url = ../librouting.git
[submodule "libs/kraken"]
    path = libs/kraken
    url = ../kraken_ios.git
[submodule "libs/route-me"]
    path = libs/route-me
    url = ../route-me.git
[submodule "libs/skmaps"]
    path = libs/skmaps
    url = ../skmaps.git

The base repository and the 4 submodule repositories referenced with relative URL all need ssh authentication. The user set up in OSX server has access to all of them. The log of the bot contains no trace of trying to pull the missing submodules.

I can manually fix the local repository, but I'd like not to have the same issue again with new projects and new submodules added.

git version 1.8.4.2

I'm out of ideas.

Ouzel

> "The log of the bot contains no trace of trying to pull the missing submodules." Are there any error message in those logs? – Jana

The only error message is the one for the failed build because of missing source files. – Ouzel

And I suppose a git submodule update --init --recursive does work manually? – Jana

yes <filling min chars> – Ouzel

Answer 1

On Xcode 6.1.1 and OS X Server 4.0, it looks like they fixed the submodules with detached HEAD issue, but there are still some bugs. One of my project has one its submodules being completely ignored by Xcode Server so the app fails to build:

$ git submodule
 8a88bc41c9dc0f57c921d82bc4e7b93e1c4cbf7a InAppStore (heads/master)
 e4203f9f61d2546868c1274da5c7a0c56b87a737 Libraries (heads/master)  <--- IGNORED
 01902f255e6c3d90f0db41cb62dd2934098b98dd MixpanelTracker (heads/master)
 e2bee59accd817d50dff881a42c9e9afe307226f XLFacility (1.4.1-5-ge2bee59)

The fix for me was to add a pre-integration script trigger as such:

cd "$XCS_SOURCE_DIR/{YOUR_APP_REPO_NAME_WITHOUT_DOT_GIT_SUFFIX}"
git submodule update --init --recursive

At this point things were building although there were non-fatal warnings about the checkout not being clean or something like that. They went away when I changed to the bot's configuration to have "Cleaning" set to "Always".

Milzie

My problem was that the submodule was not being updated to the proper commit. This solution fixed it perfectly. I don't understand why Apple cannot get submodules working correctly, but at least the scripts give us workaround. – Pimply

cd ${XCS_PRIMARY_REPO_DIR} will change to the git root. And is generic enough to be used by all your CI scripts. – Breastplate

Answer 2

In Xcode 9 server, submodules are not initialized if none of the files of the submodules is referenced by one of the Xcode projects in the workspace.

This may happen, for instance, when you include a header file from a submodule that is not part of your code and therefore you added it to the "headers search paths" (c/c++) instead of adding it to the project.

A possible workaround is to add at least one file from the submodule to the Xcode project. It can even be a Readme.md file. This is easier than checking out submodules using a custom script because it relies on Xcode Server to store your GIT credentials.

Fructificative

Answer 3

Is I understand its a bug. I had the same problems and found that there is an issue with detached submodules. See the next article that show how to fix this issue (doesn't help on public repository) http://ikennd.ac/blog/2013/10/xcode-bots-common-problems-and-workarounds/

Hope this helps you.

Lydgate

The detached HEAD issues are fixed as of Xcode 6. See Pol's answer below for a workaround for current issues. – Pimply