How to fix a permission denied (publickey) error for a git submodule update in the Github Travis CI build?

Question

I cannot update the git submodule, with the error:

$ git submodule init
Submodule 'build/html' ([email protected]:quadroid/clonejs.git) registered for path 'build/html'
...
$ git submodule update
Cloning into 'build/html'...
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

But when I do the same tasks locally, everything is OK.

How do I fix this so that the Travis CI build passes and I can still click on the submodule in the repo to direct to it?

Barquentine

fixed by changing submodule url to git://github.com/quadroid/clonejs.git But now I can't push this module locally... – Barquentine

use a deploy key or have a script change the URL only in travis, not on your local – Reames

This article helped me awolski.com/sudden-permission-denied-publickey-error-in-travis In short, you have to disable and enable Travis builds for the repository. – Nimbus

Answer 1

This can (thankfully) be easily solved by modifying the .gitmodules file on-the-fly on Travis, so that the SSH URL is replaced with the public URL, before initializing submodules. To accomplish this, add the following to .travis.yml:

# Handle git submodules yourself
git:
    submodules: false
# Use sed to replace the SSH URL with the public URL, then initialize submodules
before_install:
    - sed -i 's/[email protected]:/https:\/\/github.com\//' .gitmodules
    - git submodule update --init --recursive

Thanks to Michael Iedema for his gist from which I derived this solution.

If your submodules are private repositories, it should work to include credentials in the https URLs, I recommend making a GitHub access token with restricted permissions for this purpose:

# Replace <user> and <token> with your GitHub username and access token respectively
- sed -i 's/[email protected]:/https:\/\/<user>:<token>@github.com\//' .gitmodules

Eolian

This is by far the best solution to this problem so far! – Galwegian

This doesn't seem to work for private repositories. All of these private repositories are under the same github account, and turned on for travis. Anyone has a clue? Thanks. – Magen

@Magen I haven't tried Travis with private repositories yet, but you'll need some way to give access to your repositories. It's kind of a problem in itself. – Eolian

@Magen Can you access your private repositories by combining HTTPS with user/password, i.e., https://user:[email protected]/organization/repo.git? If so, you can modify the sed command correspondingly. Let me know if this works for you. See Travis Pro documentation for reference. – Eolian

@aknuds Yes, Travis Pro documentation works. I created a build user on GitHub and added its ssh keys as suggested in the link. Also, DELETE the submodule hack suggested above. Thanks. – Magen

If you're on a Mac OS X system and you're wondering why sed fails, it's due macs specific sed version which required the -e option after the -i option. – Pfennig

I prefer to use another delimiter than / when replacing in URLs etc. sed -i 's|[email protected]:|https://github.com/|' .gitmodules – Obie

I agree this is a great solution, you can see the screenshot of it in action here twitter.com/DinisCruz/status/729580391686934528 – Turtledove

@carlin.scott I wonder if Travis has changed its behaviour then. I would check with them. – Eolian

Answer 2

I'd recommend using the https scheme for submodules, as that'll allow you to pull on Travis and push locally: https://github.com/quadroid/clonejs.git.

Torpid

Travis adding / at the end of path and says that repo isn't exists >_< – Ophiolatry

Answer 3

Travis now supports accessing submodule using ssh, which is by far the easiest solution. You only need to associate your ssh key (or the ssh key of a dedicated CI user) with the Github project you are building, as described in the documentation for private dependencies.

$ travis sshkey --upload ~/.ssh/id_rsa -r myorg/main

Note that Travis recommends creating a dedicated user so that you do not have to use your own ssh key.

Fort

To make this explanation even more complete; find a user with access to both the repository Travis is building and that of the submodule (Travis recommends creating a specific CI user for this), and then run this command for the repo doing the build. – Lucinalucinda

It's not available on travis.org (only travis.com, not free) – Charming

This doesn't really improve security does it... having your github key spread around.. – Excursionist

@matt I updated my answer to indicate that Travis recommends using a dedicated CI user for this - which I think is a good idea for several reasons. – Fort

Answer 4

You get this error because you specified your submodules via ssh-urls. For ssh access from the travis-ci environment you need to configure a key.

Alternatively, you can just use relative URLs for your git submodules since you project and your submodules are all available on Github.

Git resolves relative urls against the ORIGIN.

Example:

Using the first 2 entries from your .gitmodules:

[submodule "lib/es5-shim"]
        path = lib/es5-shim
        url = [email protected]:kriskowal/es5-shim.git
[submodule "build/html"]
        path = build/html
        url = [email protected]:quadroid/clonejs.git

Replaced with relative URLs:

[submodule "lib/es5-shim"]
        path = lib/es5-shim
        url = ../../kriskowal/es5-shim.git
[submodule "build/html"]
        path = build/html
        url = ../clonejs.git

Then when cloning - say - via https the origin is set like this:

$ git clone https://github.com/quadroid/clonejs.git
$ cd clonejs
$ git remote -v
origin  https://github.com/quadroid/clonejs.git (fetch)
origin  https://github.com/quadroid/clonejs.git (push)

When cloning via ssh:

$ git clone [email protected]:quadroid/clonejs.git
$ cd clonejs
$ git remote -v                                
origin  [email protected]:quadroid/clonejs.git (fetch)
origin  [email protected]:quadroid/clonejs.git (push)

With relative urls, the usual submodule sequence works independently of the origin:

$ git submodule init
$ git submodule update

Gardell

The relative URL's work very well in my case, and that is possibly the most elegant solution for public repos. Thanks for coming up with this technique! – Excursionist

Annoyingly however, it seems to break github's automatic hyperlinking to the submodules github page. – Excursionist

This doesn't work for private submodules. It seems that travis hides the fact that it's using a private token to access the repo but it doesn't do this when cloning submodules. – Deedeeann

Answer 5

You can also just directly manipulate your .gitmodules file via git. (Inspired by this answer).

git config --file=.gitmodules submodule.SUBMODULE_PATH.url https://github.com/ORG/REPO.git

Imparisyllabic

Answer 6

Actual on 2022:

1. Go to the repo settings in travis-ci and enable Clone or Import option for repos which used as submodule

2. Use https protocol in your .gitmodules (but better to use relative paths without a protocol)

3. Enjoy. I spent only 3 hours to figure it out

Ophiolatry