Gitlab repository mirroring

Question

Is it possible to have gitlab set up to automatically sync (mirror) a repository hosted at another location?

At the moment, the easiest way I know of doing this involves manually pushing to the two (gitlab and the other) repository, but this is time consuming and error prone.

The greatest problem is that a mirror can resynchronize if two users concurrently push changes to the two different repositories. The best method I can come up with to prevent this issue is to ensure users can only push to one of the repositories.

Penneypenni

This has been requested at: feedback.gitlab.com/forums/176466-general/suggestions/… Go and upvote it. – Cabernet

Answer 1

Update Dec 2016: Mirroring is suported with GitLAb EE 8.2+: see "Repository mirroring".

As commented by Xiaodong Qi:

This answer can be simplified without using any command lines (just set it up on Gitlab repo management interface)

---

Original answer (January 2013)

If your remote mirror repo is a bare repo, then you can add a post-receive hook to your gitlab-managed repo, and push to your remote repo in it.

#!/bin/bash
git push --mirror [email protected]:/path/to/repo.git

As Gitolite (used by Gitlab) mentions:

if you want to install a hook in only a few specific repositories, do it directly on the server.

which would be in:

~git/repositories/yourRepo.git/hook/post-receive

---

Caveat (Update Ocotober 2014)

Ciro Santilli points out in the comments:

Today (Q4 2014) this will fail because GitLab automatically symlinks github.com/gitlabhq/gitlab-shell/tree/… into every repository it manages.
So if you make this change, every repository you modify will try to push.
Not to mention possible conflicts when upgrading gitlab-shell, and that the current script is a ruby script, not bash (and you should not remove it!).

You could correct this by reading the current directory name and ensuring bijection between that and the remote, but I recommend people to stay far far away from those things

See (and vote for) feeadback "Automatic push to remote mirror repo after push to GitLab Repo".

---

Update July 2016: I see this kind of feature added for GitLab EE (Enterprise Edition): MR 249

• Add ability to enter remote push URL under Mirror Repository settings
• Add implementation code to push to remote repository
• Add new background worker
• Show latest update date and sync errors if they exists.
• Sync remote mirror every hour.

Note that the recent Remote Mirror Repository (issues 17940) can be tricky:

I'm currently trying to shift main development of the Open Source npm modules of my company Lossless GmbH (https://www.npmjs.com/~lossless) from GitHub.com to GitLab.com

I'm importing all the repos from GitHub, however when I try to switch off Mirror Repository and switch on Remote Mirror Repository with the original GitHub URL I get an error saying:

Remote mirrors url is already in use

Here is one of the repos this fails with: https://gitlab.com/pushrocks/npmts Edited 2 months ago

turns out, it just requires multiple steps:

• disable the Mirror Repository
• press save
• remove the URl
• press save
• then add the Remote Mirror

Malt

What happens in this setup while the remote repo is not available, e.g. due to maintenance or network hickups? I'd guess that either it won't be possible to push to the gitlab repo, or the commits pushed will be missing in the remote repo? – Satiny

@Satiny the latter: your commits would still be pushed to your GitLab repo, but wouldn't make it to the slave. – Malt

Today this will fail because GitLab automatically symlinks github.com/gitlabhq/gitlab-shell/tree/… into every repository it manages. So if you make this change, every repository you modify will try to push. Not to mention possible conflicts when upgrading gitlab-shell, and that the current script is a ruby script, not bash (and you should not remove it!). You could correct this by reading the current directory name and ensuring bijection between that and the remote, but I recommend people to stay far far away from those things. – Cabernet

Another option is to setup a new github remote for each repo. – Cabernet

@CiroSantilli good points. I have included those in the answe for more visibility. – Malt

@Malt Is the mirroring repo function available for the CE version of GitLab (gitlab.com) nowadays? – Trumpeter

@XiaodongQi It is now available in CE as of version 8.15(.2?) I followed these instructions to sync between my primary and secondary servers. – Debidebilitate

@Carel, Yeah, it's working now. I also did a couple of mirror repos there :) So, this answer can be simplified without using any command lines (just set it up on Gitlab repo management interface). – Trumpeter

@XiaodongQi Thank you. I have included your comment in the answer for more visibility. – Malt

Answer 2

If not hosting your own GitLab, it's worth knowing GitLab.com has introduced this feature directly, without any workarounds.

1. From within a project use the gear icon to select Mirror Repository
2. Scroll down to Push to a remote repository
3. Checkmark Remote mirror repository: Automatically update the remote mirror's branches, tags, and commits from this repository every hour.
4. Enter the repository you want to update; for GitHub at least you can include your username and password in the URL, like so: https://yourgithubusername:[email protected]/agaric/guts_discuss_resource.git

Note that if you are pulling from a remote repository, it will still push on to the remote repository set here. I haven't tried it, but you should be able to push to and pull from the same repository.

Karilla

Just being paranoid here, but how safe is it that your password is sent as part of the URL request? I know the https will ensure SSL is used, but does that encrypt the actual URL string being requested (which would include the password in this case?) – Tenderloin

Looks like you can use a github personal access token in place of the password. GitLab seems to have introduced documentation on this recently, see here: gitlab.com/help/workflow/… – Tenderloin

@Tenderloin For your question - "Just being paranoid here, but how safe is it that your password is sent as part of the URL request? I know the https will ensure SSL is used, but does that encrypt the actual URL string being requested (which would include the password in this case?) ", the answer is that the credentials will get encrypted (See #2717490 for more details) – Intellect

Answer 3

A quick summary of configure mirroring repositories with GitLab and GitHub

1. GitHub: Settings > Developer settings > Personal access tokens
   • You need enable: public_repo
2. GitLab: GitLab project > Settings > Repository > Mirroring repositories

Push

• Git repository URL: https://[email protected]/username/repo.git
• Mirror direction: Push
• Authentication method: Password
  • Password: your_github_token

Pull

• Git repository URL: https://github.com/username/repo.git
• Mirror direction: Pull
• Authentication method: leave_empty

Official GitLab documentation

Tagliatelle

Mirroring from GitLab to GitHub over SSH connection is explained here. – Utimer

So the Push Mirroring doesn't work for private repositories? – Dissatisfied

Answer 4

I also created a project to mirror repositories in GitLab 6 through the API (API mostly used on project creation only).

https://github.com/sag47/gitlab-mirrors

Handal

Answer 5

The best option today is to use GitLab CI. It is essentially an already implemented server for the webhooks, which automatically clones for you and let's you run arbitrary shell commands: all you have to do then is to push.

services are a the best option if someone implements them: they live in the source tree, would do a single push, and require no extra deployment overhead.

The key implementation difficulty now is how to store the push credentials safely: likely the best option for GitHub is to get a key somehow (Oauth on UI through the service would be perfect) and store that plaintext.

Another option which has just been added are custom hooks.

Cabernet

Answer 6

UPDATE 2020

As you all know about Free Gitlab Mirror Support Ended on March 22 for external private Repositories.

With cron automation here is little code in php:-

https://github.com/DeepLinkCode/GithubToGitlab

Burge

Answer 7

You can use hooks to customize a script that runs after some commit. With that you can send the new changes to another repository. Look for more information about hook in the following page: http://git-scm.com/book/en/Customizing-Git-Git-Hooks

Deeannadeeanne

Answer 8

The best option is not to use post-receive hooks, but deployments commands that perform synchronisation via rsync, using Capistrano if you like Ruby, using Shipit if you prefer Javascript (Grunt).

Nkrumah

Answer 9

GitLab Enterprise Edition now supports repository mirroring as of version 8.2. Information about how to configure this is in the Repository Mirroring help topic.

Theresa

Answer 10

GitBitLabHub allows you automatically mirror repositories between Bitbucket / Gitlab / Github using simple webhooks.

1. Generate an ssh key for source and destination repositories:

ssh-keygen -t rsa -f ~/.ssh/project_id_rsa

• It will generate 2 keys, the PRIVATE key to ~/.ssh/project_id_rsa and the PUBLIC key to ~/.ssh/project_id_rsa.pub.
• The PUBLIC key is used as Deploy Key while the PRIVATE key should be used in SRC_DEPLOY_KEY and DEST_DEPLOY_KEY env variables.

2. Add this PUBLIC key as a Deploy Key for the source and destination repositories. Depending on the platform it's called Deploy Key or Access key. See how to add Deploy/Access Key to bitbucket, bitbucket access keys, gitlab, github.

• Add the PUBLIC key to the source repo with read-only access.
• Add the PUBLIC key to the destination repo with write access.

3. Run this container in docker with all environment variables configured properly.

docker run -it \
    -e [email protected]:vendor/src_repo.git \
    -e [email protected]:2222/vendor/dest_repo.git \
    -e SRC_DEPLOY_KEY=base64_encoded_key== \
    -e DEST_DEPLOY_KEY=base64_encoded_key== \
    -p 8181:8080/tcp \
    karser/gitbitlabhub

• Use the PRIVATE key for SRC_DEPLOY_KEY and DEST_DEPLOY_KEY env variables. Don't forget to encode it with base64: base64 -w 0 < ~/.ssh/project_id_rsa
• If you configured everything properly it should mirror the repo after the first launch. If it didn't see the logs in the container output.

4. Create the webhook in the source repository. Using default settings should be enough. See how to create a webhook in bitbucket, gitlab, github.

Pfeiffer