Bitbucket clone all team repositories

Question

I'd like to clone all my bitbucket team repositories using a bash script and http. I have found a few examples that use the Bitbucket api however, they all seem to return no repositories. Any ideas? Using mac.

Gondola

Are you authenticated at all? And does the user you're authenticating as have access to the Team repositories? If not I recommend you set up an App Password (at bitbucket.org/account/user/<YOUR_USERNAME>/app-passwords) that has sufficient permissions (read access to account/team membership/projects/repositories should cover most of the things you need). – Delcine

Answer 1

Here is my simple solution. Make file downloader.sh

#!/bin/bash

USER=${1}
TEAM=${2}

rm -rf "$TEAM" && mkdir "$TEAM" && cd $TEAM

NEXT_URL="https://api.bitbucket.org/2.0/repositories/${TEAM}?pagelen=100"

while [ ! -z $NEXT_URL ] && [ $NEXT_URL != "null" ]
do
    curl -u $USER $NEXT_URL > repoinfo.json
    jq -r '.values[] | .links.clone[1].href' repoinfo.json > ../repos.txt
    NEXT_URL=`jq -r '.next' repoinfo.json`

    for repo in `cat ../repos.txt`
    do
        echo "Cloning" $repo
        if echo "$repo" | grep -q ".git"; then
            command="git"
        else
            command="hg"
        fi
        $command clone $repo
    done
done

cd ..

you can run it by:

sh downloader.sh username teamname # or username instead team name

Diatribe

Warn: it will ask you for a password every 100 repos – Vaenfila

@chillappreciator do not use http protocol. Use git@ protocol and ssh keys. – Diatribe

This works nicely, but when bitbucket has less than 100 repos, after downloading all the repos, say just 6, the script asks me for a password again, and if I enter it the script will delete all the previously downloaded folders (of course I could press Ctrl-C and all would be fine). I tried setting pagelen=2 and this behaviour no longers occurs. – Hairsplitting

Solution to my comment above: Change the while loop to check for $NEXT_URL != "null" ........... while [ ! -z $NEXT_URL ] && [ $NEXT_URL != "null" ] – Hairsplitting

Thanks for the script. Small addendum: bitbucket stopped using username+password as authentication method. To make your script work, you need to create an app password and use username:apppassword as username in your command line. Username is found in the account settings. Using e-mail address with app password will lead to an error message. – Bradley

Answer 2

Ensure you have your ssh keys setup in bitbucket and git is installed by cloning one repo manually:

This will clone all repos that are owned by you:

USER=bitbucket_username; curl --user ${USER} https://api.bitbucket.org/2.0/repositories/${USER} | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone

To backup your team repositories use the same script but hard code your bitbucket team name like this:

USER=bitbucket_username; curl --user ${USER} https://api.bitbucket.org/2.0/repositories/TEAMNAME | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone

Here's a better way:

curl -u ${1} https://api.bitbucket.org/1.0/users/TEAMNAME > repoinfo

for repo_name in `cat repoinfo | sed -r 's/("name": )/\n\1/g' | sed -r 's/"name": "(.*)"/\1/' | sed -e 's/{//' | cut -f1 -d\" | tr '\n' ' '`
do
    echo "Cloning " $repo_name
    git clone [email protected]:TEAMNAME/$repo_name.git
    echo "---"
done

Amaryl

Just for those who may be trying to use this. make sure that ssh: is the proper signature for your repository. For me it was git:. See like this : USER=myUserName; curl --user ${USER} https://api.bitbucket.org/2.0/repositories/THETEAM | grep -o '"git@[^ ,]\+' | xargs -L1 git clone – Brunildabruning

in my mac I downloaded gsed and replaced sed with gsed in the code – Moxa

Answer 3

Another alternative using jq.

#!/bin/bash

user=username:password

curl -u $user 'https://api.bitbucket.org/2.0/user/permissions/teams?pagelen=100' > teams.json

jq -r '.values[] | .team.username' teams.json > teams.txt

for team in `cat teams.txt`
do
  echo $team

  rm -rf "${team}"

  mkdir "${team}"

  cd "${team}"

  url="https://api.bitbucket.org/2.0/repositories/${team}?pagelen=100"

  echo $url

  curl -u $user $url > repoinfo.json

  jq -r '.values[] | .links.clone[0].href' repoinfo.json > repos.txt

  for repo in `cat repos.txt`
  do
    echo "Cloning" $repo
    git clone $repo
  done

  cd ..

done

Wistrup

I couldn't get this to work with username:password, but it worked well with a --netrc-file. Not sure about deleting the whole team directory every time... probably better to pull in new changes, but it's not trivial to update the script for that. – Youthen

Answer 4

You can only use a simple command if you have less than 100 repositories and add 'pagelen=100' to the query, since that is the most that the bitbucket API will report at one time. If you have more than 100 repositories, you need to process the "next" link in the JSON returned, to get the URL to query the next set of repositories, which would be easier with a script.

If you use http to clone instead of ssh, then you either need to enter the password for protected repositories, or else get a bitbucket app password and modify the URLs to insert that into them, so they look like:

https://bitbucketuserhere:[email protected]/teamorusername/repositoryname.git

Additionally, cloning will not get all versions of git LFS files, so be aware of that. According to bitbucket, use 'git lfs fetch --all' to copy all LFS file versions locally.

To get the list of your personal repositories, use a URL like:

https://api.bitbucket.org/2.0/repositories/BITBUCKETUSERNAME?pagelen=100

To get the list of your team repositories, use a URL like this to get the list of all repositories you are a member of:

https://api.bitbucket.org/2.0/repositories/TEAMNAME?pagelen=100&role=member

The following is an example perl script you could use to clone and then maintain copies of your repositories, using http instead of ssh to fetch. It makes --mirror clones instead of a fully populated working copy (perfect for moving or disaster recovery). It does not back up all LFS files.

#!/usr/bin/env perl

use warnings;
use strict;
use JSON::Parse 'parse_json';

# CONFIGURATION:
# Bitbucket team or user name to get list of repositories from
my $teamORuserName = "myteam";

# Bitbucket app password with access to query the API for the
# list of repositories. Format: "user-name:app-token"
my $appPassword= "frank-james:LAYDxtc8H6FGKUZeHEef";

#------------------------------------------------------------------------------

my $nextPageLink = "https://api.bitbucket.org/2.0/repositories/$teamORuserName?pagelen=100&role=member";
while (defined $nextPageLink)
{
    $nextPageLink =~ m/page=(\d+)/;
    print "Fetching page " . ($1 || 1). "\n";
    my $response = `curl -q --silent --request GET --user '$appPassword' '$nextPageLink'`;
    my $json = parse_json($response);
    my $values = $json->{values};

    foreach my $repo (@$values)
    {
        die "'$repo->{name}' is not a 'git' repo: $repo->{scm}" unless $repo->{scm} eq "git";
        my $links = $repo->{links} || die "no links data for '$repo->{name}'";
        my $clones = $links->{clone} || die "no clone data for '$repo->{name}'";
        my $url = $clones->[0]->{href} || die "no clone url found for $repo->{name}";

        # use uuid as directory name, to survive project name changes
        my $uuid = $repo->{uuid}; $uuid =~ s/[\{\}]//g;
        if (not -e $uuid)
        {
            print "cloning '$repo->{name}' into $uuid\n";
            # replace user name with token to avoid password prompts
            $url =~ s|(https?://).+(\@bitbucket.org)|$1$appPassword$2|;
            system("git clone --progress --mirror '$url' $uuid") == 0 or die "clone failed";
            # make a human friendly link to current repository name
            symlink $uuid, $repo->{slug} or warn "symlink failed: $!";
        }
        else
        {
            print "updating '$repo->{name}' in $uuid\n";
            system("cd $uuid && git fetch --all --tags --prune") == 0 or die "fetch failed";
        }
        print "\n";
    }

    $nextPageLink = $json->{next};
}
exit 0;

Lightfoot

Answer 5

Building upon @eric-nord's answer, I've replaced sed with jq to get to the repo slug (which I found simpler, considering bitbucket outputs json).

So, if you have fewer than 100 repositories and you're looking to clone the repositories for a specific project, the following shell script should do the trick.

#!/usr/bin/env sh

if [ $# -eq 0 ] || [ "$1" == "-?" ] || [ "$1" == "--help" ] || [ "$1" == "-h" ]
  then
    echo "Usage: `basename $0` <URL> <USERNAME> <PROJECTNAME>"
    echo "e.g. `basename $0` bitbucket_url some_user some_project"
    exit 1
fi

curl -u ${2} https://${1}/rest/api/1.0/projects/${3}/repos?limit=100 > repos.json

for repo_name in `cat repos.json | jq -r '.values[] .slug'`
do
  echo "cloning" $repo_name
  git clone https://${2}@${1}/scm/${3}/$repo_name.git ./${3}/$repo_name
done

Swindle

Answer 6

Here is a python script for cloning all the team or user's repositories in bitbucket. As the team repositories are usually private, I took care of that while using the bitbucket API. So just enter your bitbucket username, password, and the team's username and it will take care of cloning all the team repository for you.

import subprocess
import json

cmd = "curl -u <bitbucket_username>:<bitbucket_password>  https://api.bitbucket.org/2.0/repositories/<team_name_or_project_name>"
cmd = cmd.split()

while 1:
    from_api = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    from_api = from_api.communicate()
    json_from_api = json.loads(from_api[0])
    for unit_dict in json_from_api["values"]:
        clone_cmd = "git clone " + unit_dict["links"]["clone"][1]["href"]
        clone_cmd = clone_cmd.split()
        clone_out = subprocess.call(clone_cmd, shell=False)
    if "next" not in json_from_api:
        break
    else:
        cmd[-1] = json_from_api["next"]

Execrable

Answer 7

Here is a node script to download all repositories in a bitbucket account. Please don't forget to add necessary npm packages.

const argv = require('yargs').argv
const request = require('request');
const nodegit = require('nodegit');

let repos = [];
let opts = {
    fetchOpts: {
        callbacks: {
            credentials: function() {
                return nodegit.Cred.userpassPlaintextNew(argv.username, argv.password);
            },
            certificateCheck: function() {
                return 1;
            }
        }
    }
};

function cloneRepository(index) {
    let repo = repos[index];
    console.log('Cloning ' + repo.full_name);
    nodegit.Clone(repo.links.clone[0].href, 'repositories/' + repo.full_name, opts)
    .then(function(repo) {
        if (repos.length - 1 == index) {
            console.log("All repositories cloned");
        } else {
            cloneRepository(index + 1);
        }
    })
    .catch(function(err) {
        if (err) {
            console.log(err);
        }
    });
}

function loadRepositories(url) {
    request.get(url, {
        'auth': {
            'user': argv.username,
            'pass': argv.password
        }
    }, function (err, response, body) {
        if (err) return console.log(err);
        let data = JSON.parse(body);
        for (var i = 0; i < data.values.length; i++) {
            repos.push(data.values[i]);
        }
        if (data.next){
            loadRepositories(data.next);    
        } else if (repos.length > 0) {
            console.log('Started cloning..');
            cloneRepository(0);
        } else {
            console.log("No repositories found");
        }
    });
}

if (argv.username && argv.password) {
    console.log('Loading all repositories..');
    loadRepositories('https://api.bitbucket.org/2.0/repositories/?role=member');
} else {
    console.log('Please specify both the --username and --password options');
}

You can also check out this GitHub repository. Bitbucket Repository Downloader

Extravaganza

The OP asks for bash. This does not look like BASH code. – Centric

@StephenRauch, Eric has already provided the answer for the bash script. My answer might be useful for some other people who are looking for nodejs implementation. – Extravaganza

Answer 8

I built a CLI in rust for concurrent cloning and pulling git repositories (user and project). https://github.com/jensim/bitbucket_server_cli It works with interactive or batch-mode, has shell completion and ability to remember choices in the interactive mode, for a faster user experience.

For a mac user like yourself :

$> brew install jensim/bitbucket_server_cli/bitbucket_server_cli

$> bitbucket_server_cli clone

BitBucket server address: http://localhost
BitBucket username: jensim
✔ BitBucket password · ********
Clone/update all projects yes
Fetching users [00:00:15] [########################################] 2011/2011 (eta:0s)
Fetching projects [00:00:00] [########################################] 35/35 (eta:0s)
Working repos [00:01:07] [########################################] 1337/1337 (eta:0s)

Disclaimer : This is quite experimental, so its mostly used by mac people at the moment, as far as I'm aware.. But there are windows and debian binaries available on Github Releases. Also, this works concurrently, with a configurable number of simultanious http requests/git clone, in order not to kill your CI pipelines

Women

Answer 9

This is what worked for me in Python:

import subprocess
import json
import cookielib
import urllib2
import base64

def _create_opener(proxy=None):
    cj = cookielib.LWPCookieJar()
    cookie_handler = urllib2.HTTPCookieProcessor(cj)
    if proxy:
        proxy_handler = urllib2.ProxyHandler(proxy)
        opener = urllib2.build_opener(cookie_handler, proxy_handler)
    else:
        opener = urllib2.build_opener(cookie_handler)
    return opener

def get_repos(_opener, _auth, user, password, team_username, paging):
    query_url = 'https://bitbucket.org/!api/2.0/repositories/'+team_username+paging
    try:
        req = urllib2.Request(query_url, None, {"Authorization": _auth })
        handler = _opener.open(req)
    except urllib2.HTTPError, e:
        print e.headers
        raise e
    for unit_dict in json.load(handler)["values"]:
        clone_cmd = "git clone " + unit_dict["links"]["clone"][0]["href"].replace('https://'+user,'https://'+password+':'+password)
        clone_cmd = clone_cmd.split()
        clone_out = subprocess.call(clone_cmd, shell=False)
encodedstring = base64.encodestring("%s:%s" % (user, password))[:-1]
_auth = "Basic %s" % encodedstring
_opener = _create_opener()
get_repos(_opener,_auth,'bitbucket-user','bitbucket-password','team-username','pagelen=100&page=1')

code credit to: this and this

Radioscope

Which Python version are you on for this? It gives an error in Python3.7 - a lot of these libraries are in Python2 I see, but at this time Python3 should be the utilized version going forward (just my opinion). – Breunig

Answer 10

Here's a Bash script that downloads via https.

Save to a file download_bitbucket_repos.sh, chmod +x download_bitbucket_repos.sh then ./download_bitbucket_repos.sh.

#!/bin/bash

USER='yourBitBucketUsername' #not email
PASS='yourPassword'
TEAM='teamNameInBitbucket'

curl -u $USER:$PASS https://api.bitbucket.org/1.0/users/$TEAM > repoinfo

for repo_name in `cat repoinfo | sed -r 's/("name": )/\n\1/g' | sed -r 's/"name": "(.*)"/\1/' | sed -e 's/{//' | cut -f1 -d\" | tr '\n' ' '`
do
    echo "Cloning " $repo_name
    git clone https://[email protected]/$TEAM/$repo_name.git
    echo "---"
done

Axenic

Answer 11

I made a simple script available here: https://github.com/mimo84/cloner It works similarly to the answer from Nikolai, however it uses the api on bitbucket to fetch them. It has jq as dependency as well.

Usage: Download it (maybe doing git clone https://github.com/mimo84/cloner.git ) then

chmod +x ./bitbucket.sh

./bitbucket.sh username:password

It is going to create a new folder called "bitbucket_repos" above the current directory and in there clone all of your repos.

Biggers

Answer 12

#!/bin/sh


# this script is inspired by the following url
#https://bytefreaks.net/gnulinux/bash/script-to-clone-all-git-repositories-from-all-projects-of-a-privately-hosted-bitbucket-server

project=myproject
stash_url=https://stash.of.my.company:8081
curl -s --location --request GET '$stash_url/8081/rest/api/1.0/projects/$project/repos' \
--header 'Authorization: Basic XXXXXXXXXXXXx' > repoinfo

for repo_name in `cat repoinfo | grep -o '\"ssh:[^ ,]\+' | xargs -L1`
do
    echo "Cloning " $repo_name
    git clone $repo_name
done

Handed