How to extract the list of all repositories in Stash or Bitbucket?

Question

I need to extract the list of all repos under all projects in Bitbucket. Is there a REST API for the same? I couldn't find one.

I have both on-premise and cloud Bitbucket.

Ep

Please provide whether you're using Bitbucket Cloud or Self Hosted. – Arst

I have two instances one on cloud and the other on-premise – Ep

@Ep Does this answer solve your question? #36090575 – Balf

Answer 1

For Bitbucket Cloud

You can use their REST API to access and perform queries on your server.

Specifically, you can use this documentation page, provided by Atlassian, to learn how to list you're repositories.

---

For Bitbucket Server

Edit: As of receiving this tweet from Dan Bennett, I've learnt there is an API/plugin system for Bitbucket Server that could possibly cater for your needs. For docs: See here.

Edit2: Found this reference to listing personal repositories that may serve as a solution.

AFAIK there isn't a solution for you unless you built a little API for yourself that interacted with your Bitbucket Server instance.

Atlassian Documentation does indicate that to list all currently configured repositories you can do git remote -v. However I'm dubious of this as this isn't normally how git remote -v is used; I think it's more likely that Atlassian's documentation is being unclear rather than Atlassian building in this functionality to Bitbucket Server.

Arst

For BitBucket Server : I get 404 error The requested URL /rest/api/1.0/projects/ was not found on this server. Following is my Request: http://<user>:<password>@BaseURL/rest/api/1.0/projects/ – Ep

I also do not receive the claimed result. When I use this API command, I receive back the text of an HTML document that is essentially a generic boilerplate Bitbucket page containing nothing but links to other Bitbucket pages. No mention of repos or, in fact, anything useful. – Benedikt

Answer 2

Clone ALL Projects & Repositories for a given stash url

    #!/usr/bin/python
    # 
    # @author Jason LeMonier
    #
    # Clone ALL Projects & Repositories for a given stash url
    #
    # Loop through all projects: [P1, P2, ...]
    #    P1 > for each project make a directory with the key "P1"
    #    Then clone every repository inside of directory P1
    #    Backup a directory, create P2, ... 
    # 
    # Added ACTION_FLAG bit so the same logic can run fetch --all on every repository and/or clone.

    import sys
    import os
    import stashy

    ACTION_FLAG = 1     # Bit: +1=Clone, +2=fetch --all 

    url  = os.environ["STASH_URL"]  # "https://mystash.com/stash"
    user = os.environ["STASH_USER"] # joedoe"
    pwd  = os.environ["STASH_PWD"]  # Yay123

    stash = stashy.connect(url, user, pwd)

    def mkdir(xdir):
        if not os.path.exists(xdir):
            os.makedirs(xdir)

    def run_cmd(cmd):
        print ("Directory cwd: %s "%(os.getcwd() ))
        print ("Running Command: \n    %s " %(cmd))
        os.system(cmd)

    start_dir = os.getcwd()

    for project in stash.projects:
        pk = project_key = project["key"]
        mkdir(pk) 
        os.chdir(pk)

        for repo in stash.projects[project_key].repos.list():
            for url in repo["links"]["clone"]:
                href = url["href"]
                repo_dir = href.split("/")[-1].split(".")[0]

                if (url["name"] == "http"):
                    print ("        url.href: %s"% href)  # https://[email protected]/stash/scm/app/ae.git
                    print ("Directory cwd: %s Project: %s"%(os.getcwd(), pk))

                    if ACTION_FLAG & 1 > 0:
                        if not os.path.exists(repo_dir):
                            run_cmd("git clone %s" % url["href"])
                        else:
                            print ("Directory: %s/%s exists already.  Skipping clone. "%(os.getcwd(), repo_dir))

                    if ACTION_FLAG & 2 > 0:
                        # chdir into directory "ae" based on url of this repo, fetch, chdir back
                        cur_dir = os.getcwd()
                        os.chdir(repo_dir)
                        run_cmd("git fetch --all ")
                        os.chdir(cur_dir)

                    break

        os.chdir(start_dir) # avoiding ".." in case of incorrect git directories

Wreathe

This is super cool, and I will probably use it someday. But it doesn't answer the question, "how do I list all the repos in Bitbucket?" Cloning all repos to get a list of them would be massive time and space consuming overkill. – Benedikt

Answer 3

1. Once logged in: on the top right, click on your profile pic and then 'View profile'

2. Take note of your user (in the example below '[email protected]', but keep in mind it's case sensitive)

3. Click on profile pic > Manage account > Personal access token > Create a token (choosing 'Read' access type is enough for this functionality)

For all projects:

4. Open a CLI and use the command below (remember to fill in your server domain!):

curl -u "[email protected]" -X GET https://<my_server_domain>/rest/api/1.0/projects/?limit=1000

5. It will ask you for your personal access token, you comply and you get a JSON file with all repos requested

For all repos in a given project:

4. Pick the project you want to get repos from. In my case, the project URL is: <your_server_domain>/projects/TECH/ and therefore my {projectKey} is 'TECH', which you'll need for the command below.

5. Open a CLI and use this command (remember to fill in your server domain and projectKey!):

curl -u "[email protected]" -X GET https://<my_server_domain>/rest/api/1.0/projects/{projectKey}/repos?limit=50

Final touches

6. (optional) If you want just the titles of the repos requested and you have jq installed (for Windows, downloading the exe and adding it to PATH should be enough, but you need to restart your CLI for that new addition to be detected), you can use the command below:

curl -u $BBUSER -X GET <my_server_domain>/rest/api/1.0/projects/TECH/repos?limit=50 | jq '.values|.[]|.name'

(tested with Data Center/Atlassian Bitbucket v7.9.0 and powershell CLI)

Balf

I am getting this error: Bitbucket Cloud recently stopped supporting account passwords for API authentication. See our community post for more details: atlassian.community/t5/x/x/ba-p/1948231 App passwords are recommended for most use cases and can be created in your Personal settings: bitbucket.org/account/settings/app-passwords For more details on API authentication methods see our documentation: developer.atlassian.com/cloud/bitbucket/rest/intro/… – Aurlie

Answer 4

I ended up having to do this myself with an on-prem install of Bitbucket which didn't seem to have the REST APIs discussed above accessible, so I came up with a short script to scrape it out of the web page. This workaround has the advantage that there's nothing you need to install, and you don't need to worry about dependencies, certs or logins other than just logging into your Bitbucket server. You can also set this up as a bookmark if you urlencode the script and prefix it with javascript:.

To use this:

1. Open your bitbucket server project page, where you should see a list of repos.
2. Open your browser's devtools console. This is usually F12 or ctrl-shift-i.
3. Paste the following into the command prompt there.

JSON.stringify(Array.from(document.querySelectorAll('[data-repository-id]')).map(aTag => {
  const href = aTag.getAttribute('href');
  let projName = href.match(/\/projects\/(.+)\/repos/)[1].toLowerCase();
  let repoName = href.match(/\/repos\/(.+)\/browse/)[1];
  repoName = repoName.replace(' ', '-');
  const templ = `https://${location.host}/scm/${projName}/${repoName}.git`;
  return {
    href,
    name: aTag.innerText,
    clone: templ
  }
}));

The result is a JSON string containing an array with the repo's URL, name, and clone URL.

[{
  "href": "/projects/FOO/repos/some-repo-here/browse",
  "name": "some-repo-here",
  "clone": "https://mybitbucket.company.com/scm/foo/some-repo-here.git"
}]

Tartrazine

neat trick but it doesn't seem to grab ALL my repos.. probably the HTML has changed slightly – Vig

I am getting an empty array. Your code needs to be changed. – Aurlie

Answer 5

This ruby script isn't the greatest code, which makes sense, because I'm not the greatest coder. But it is clear, tested, and it works.

The script filters the output of a Bitbucket API call to create a complete report of all repos on a Bitbucket server. Report is arranged by project, and includes totals and subtotals, a link to each repo, and whether the repos are public or personal. I could have simplified it for general use, but it's pretty useful as it is.

There are no command line arguments. Just run it.

#!/usr/bin/ruby
#
# @author Bill Cernansky
#
# List and count all repos on a Bitbucket server, arranged by project, to STDOUT.
#
require 'json'

bbserver   = 'http(s)://server.domain.com'
bbuser     = 'username'
bbpassword = 'password'
bbmaxrepos = 2000       # Increase if you have more than 2000 repos

reposRaw = JSON.parse(`curl -s -u '#{bbuser}':'#{bbpassword}' -X GET #{bbserver}/rest/api/1.0/repos?limit=#{bbmaxrepos}`)

projects = {}

repoCount = reposRaw['values'].count

reposRaw['values'].each do |r|
    projID = r['project']['key']
    if projects[projID].nil?
        projects[projID] = {}
        projects[projID]['name'] = r['project']['name']
        projects[projID]['repos'] = {}
    end
    repoName = r['name']
    projects[projID]['repos'][repoName] = r['links']['clone'][0]['href']
end

privateProjCount = projects.keys.grep(/^\~/).count
publicProjCount = projects.keys.count - privateProjCount

reportText = ''
privateRepoCount = 0
projects.keys.sort.each do |p|
    # Personal project slugs always start with tilde
    isPrivate = p[0] == '~'
    projRepoCount = projects[p]['repos'].keys.count
    privateRepoCount += projRepoCount if isPrivate
    reportText += "\nProject: #{p} : #{projects[p]['name']}\n  #{projRepoCount} #{isPrivate ? 'PERSONAL' : 'Public'} repositories\n"

    projects[p]['repos'].keys.each do |r|
        reportText += sprintf("    %-30s : %s\n", r, projects[p]['repos'][r])
    end
end

puts "BITBUCKET REPO REPORT\n\n"
puts sprintf("  Total Projects: %5d     Public: %5d    Personal: %5d", projects.keys.count, publicProjCount, privateProjCount)
puts sprintf("  Total Repos:    %5d     Public: %5d    Personal: %5d", repoCount, repoCount - privateRepoCount, privateRepoCount)
puts reportText

Benedikt

Answer 6

The way I solved this issue, was get the html page and give it a ridiculous limit like this. thats in python :

 cmd = "curl -s  -k --user " +  username +  " https://URL/projects/<KEY_PROJECT_NAME>/?limit\=10000"

then I parsed it with BeautifulSoup

make_list = str((subprocess.check_output(cmd, shell=True)).rstrip().decode("utf-8"))
  html = make_list 
  parsed_html = BeautifulSoup(html,'html.parser')
list1 = [] 
  for a in parsed_html.find_all("a", href=re.compile("/<projects>/<KEY_PROJECT_NAME>/repos/")):
    list1.append(a.string)
print(list1)

to use this make sure you change and , this should be the bitbucket project you are targeting. All , I am doing is parsing an html file.

Alpenstock

Answer 7

Here's how I pulled the list of repos from Bitbucket Cloud.

Setup OAauth Consumer

Go to your workspace settings and setup an OAuth consumer, you should be able to go here directly using this link: https://bitbucket.org/{your_workspace}/workspace/settings/api

The only setting that matters is the callback URL which can be anything but I chose http://localhost

Once setup, this will display a key and secret pair for your OAuth consumer, I will refer to these as {oauth_key} and {oauth_secret} below

Authenticate with the API

Go to https://bitbucket.org/site/oauth2/authorize?client_id={oauth_key}&response_type=code ensuring you replace {oauth_key}

This will redirect you to something like http://localhost/?code=xxxxxxxxxxxxxxxxxx, make a note of that code, I'll refer to that as {oauth_code} below

In your terminal go to curl -X POST -u "{oauth_key}:{oauth_secret}" https://bitbucket.org/site/oauth2/access_token -d grant_type=authorization_code -d code={oauth_code} replacing the placeholders.

This should return json including the access_token, I’ll refer to that access token as {oauth_token}

Get the list of repos

You can now run the following to get the list of repos. Bear in mind that your {oauth_token} lasts 2hrs by default.

curl --request GET \
  --url 'https://api.bitbucket.org/2.0/repositories/pageant?page=1' \
  --header 'Authorization: Bearer {oauth_token}' \
  --header 'Accept: application/json'

This response is paginated so you'll need to page through the responses, 10 repositories at a time.

Valentino

Welcome to Stack Overflow. I see you've posted this exact same answer in two different locations (reference). Please be careful about that. If the exact same answer can solve two questions, you should consider whether those questions are duplicates of one another; if so, the less useful version of the question should be flagged as a duplicate of the more useful version. Alternatively, if they aren't duplicates, you should customize the answer to addre – Ballot