How can I download a single raw file from a private github repo using the command line?

Question

On the CI server, I want to fetch a config file that we maintain on Github so it can be shared between many jobs. I'm trying to get this file via curl, but these approaches both fail (I get a 404):

# As advised by the oAuth docs
curl -H 'Authorization: token the_token' -L -o setup.sh https://raw.github.com/org/repo/file

# The url of the raw file after clicking to view it
curl -L https://raw.github.com/org/repo/file?login=username&token=the_token 

Gargoyle

Answer 1

The previous answers don't work (or don't work anymore).

You can use the V3 API to get a raw file like this (you'll need an OAuth token):

curl -H 'Authorization: token INSERTACCESSTOKENHERE' \
  -H 'Accept: application/vnd.github.v3.raw' \
  -O \
  -L https://api.github.com/repos/owner/repo/contents/path

All of this has to go on one line. The -O option saves the file in the current directory. You can use -o filename to specify a different filename.

To get the OAuth token follow the instructions here:

• https://help.github.com/articles/creating-an-access-token-for-command-line-use

I've written this up as a gist as well:

• https://gist.github.com/madrobby/9476733

EDIT: API references for the solution are as follows:

• https://developer.github.com/v3/#authentication
• https://developer.github.com/v3/media/#request-specific-version
• https://developer.github.com/v3/repos/contents/#get-contents

Handley

Note that if the file is public, there's no need for the authorization token: curl -H 'Accept: application/vnd.github.v3.raw' https://api.github.com/repos/owner/repo/contents/path will return the raw file. – Shortcake

Is the -H 'Accept: application/vnd.github.v3.raw' necessary? I was able to access a private file without that part. – Burnisher

@NickChammas: without that header I get a JSON response with metadata and the actual file contents base64 encoded, rather than the file as plain text. – Eventuate

What if the above command gives error as curl: (56) Proxy CONNECT aborted what does that mean. – Brunabrunch

@Handley I've missed something obvious with the file path. Can you explicitly define a harmless README.md file in your command to demonstrate? – Atomize

Note that the URL is different from the URL you'd use in a browser. I've highlighted the difference here: https:// api. github.com/ repos/ <owner>/<repo>/ contents/ <path/to/file> (sorry it's a bit messy) – Septi

@Max: sounds like you need to load your proxy config into your terminal, most likely by setting an http_proxy environment variable or similar. – Septi

For multiple files use -O -L <url> for each link/url. Tested in a docker/linuxkit-4.19.76. – Logogriph

This works also with a personal access token. The minimal set of permission that are required are repo and admin:org/read:org (on a private repository). – Magyar

how can I specify a specific sha/revision of the file? – Lubricator

When writing a script, if you've set up git credential-helper, you can get your access token in there with -H "Authorization: token $(printf 'protocol=https\nhost=github.com\n' | git credential fill | sed -n 's/^password=//p')", which fetches your token from git so you don't need to have it in your environment or hard-coded. – Emmer

Does anybody have documentation to when this behavior was changed? Does Github have some official deprecation documentation or something? – Interested

Answer 2

Alternatively, you can use a github "personal access token" (https://github.com/settings/tokens):

TOKEN=...
curl -s https://[email protected]/<user or organization>/<repo name>/<branch>/<path to file>/<file_name>

Example:

$ curl -s https://[email protected]/concourse/concourse/master/README.md
....

Dercy

@EM0 -- I just tried, it worked. A few things worth double-checking: 1. the host portion is raw.githubusercontent.com, 2. the path is <username>/<repo name>/<branch>/<file name> 3. the token needs to have repo access scope. – Dercy

Yes, that's the path. I took the path from "Download" link for the file, but stripped the "?token=..." from the end and added the token. It does have repo access scope, but this only talks about public repositories. This is an organization private repository. Also, we have 2-factor authentication enabled, but I think if that's the issue it's supposed to give error 401, not 404. – Boating

Yes, this all sounds right. The path sounds good (this is the path I get when I click "Raw", stripped of the ?token=... param, like you said). My user also has 2-factor auth, and I am assuming we are talking about the same token scope (the repo checkbox on github.com/settings/tokens/new). For what it's worth, if the token is invalid, or doesn't have the repo scope, you WILL get 404 (not 401). Not sure why this is not working in your setup... – Dercy

OK, well I guess I'll just use a link so people have to click one extra time - no big deal. Thanks for your help anyway! – Boating

Strange thing: For me using the curl command as above works, but if I open the same link in the browser or try requesting it via java.net.URL.openStream, I get a 404... – Sateen

This was the only way I could get it working for an internal GitHub instance in CMD. Using curl -H 'Authorization: token $TOKEN' $file_url always 404'd on me. I'm not sure why one works and not the other but I've never deep-dived CURL's documentation. – Glisson

This worked for me for pulling a file as a member in an organization, I couldn't get the accepted answer to work for my use case – Putrid

It has stopped working recently. I am still trying to figure out why – Radio

Answer 3

I know this is an old question, but none of the solutions proposed above worked for me. Perhaps the API has changed since then.

This worked:

curl -H 'Authorization: token [insert your token here]' -o output.txt https://raw.githubusercontent.com/[organization]/[repo]/[branch]/[path to file]

Candracandy

this is the only one that worked for me too but there's a small typo in your answer mark. it should be [organization]/[repo]/[branch]... – Atomize

Thanks, only thing that worked for me with Github Enterprise. Note the token required is a Personal Access Token. – Harumscarum

@OliverPearmain did you ever try curl -s https://[email protected]/OrgOrUser/RepoName/BranchOrCommitID/file_name.file_extension ? That's the only thing I could ever get working, but other people in the company used this solution instead, so I'm thinking there might be some kind of config for CURL that makes it less restrictive to use curl -s... instead of this CLI set of parameters. – Glisson

Answer 4

A simpler solution is to use Official GitHub CLI gh.

1. First you must be logged in :

gh auth login

For me, this command is not required since I'm already logged in.

2. Then we need API URL targeting the file to download. And call gh to download it:

API_URL=https://api.github.com/repos/owner/repo/contents/path/file.ext
gh api $API_URL -H "Accept: application/vnd.github.raw" > file.ext

An real example is maybe better. Here it is to download install_linux.md from gh cli:

API_URL=https://api.github.com/repos/cli/cli/contents/docs/install_linux.md
gh api $API_URL -H "Accept: application/vnd.github.raw" > install_linux.md

In API_URL:

• User owner is cli
• Repository name repo is cli too
• Path to file (path/file.ext) is docs/install_linux.md

Sawbuck

To avoid installing jq, you can pass it as option: curl $(gh api $API_URL --jq .download_url) -o file.ext – Dissonant

@BertrandPestre thanks for the trick. It's very new option : 8 days ago ! – Sawbuck

This can be simplified to a single gh call by adding the -H "Accept: application/vnd.github.raw" argument to the gh command. That will tell the API to return the file contents rather than the metadata. – Finn

@Finn Very nice ! I've updated my answer. – Sawbuck

@Finn vnd.github.raw seems undocumented though (only raw+json is, but that returns base64) – Municipal

@Municipal vnd.github.raw is well documented. See here: docs.github.com/en/rest/repos/… – Finn

@Finn that's exactly where I was looking at, and as I said, there's no raw, only raw+... – Municipal

It appears it does the same thing when using curl, but gh tries to interpret it as JSON and complains when the response is binary data. – Finn

Answer 5

I was struggling with this for a few minutes until I realized all that is needed is to wrap the url in quotes to escape the ampersand.

curl "https://raw.github.com/org/repo/file?login=username&token=the_token"

That worked for me in my private repo.

Singletree

Nope, it gets ERROR 404: Not Found. – Bootery

Answer 6

Or, if you don't have a token:

curl --user [your_user] 'https://raw.github.com/path/to/file.config' > file.config

It will ask you for your password.

UPDATE: To those having 404s, you may have to create an SSH key for your machine. Refer to the SSH and GPG Keys in your Settings.

This works if you tap the Raw button on the respective file (It's a button on the top right, available after you tap the file you want to DL). Copy the URL for that raw resource. If you attempt to DL a resource that is not hosted through raw.github.com (it may have changed to raw.githubusercontent.com), it won't work.

I successfully DL'ed a resource from a personal repository, as well as non-personal one, using macOS as of Jul 14 2023 using this method.

Alternative

I get asked for a password, but the response is always a 404. – Getter

Same here: always 404 – Impeccable

It seems like not working anymore – Hoecake

I wonder if associating a SSH / GPG key to your machine would solve the issue. It will ask you for your password. I successfully DL'ed a file from a personal repository as well as a non-personal repository using this method on macOS. – Alternative

Answer 7

I've got a token from an app installation.

Previously, you could use the query ?access_token=MY_TOKEN, but that has been deprecated and eventually removed in September 2021.

In their docs on Authenticating with GitHub Apps they're saying you can clone a repo with the access token and the username x-access-token in the URL.

This also seems to work to download a raw file (the ghs_... is the token):

$> curl "https://x-access-token:[email protected]/Octocat/codertocat/main/README.md"

Manhunt

Tried it -- works. (Thanks for this recent answer!) – Shantay

Thank you for the response. Indeed, it is possible to directly access the file using the method mentioned. – Puma

This no longer works i get a 404, can anyone else confirm? – Splenius

@Kay, it works. Make sure to use curl and the domain to be raw.githubusercontent.com and the path not containing /blob. – Weave

This no longer works. Error in response: Deprecated authentication method. Create a Personal Access Token to access: https://github.com/settings/tokens – Garek

Answer 8

I was able to get it to work for github enterprise, thanks for the suggestions above. Had to take all your suggestions and try and finally i was able to make it work. These are the steps i followed to get it to work.

1. Create personal token, followed these steps:

https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token

2. Make sure you have minimum following permissions for the token:

   • repo (Select all under repo)
   • admin:org -> read:org (select "read:org" under "admin:org")

3. Use the following curl command to get the content:

curl -H "Authorization: token [yourPersonalToken]" -H "Accept: application/vnd.github.v3.raw" -o [filePath]-content.json -L https://github.[company].com/api/v3/repos/[ORG]/[REPO_NAME]/contents/[PATH_TO_FILE]/content.json?ref=[BRANCH_NAME]

Where->

 [yourPersonalToken] is the token you created.
 [filePath] is a path where you want to save the downloaded copy.
 [company] is the name of company which hosted the github enterprise.
 [ORG] is the github organization is which repo is created.
 [REPO_NAME] is the name of the repository.
 [PATH_TO_FILE] is the path where file is located.
 [BRANCH_NAME] is the name of the branch you want to use, e.g. master, develop etc.

Example:

curl -H "Authorization: token 5a86ecda9ff927baaa66fad2af5bee8" -H "Accept: application/vnd.github.v3.raw" -o C:\Downloads\manifest.json -L https://github.example.com/api/v3/repos/cms/cms_one/contents/app/data/manifest.json?ref=master

Homebred

This API can download a file smaller than 1 MB, as of today. If we need download a big file, use this method: caludio.medium.com/… – Rahmann

Answer 9

1. in browser open your github repo: click on file
2. open Developer Tools in browser: select Network tab
3. in browser github: click on Download button
4. close pop-up
5. in browser dev tools: right click on list that has file_name?token=ABAHQCAT6KG...

6. select copy -> copy link address

   url is in format:

   https://raw.githubusercontent.com/<USERNAME>/<PATH>/<FILENAME>?token=ABAHQCAT6KGHYHMG2SLCDT243PH4I

7. in terminal:

   wget -O myFilename https://raw.githubusercontent.com/<USERNAME>/<PATH>/<FILENAME>?token=ABAHQCAT6KGHYHMG2SLCDT243PH4I

Link is valid only for limited time or you can create your token: GitHub article

Flabbergast

Answer 10

Surprisingly none of the answers didn't worked for me until I found a workaround.

You can use personal access token https://github.com/settings/tokens as answered by @thomasfuchs

Note : while creating a token, you must check the admin permissions. See the related issue

https://github.com/octokit/octokit.net/issues/1812

Robertroberta

The admin hint got it working for me. Else i just received a 404. – Vidda

The minimal set of permission that made it work for me on a private repo was repo and admin:org/read:org. – Magyar

Answer 11

I ran into an authentication error when the url was redirected to Amazon S3:

Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter...

Changing from the Authorization: token X header to the ?access_token=<token> query param worked for me.

Puckery

Answer 12

curl -H 'Authorization: token YOUR_TOKEN' \
  -H 'Accept: application/vnd.github.v4.raw' \
  -O \
  -L https://api.github.com/repos/INSERT_OWNER_HERE/INSERT_REPO_HERE/contents/PATH/TO/FILE

So if the url of the raw file (when logged in) is

https://raw.githubusercontent.com/mr_coder/my_repo_name/master/my_script


Then 
  -L https://api.github.com/repos/INSERT_OWNER_HERE/INSERT_REPO_HERE/contents/PATH/TO/FILE
becomes
  -L https://api.github.com/repos/mr_coder/my_repo_name/contents/my_script

Note: We have API v4

Paulina

Answer 13

For GitHub Enterprise and API v3, my bash solution looked like this (includes TOKEN cleanup / privacy):

TOKEN=yourTokenHere; history -d $((HISTCMD-1)) > /dev/null

curl -H "Authorization: token $TOKEN" \
  -H 'Accept: application/vnd.github.v3.raw' \
  -o file.ext \
  -L http://github.company.com/api/v3/repos/[org]/[repo]/contents/path/file.ext?ref=[branch]

unset TOKEN

Wallinga

Answer 14

We had to download files from private GitHub repos fairly often and hacky shell scripts weren't quite cutting it, so we created fetch, which is an open source, cross-platform tool that makes it easy to download source files and release assets from a git tag, commit, or branch of public and private GitHub repos.

For example, to download the file baz from version 0.1.3 of a private GitHub repo to /tmp, you would do the following:

GITHUB_OAUTH_TOKEN="your token"
fetch --repo="https://github.com/foo/bar" --tag="0.1.3" --source-path="/baz" /tmp

Parham

Answer 15

Just an addition to the accepted answer, If you are using Github Enterprise url is slightly different:

curl -H 'Authorization: token [your token]' \
-H 'Accept: application/vnd.github.v3.raw' \
-L https://[your domain]/api/v3/repos/[owner]/[repo-name]/contents/[path of file]

Trireme

Answer 16

A lot of the answers here either suggest to use curl and supply your own Authorization: Bearer **** header, or use the GitHub CLI (gh) to get a pre-authenticated download URL and then using curl with that URL.

However, gh supports custom headers so you can accomplish this using gh with a single request:

gh api "/repos/:owner/:repo/contents/README.md" -H "Accept: application/vnd.github.raw" > README.md

Finn

How to use the GitHub CLI to get a pre-authenticated download URL? – Marquisette

Answer 17

I think it is a little bit dangerous and not good way to issue a personal access token which can access all repositories even just for download a single file from my private repository.

How to -

I would love to recommend using url with token for single file. Don't worry. The token string will generated by github automatically. You can get this url on your source code page.

1. Go to source code page what you want to download via curl or wget, etc
2. Find 'raw' button and click it.
3. New page opened and just copy the url. This url look like below:
   (https://raw.githubusercontent.com/USERNAME/REPONAME/BRANCHNAME/FILENAME?token=TOKENSTRING).
4. You can download a file using this url

Thrive

Be careful, as the token appended to that url will expire after a certain number of days, so you can't really use this solution in production code. – Cockscomb

Furthermore, it seems like the token is different for each file. – Anchorage

It appears to me that this token expires even faster. Ive had it refresh multiple times a day, so this really isnt a viable solution. – Dehydrogenate

"so this really isnt a viable solution" that really depends on your use case. In case the token might be leaked to the public, then "I think it is a little bit dangerous and not good way to issue a personal access token which can access all repositories even just for download a single file from my private repository" makes perfect sense. – Girardo

Answer 18

1. Click your profile photo in the upper-right corner of any GitHub page.
2. Select Settings.
3. In the left sidebar, click Developer settings.
4. Click Personal access tokens.
5. Select Tokens (Classic)
6. From the drop-down menu on the top right corner select Generate new token (classic)
7. Check this scopes:
8. Now use curl and your token with desired "raw" URL, for example:

curl "https://x-access-token:{put_your_token_here}@raw.githubusercontent.com/mvysotsky/docker-miner/main/Dockerfile"

Eclecticism

Answer 19

I tried a simple trick to open a GitHub private .iypnb file in Pycharm as well as Colab and it worked well for me.

1. get raw text for your .ipynb file by pressing Raw button this will open some text like this.

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": []
}]
}

2. open notepad/text editor on os(eg. windows) copy all the text to a new notepad file .

3. save notepad as name.ipynb instead of name.txt and make save as file type All Files(.) instead of Text Documents (*.txt)

4. finally open file in your IDE or colab.

Powel

Answer 20

2024 updated simple method with GitHub CLI gh and curl

login to your account with gh auth login

curl https://$(gh auth token)@raw.githubusercontent.com/ORG/REPOS/BRANCH/PATHTO/FILE

Secretary

Answer 21

Below should work fine. A "raw" before your branch name (master in this case).

curl -L -O https://github.com/your/repo/raw/master/fetch_file.sh

Chinkiang

this question is about private repos – Spindly

Answer 22

You can do this with a raw link.

curl -O https://raw.githubusercontent.com/owner/repo/branchname/path/to/file

Toad

The question was about private repos – Tertias