Can't find kaggle.json file in google colab

Question

I'm trying to download the kaggle imagenet object localization challenge data into google colab so that I can use it to train my model. Kaggle uses an API for easy and fast access to their datasets. (https://github.com/Kaggle/kaggle-api) However, when calling the command "kaggle competitions download -c imagenet-object-localization-challenge" in google colab, it can't find the kaggle.json file which contains my username and api-key.

I haven't had this problem on my mac when running a jupyter notebook, but since I want to use google's gpu for my model, I started using google colab. Because the kaggle API expects the username and api-key to be in a kaggle.json file located in a .kaggle directory, I first created the directory .kaggle and then the file kaggle.json, into which I wrote my username and api-key (The example below doesn't display my username and api-key). I then tried to configure the path to my json file for kaggle to use when calling the kaggle download command.

!pip install kaggle

!mkdir .kaggle
!touch .kaggle/kaggle.json

api_token = {"username":"username","key":"api-key"}

import json
import zipfile
import os
with open('/content/.kaggle/kaggle.json', 'w') as file:
    json.dump(api_token, file)

!chmod 600 /content/.kaggle/kaggle.json
!kaggle config path -p /content

However, when running the last command, I got the error:

IOError: Could not find kaggle.json. Make sure it's located in /root/.kaggle. Or use the environment method.

My goal was to use the following commands to get the dataset from kaggle:

!kaggle competitions download -c imagenet-object-localization-challenge
os.chdir('/content/competitions/imagenet-object-localization-challenge')
for file in os.listdir():
    zip_ref = zipfile.ZipFile(file, 'r')
    zip_ref.extractall()
    zip_ref.close()

I don't understand why the kaggle API can't find my json file. How can I use the API in google colab?

Matriarchy

Answer 1

As the error said, you need to put kaggle.json in the right place.

Try:

!mv .kaggle /root/

Then run your code again.

Antioch

Answer 2

According to kaggle api documentation the location where credentials json is looking for is ~/.kaggle/kaggle.json as google colab environment is Linux based. In your snippet you try to config path parameter, but it is not used to looking for credential json:

- path: Folder where file(s) will be downloaded, defaults to current working directory

So the full working snippet for google colab environment would be:

!mkdir ~/.kaggle
!touch ~/.kaggle/kaggle.json

api_token = {"username":"username","key":"api-key"}

import json

with open('/root/.kaggle/kaggle.json', 'w') as file:
    json.dump(api_token, file)

!chmod 600 ~/.kaggle/kaggle.json

And then some api call like

!kaggle datasets download -d datamunge/sign-language-mnist

Plainsman

This is by far the best answer and should be the accepted answer! – Wehner

Good answer. Please don't write code as root though. – Wellgrounded

The last step chmod is recommended by Kaggle and a warning message would pop up on Kaggle if the file permission isn't set to 600 to prevent other users on the system to see the API token. – Pazia

Answer 3

You can find the JSON file that you need to place form Kaggle's portal itself. Just log into kaggle, go to your account page and then click 'Create New API Token' and it will give you a JSON file that needs to be placed.

Intermediary

Amazing answer, thank you! After I clicked on 'Create New API Token', a save to location popup window appeared and I saved it under ~/.kaggle/kaggle.json and now the kaggle download command works. – Alloy

Answer 4

My own method, and if you like to minimise '!' addition to Egor B Eremeev answer above.

This may be beneficial depending on your use case and in the long run lead to cleaner code if you are scripting.

import json 
import os
from pathlib import Path

# your api key
api_key = {
'username':"username" ,
'key':"some_api_key"}

# uses pathlib Path
kaggle_path = Path('/root/.kaggle')
os.makedirs(kaggle_path, exist_ok=True)

# opens file and dumps python dict to json object 
with open (kaggle_path/'kaggle.json', 'w') as handl:
    json.dump(api_key,handl)

os.chmod(kaggle_path/'kaggle.json', 600)  

Culver

Nice. This is the only answer that worked for me. Good job. – Paleoclimatology

Answer 5

Run this code in cell to config the environment

import os
os.environ["KAGGLE_CONFIG_DIR"] = "/path_to_your_kaggle.json_file"

They've written their GitHub repo that you can config the environment of kaggle.json error appears in colab or windows too:

You can define a shell environment variable KAGGLE_CONFIG_DIR to change this location to $KAGGLE_CONFIG_DIR/kaggle.json (on Windows it will be %KAGGLE_CONFIG_DIR%\kaggle.json).

-- Refer this

Escudo

Answer 6

In case you are a windows user, move your kaggle.json into .kaggle folder, that you are able to find it in your system root (which is in your username folder).

Foramen

Answer 7

This link is helpful for uploading kaggle dataset into collab: https://www.kaggle.com/discussions/general/74235

You can download kaggle.json file from https://www.kaggle.com/settings, look into API

& for dataset download command- go on respective dataset which you want to download click on 3 dots(look into attached image) enter image description here

Fixity