Setting environment variables in Google Colab

Question

I'm trying to use the Kaggle CLI API, and in order to do that, instead of using kaggle.json for authentication, I'm using environment variables to set the credentials.

!pip install --upgrade kaggle

!export KAGGLE_USERNAME=abcdefgh
!export KAGGLE_KEY=abcdefgh

!export -p

However, the printed list of env. variables doesn't contain the ones I set above.

declare -x CLICOLOR="1"
declare -x CLOUDSDK_CONFIG="/content/.config"
declare -x COLAB_GPU="1"
declare -x CUDA_PKG_VERSION="9-2=9.2.148-1"
declare -x CUDA_VERSION="9.2.148"
declare -x CUDNN_VERSION="7.4.1.5"
declare -x DATALAB_SETTINGS_OVERRIDES="{\"kernelManagerProxyPort\":6000,\"kernelManagerProxyHost\":\"172.28.0.3\",\"jupyterArgs\":[\"notebook\",\"-y\",\"--no-browser\",\"--log-level=DEBUG\",\"--debug\",\"--NotebookApp.allow_origin=\\\"*\\\"\",\"--NotebookApp.log_format=\\\"%(message)s\\\"\",\"--NotebookApp.disable_check_xsrf=True\",\"--NotebookApp.token=\",\"--Session.key=\\\"\\\"\",\"--Session.keyfile=\\\"\\\"\",\"--ContentsManager.untitled_directory=\\\"Untitled Folder\\\"\",\"--ContentsManager.untitled_file=\\\"Untitled File\\\"\",\"--ContentsManager.untitled_notebook=\\\"Untitled Notebook\\\"\",\"--KernelManager.autorestart=True\",\"--ip=\\\"172.28.0.2\\\"\"]}"
declare -x DEBIAN_FRONTEND="noninteractive"
declare -x ENV="/root/.bashrc"
declare -x GIT_PAGER="cat"
declare -x GLIBCPP_FORCE_NEW="1"
declare -x GLIBCXX_FORCE_NEW="1"
declare -x HOME="/root"
declare -x HOSTNAME="2ced809e9844"
declare -x JPY_PARENT_PID="57"
declare -x LANG="en_US.UTF-8"
declare -x LD_LIBRARY_PATH="/usr/lib64-nvidia"
declare -x LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4"
declare -x MPLBACKEND="module://ipykernel.pylab.backend_inline"
declare -x NCCL_VERSION="2.3.7"
declare -x NVIDIA_DRIVER_CAPABILITIES="compute,utility"
declare -x NVIDIA_REQUIRE_CUDA="cuda>=9.2"
declare -x NVIDIA_VISIBLE_DEVICES="all"
declare -x OLDPWD="/"
declare -x PAGER="cat"
declare -x PATH="/usr/local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin"
declare -x PWD="/content"
declare -x PYTHONPATH="/env/python"
declare -x SHELL="/bin/bash"
declare -x SHLVL="2"
declare -x TERM="xterm-color"
declare -x TF_FORCE_GPU_ALLOW_GROWTH="true"
declare -x _="/tools/node/bin/forever"
declare -x __EGL_VENDOR_LIBRARY_DIRS="/usr/lib64-nvidia:/usr/share/glvnd/egl_vendor.d/"

Printer

Possible duplicate of Is it possible to set environment variables in Google Colaboratory? – Chelicera

Answer 1

I think you want to do something like:

import os
os.environ['KAGGLE_USERNAME'] = ...
os.environ['KAGGLE_KEY'] = ...

The reason is that !export will assign the environment variable in an ephemeral sub-shell. But, you want to update the environment for the Python subprocess that spawns those sub-shells.

Modestomodesty

Answer 2

If you like %magic, you can also use %env to make it a bit shorter.

%env KAGGLE_USERNAME=abcdefgh

If the value is in a variable you can also use

%env KAGGLE_USERNAME=$username

Uncinus

To set multiple env variables in one line: #26190162 – Ladyship

This is just much simpler and should be the best answer – Decosta

This should be THE answer! – Seeseebeck

This answer does not explain why the most obvious approach (!export name=value) fails. – Klemm

Answer 3

You could either use intput or simply getpass to set env variables.(Assuming yr using JP noteb)

!pip install getpass

import getpass
import os


def SetEnv(name):
    secret = getpass.getpass(f"Enter value of {name}: ")
    os.environ[name] = secret

# to get var
mys = os.environ['secret']
print(mys)

# Usage to set var
SetEnv("api_id")
SetEnv("api_hash")
SetEnv("bot_token")

Awesome