How do I use databricks-cli without manual configuration

Question

I want to use databricks cli:

databricks clusters list

but this requires a manual step that requires interactive work with the user:

databricks configure --token

Is there a way to use databricks cli without manual intervention so that you can run it as part of a ci/cd pipeline?

Benis

Answer 1

The following bash script, configured the databricks cli automatically:

echo "configuring databrick-cli authentication"

declare DATABRICKS_URL="https://westeurope.azuredatabricks.net"
declare DATABRICKS_ACCESS_TOKEN="authentication_token_generated_from_databricks_ux"

declare dbconfig=$(<~/.databrickscfg)
if [[ $dbconfig = *"host = "* && $dbconfig = *"token = "* ]]; then
  echo "file [~/.databrickscfg] is already configured"
else
  if [[ -z "$DATABRICKS_URL" || -z "$DATABRICKS_ACCESS_TOKEN" ]]; then
    echo "file [~/.databrickscfg] is not configured, but [DATABRICKS_URL],[DATABRICKS_ACCESS_TOKEN] env vars are not set"
  else
    echo "populating [~/.databrickscfg]"
    > ~/.databrickscfg
    echo "[DEFAULT]" >> ~/.databrickscfg
    echo "host = $DATABRICKS_URL" >> ~/.databrickscfg
    echo "token = $DATABRICKS_ACCESS_TOKEN" >> ~/.databrickscfg
    echo "" >> ~/.databrickscfg
  fi
fi

Benis

apparently you can now log on with a username, and password... but on Azure i can never get it to work #voodoo 😕 – Verner

I am trying to run this in R. Do you know how should I go about it using this bash script to run in R? – Polychromy

Answer 2

As mentioned above by @usingnamespace, and from the official docs:

CLI 0.8.0 and above supports environment variables, an environment variable setting takes precedence over the setting in the configuration file.

DATABRICKS_HOST
DATABRICKS_USERNAME
DATABRICKS_PASSWORD
DATABRICKS_TOKEN

With that, not only you will not be exposing sensitive data in clear text files (~/.databrickscfg), you won't need to add any more code to your script.

Trossachs

This should be the accepted answer now. It's much better than populating a config file. – Randellrandene

Even with the DATABRICKS_TOKEN set in the env variables, the token still gets written in plain text into the cfg file if I run databricks configure --host ..... Is that expected behavior? – Pontormo

Answer 3

You can just export the variables DATABRICKS_HOST and DATABRICKS_TOKEN. With those variables, you don't need a config file.

Doubleacting

Answer 4

This is a PowerShell version of the script.

write-host Configure databricks access
$Env:DATABRICKS_CONFIG_FILE = "$(System.DefaultWorkingDirectory)/.databrickscfg"
Set-Location $(System.DefaultWorkingDirectory)
Set-Content .databrickscfg "[DEFAULT]"
Add-Content .databrickscfg "host = https://westeurope.azuredatabricks.net/"
Add-Content .databrickscfg "token = $(db-token)"

Betancourt

To make the configure effective, the databricks-cli commands must be called in the same PS session. – Betancourt

Answer 5

As of Databricks CLI v0.224.0 (2024-08), I think these two ways would work best:

• Environment variables [link]

  • DATABRICKS_HOST, set to the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
  • DATABRICKS_TOKEN

• Configuration profile [link]

  Write the following lines to .databrickscfg file

  [<some-unique-configuration-profile-name>]
  host  = <per-workspace-url>
  token = <token>
  

  Then you can use databricks --profile <profile-name> <command> to manage that workspace

Reverberate