Create a new cluster in Databricks using databricks-cli

Question

I'm trying to create a new cluster in Databricks on Azure using databricks-cli.

I'm using the following command:

databricks clusters create --json '{ "cluster_name": "template2", "spark_version": "4.1.x-scala2.11" }'

And getting back this error:

Error: {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required field: size"}

I can't find documentation on this issue, would be happy to receive some help.

Scrotum

Answer 1

I found the right answer here.

The correct format to run this command on azure is:

databricks clusters create --json '{ "cluster_name": "my-cluster", "spark_version": "4.1.x-scala2.11", "node_type_id": "Standard_DS3_v2", "autoscale" : { "min_workers": 2, "max_workers": 50 } }'

Scrotum

Yes however this creates Multi mode cluster. If you need to create single node cluster different set of parameters you need to pass – Halfsole

I used the API and it worked like a charm, I used it to clone an existing cluster and just had to rename and remove the cluster_id. Btw, your link is not working anymore. docs.databricks.com/api/workspace/clusters/create – Gustaf

Answer 2

Just to add to the answer that @MorShemesh gave, you can also use a path to a JSON file instead of specifying the JSON at the command line.

databricks clusters create --json-file /path/to/my/cluster_config.json 

If you are managing lots of clusters this might be an easier approach.

Loram

Answer 3

databricks clusters create --json "{ "cluster_name": "custpm-cluster", "spark_version": "4.1.x-scala2.09", "node_type_id": "Standard_DS3_v2", "autoscale" : { "min_workers": 2, "max_workers": 50 }}"

Escobedo

this is incorrect answer... – Breakage

Answer 4

databricks --json-file is deprecated.

Now it is as below

databricks clusters create --json @cluster_config.json

The actual json file path has to be prefixed with @ (or you can supply the full json string within double quotes). If not prefixed with @, you will receive an error message which is similar to below

Error: invalid character 'c' looking for beginning of value

The 'c' will be replaced with the first character of your json file as it does not recognise it is a file and not a json literal. Might be useful for someone I hope.

Neves

Is there any doco on this format? I can't find it. – Kinghorn