I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findsp...

Actually I want to install a library on my Azure databricks cluster but I cannot use the UI method because every time my cluster would change and in transition I cannot add library to it using UI. ...

I am using Azure Databricks to make a delta table in Azure Blob Storage using ADLS Gen2 but i am getting the error "Failure to initialize configurationInvalid configuration value detected for ...

I have modified some code kindly provided by @Bhagyashree in an attempt to mount abfss on ADLS Gen 2 container_name = "root" storage_account = "mystorageaccount" key = "xxx...

I have been following the document: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token to create a service principal and use it to access Databricks....

Can someone let me know how to use the databricks dbutils to delete all files from a folder. I have tried the following but unfortunately, Databricks doesn't support wildcards. dbutils.fs.rm('adl...

I'm trying to use a module in databrick's notebook but I am completely blocked. I'd like to execute the following command or anything similar which allow my to make instances of MyClass from mypac...

I am fairly new to Databricks, so forgive me for the lack of knowledge here. I am using the Databricks resource in Azure. I mainly use the UI right now, but I know some features are only available ...

I am trying to read data on an Azure SQL instance from an Azure Databricks workspace, avoiding using username/password personal credentials for automated, regular data fetch & analysis. I thoug...

I want to delete data from a delta file in databricks. Im using these commands Ex: PR=spark.read.format('delta').options(header=True).load('/mnt/landing/Base_Tables/EventHistory/') PR.write.format(...

I'm getting this error Can't pickle <class 'google.protobuf.pyext._message.CMessage'>: it's not found as google.protobuf.pyext._message.CMessage when I try to create a UDF in PySpark. A...

I've seen a couple of posts on using Selenium in Databricks using %shto install Chrome Drivers and Chrome. This works fine for me, but I had a lot of trouble when I needed to download a file. The f...

I am trying to mount adls gen2 in dattabricks with following configuration configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs...

I am using DataBricks as a service on Azure. This is my cluster info : I ran below command and everythings was o. %sql Select * from db_xxxxx.t_fxxxxxxxxx limit 10 Then I have updated some...

I want to use Dask on Databricks. It should be possible (I cannot see why not). If I import it, one of two things happens, either I get an ImportError but when I install distributed to solve this D...

The following code works as expected for batch, but fails for streaming queries with the infamous AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets f...

I am trying get the workspace name inside a python notebook. Is there any way we can do this? Ex: My workspace name is databricks-test. I want to capture this in variable in python notebook

I'm trying to upload a python file as a workspace file to my Shared Databricks workspace. Using both the CLI and the REST API each Python file I upload turns into a notebook automatically, but I wa...

I would like to create a Databricks job using a Git reference to an Azure Devops repo, the reference would ideally use the git credentials of a service principal. Adding the service principal to th...

When trying to execute local spark code with databricks-connect 13.2.0, it does not work. I have the following issue: Error: details = "INVALID_STATE: cluster xxxxx is not Shared or Single Us...

I'm struggling to write back to an Azure Blob Storage Container. I'm able to read from a container using the following: storage_account_name = "expstorage" storage_account_key = "1VP...

I need to read dataset into a DataFrame, then write the data to Delta Lake. But I have the following exception : AnalysisException: 'Incompatible format detected.\n\nYou are trying to write to `d...

If we have PyPi Packages added as Artifacts to an Azure DevOps Project Feed, how can we use these packages as a source for installing packages in DatabricksStep of Azure Machine Learning Service? W...

We are trying out Unity catalog in Azure Databricks. We connected a pre-existing workspace to the new metastore. I created a new catalog. When I run a notebook and try to write to table "myfir...

i want to list all the tables in every database in Azure Databricks. so i want the output to look somewhat like this: Database | Table_name Database1 | Table_1 Database1 | Table_2 Database1 | Table...