Apache spark queries through C# [closed]

Question

I was wondering if there is a way I can use C# to write queries to run on Apache spark. I know spark SQL queries can be written in java/scala/python. Is there any interface for c#?

Wiersma

what did searching for it online give ? – Tottering

consider using github.com/spark-jobserver/spark-jobserver - and creating a rest interface in c# serialising json for data structures. I suggest this approach as jobserver provides context's which allow you to go back and re-use previous dataframes as well. – Affiliation

github.com/dotnet/spark will do this. – Transliterate

Answer 1

What exactly you are trying to achieve? If you want to write programs which uses spark apis, probably you are out of luck as only scala,java and python apis are exposed. But if you want to query spark storage data then you can see the thriftserver comes with spark and pass your queries through it. Essentially it should support any jdbc connection (see beeline example in documentation)

Sharonsharona

Would this work with Cassandra? Is there a way to pass an Spark SQL query through thriftserver to get back data without having to use Spark Jobs? – Brandes