As far as I understood, those two packages provide similar but mostly different wrapper functions for Apache Spark. Sparklyr is newer and still needs to grow in the scope of functionality. I therefore think that one currently needs to use both packages to get the full scope of functionality.
As both packages essentially wrap references to Java instances of scala classes, it should be possible to use the packages in parallel, I guess. But is it actually possible? What are your best practices?
These two packages use different mechanisms and are not designed for interoperability. Their internals are designed in different ways, and don't expose JVM backend in the same manner.
While one could think of some solution that would allow for partial data sharing (using global temporary views comes to mind) with persistent metastore, it would have rather limited applications.
If you need both I'd recommend separating your pipeline into multiple steps, and passing data between these, using persistent storage.
© 2022 - 2024 — McMap. All rights reserved.
