Example of zero-copy share of a Polars dataframe between Python and Rust?

Question

I have a Python function such as

def add_data(input_df): """ some manipulation of input_df (Polars dataframe) such as filling some columns with new values """

I would like to use this function from a Rust function. input_df can be tens of megabytes big, so I'd like to use zero-copy share between Python and Rust. Is there any example code on this available?

I found Is it possible to access underlying data from Polars in cython? but this seems to be Cython. I am looking for a pure Python way.

Lexicology

Did you look at ffi.rs and lib.rs? It uses PyO3 – Eros

Answer 1

I made a crate for this to make this easy: https://github.com/pola-rs/pyo3-polars

See the examples to get started: https://github.com/pola-rs/pyo3-polars/tree/main/example

Gastrolith

Thanks. I see that this example sends dataframe from Python to Rust. My use case is that Rust is the main program and it will use pyo3 to call a method of a Python class. Thus, DataFrame is sent from Rust to Python. Is any such example available as well? – Lexicology

There is no example, but the same should be possible with those extension types. – Gastrolith

@Gastrolith From looking at the crate, I'm not sure how converting from a Rust DataFrame to a Python DataFrame is zero-copy. It looks like into_py calls clone on every Series in the DataFrame. Does that not constitute a complete copy of the DataFrame's data? I have to admit that I might be misunderstanding something here as I have very little experience with Rust. – Mummer

No the data is behind an Arc. So we only increment a reference count. – Gastrolith

hopefully, there is a working example. I searched around with Google and couldn't find one. LOL, I have read all the materials I found and still couldn't fully understand how to convert rust polars dataframe to python pandas/polars dataframe. – Prodigious