AttributeError: Can't get attribute '_unpickle_block'

Question

While using:

with open("data_file.pickle", "rb") as pfile:
     raw_data = pickle.load(pfile)  

I get the error:

AttributeError: Can't get attribute '_unpickle_block' on <module 'pandas._libs.internals' from '/opt/conda/lib/python3.8/site-packages/pandas/_libs/internals.cpython-38-x86_64-linux-gnu.so'>

Another answer to a similar question suggests checking the version of pickle I am using. It is the same on my machine, where I developed the code and on server, where I am running the code. I have searched everywhere with no answers. Please help.

Ster

Answer 1

I don't think the problem is pickle module but Pandas version. Your file was probably created with an older version of Pandas. Now you use a newer version, pickle can't "deserialize" the object because the API change.

Try to downgrade your Pandas version and reload file. You can also try to use pd.read_pickle.

Geodetic

I don't know if this is the way to proceed. However, this is what I did: I read data using pandas v1.4.0 and serialized it as hdf5 file. Then downgraded pandas to pandas v1.1.5 serialized again using pickle. @Geodetic was right, the issue was with pandas and not pickle. – Ster

The inverse can also be true. Creating a pickle with a later version, and trying to deserialise with an older version. For example: creating with 1.5.3 and deserialising with 1.2.4. – Pilose

Answer 2

In my case I had to upgrade instead of downgrade the Pandas version. Just make sure they match. Some tips for future readers:

Ask the version with:

import pandas as pd
pd.__version__

And change the version with (replace with your own version)

%pip install pandas==1.4.1

Deliciadelicious

another tip is to iterate minor version, that is something like pip install pandas==1.x.0 nad changes the x one by one. Also, now pandas go pandas 2, so you might be checking pandas==2.x.0 as well. – Dentifrice

Can also check pandas version with pip list | grep pandas. In my case I saved the data using 1.5.2 and could not open with pandas 1.3.0. – Sakovich

Answer 3

I got this error when using vscode and .net interactive notebooks extension. It got resolved when I updated pandas AND restarted the notebook.

Baribaric

Answer 4

Just change the read function from pickle.load(f) to pd.read_pickle(f) without changing anything in the versions.

# from this :
with open('Data.pkl', 'rb') as f:
     t = pickle.load(f)

# to this:
import pandas as pd 
with open('Data.pkl', 'rb') as f:
      t = pd.read_pickle(f)

This solves the problem for me.

Plovdiv

why the downvote? pandas keeps track of internal changes and has an unpickler that accounts for them. – Seamus

Answer 5

Faced same issue. Reading pickle in new version of pandas (the one that it WA originally saved in), saving file as parquet, then reading in older version and saving as pickle helped me.

Magdala

Answer 6

all you have to do is downgrade your sklearn version to 1.0.2

doesn't work? make sure the version you had used in your code and env as same.

that'll work.

happy coding!!

Relation

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review – Karol