Is pickle file of python cross-platform?

Asked 4/12, 2009 at 20:39 Answered 10/7, 2024 at 21:29

I have created a small python script of mine. I saved the pickle file on Linux and then used it on windows and then again used it back on Linux but now that file is not working on Linux but it is working perfectly on windows. Is is so that python is coss-platform but the pickle file is not. Is there any solution to this one???

Tibbetts answered 4/12, 2009 at 20:39 Comment(1)

Python pickle possibly prickly. Pickle platform problems potentially preventable. Present possible problem prevention proposal, please. – Inexcusable 4/12, 2009 at 23:48

Python's pickle is perfectly cross-platform.

This is likely due to EOL (End-Of-Line) differences between Windows and Linux. Make sure to open your pickle files in binary mode both when writing them and when reading them, using open()'s "wb" and "rb" modes respectively.

Note: Passing pickles between different versions of Python can cause trouble, so try to have the same version on both platforms.

Schacker answered 5/12, 2009 at 1:45 Comment(2)

pickle is guaranteed to be "backwards compatible". I assume this means that newer versions will read pickles from older versions, but older versions may not be able to read pickles from newer versions. – Tiernan 28/12, 2012 at 18:12

Note that while the pickle serialization format is guaranteed to be backwards compatible across Python releases, other things in Python are not. For example, functions and classes may be added or removed and modules may be renamed. Pickle stores references to functions and classes using their name and module, and such references may break between different versions of Python. So it isn't generally safe to load pickles saved with a different version of Python. – Schacker 9/7, 2019 at 9:59

The pickle module supports several different data formats. If you are specifying a particular pickle format instead of using the default (0), you may be running into cross-platform binary file problems. You can use plain ASCII pickle files by specifying protocol 0.

Britton answered 4/12, 2009 at 21:2 Comment(0)

Maybe you don't open the file in binary mode? See this stackoverflow question

Palpitant answered 4/12, 2009 at 20:46 Comment(0)

Pickle should be cross-platform, there are versioning/protocol issues, (see http://docs.python.org/library/pickle.html#data-stream-format) but in general if you're using the same release of python on your windows and unix boxes, they should be interoperable.

If you're using pickle as a data transport mechanism, you might want to consider less-implementation specific formats for data storage, such as json, xml, csv, yaml, etc.

Caliber answered 4/12, 2009 at 21:5 Comment(0)

You could use json instead of pickle. If it can save your data, you know it's cross platform.

Textual answered 4/12, 2009 at 20:44 Comment(4)

Not downvoting, but using json for this situation seems kind of awkward. – Perplex 4/12, 2009 at 20:55

It really depends on what he is saving. If it's all lists of numbers and strings, JSON is great. If it's dictionaries and class instances, not so much. – Textual 5/12, 2009 at 0:8

If you are saving class instances, then just use pickle. – Textual 14/12, 2009 at 18:50

The docs say: The pickle module is not secure. Only unpickle data you trust. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.. LANGSEC not included. Do not use except for internal purposes. At least with JSON you can properly parse and (kinda) check what's in there (though much less than with XML, which is the the gold standard for serious applications) – Faythe 24/3, 2024 at 15:38

One interesting idea to try out is PyON (Python Object Notation). The current version seems to work at least for simple cases according to my tests. There seems to have been some disagreement on mailing lists whether the project is a good idea, though.

Daphnedaphnis answered 4/12, 2009 at 22:50 Comment(0)

When Pickle pickles an object, it stores the module and method used to create it and the arguments to feed to that callable.

>>> import pathlib
>>> import pickle
>>> import pickletools
>>> myPath = pickle.dumps(pathlib.Path.home())
>>> pickletools.dis(pickletools.optimize(myPath))

    0: \x80 PROTO      3
    2: c    GLOBAL     'pathlib WindowsPath'
   23: X    BINUNICODE 'C:\\'
   31: X    BINUNICODE 'Users'
   41: X    BINUNICODE '<your_username>'
   59: \x87 TUPLE3
   60: R    REDUCE
   61: .    STOP
highest protocol among opcodes = 2

Conceptually, unpickling this just does:

from pathlib import WindowsPath
return WindowsPath('C:\\', 'Users', '<your_username>')

Several classes and methods are, under the hood, different between Windows and Linux, or have different behaviour between the two systems.

pathlib.Path() will create WindowsPath or PosixPath depending on environment.
os.path will actually be ntpath on Windows and posixpath on Linux.

Likewise, if you have different modules (or different versions of modules) accessible between systems, or the constructor of your object changes, that will make the pickle incompatible.

Likewise, in general you must load and dump pickle files from the same main. If you run two separate scripts

serialize_a.py:

class A:
    def __init__(self, num):
        self.num = num

if __name__ == "__main__":
    import pickle
    with open('pickle.pkl', 'wb') as f:
        pickle.dump(A(42), f)

import pickle
import serialize_a

if __name__ == "__main__":
    with open('pickle.pkl', 'rb') as f:
        pickle.load(f)

You'll get an error because the pickle stores A as being in __main__, but now the code needs to find it as serialize_a.A():

AttributeError: Can't get attribute 'A' on <module '__main__' from ...>

Pelops answered 10/7, 2024 at 21:29 Comment(1)

(Old question for sure, but it comes up when you search 'pickle between computers' etc) – Pelops 10/7, 2024 at 22:14

Recommended topics

Hot tags