I can't load my model because I can't put a PosixPath

Question

I'm setting up a script and I need to use some functions from fast-ai package. The fact is that I'm on Windows and when I define my paths, the function from fast-ai named load_learner can't load the model.

I've tried to change the function into the package as:

state = pickle.load(open(str(path) + '/' + str(fname), 'rb'))

instead of:

state = pickle.load(open(path/fname, 'rb'))

but I obtain this error:

 File "lib\site-packages\fastai\basic_train.py", line 462, in load_learner
    state = pickle.load(open(path/fname, 'rb'))
  File "\lib\pathlib.py", line 1006, in __new__
    % (cls.__name__,))
NotImplementedError: cannot instantiate 'PosixPath' on your system

My paths are defined as:

folder_path = './models/model1'
fname = 'model.pkl'

and I call the function as: model = load_learner(folder_path, fname)

How can I use Windows paths in this function?

---

UPDATE 1

The answer posted was correct only on Linux. I still have the issue on Windows. I didn't find a way to pass through the PosixPath on Windows. The only solution that I found is to change the internal packages from my modules but it is not a secure way to solve this kind of issue.

---

Thanks in advance.

Stickweed

This can be a problem with the windows separator which is double backslash \\ for paths, you can easily obtain the right separator for all systems by using os.sep see. Try to replace all / with the windows format separator – Bessiebessy

Please post the entire stack trace of the exception. – Megaphone

First try to update your package to the latest version. The offending piece of code is not there, so it may fix the problem you're having. – Predicative

Answer 1

Just redirect PosixPath to WindowsPath.

import pathlib
temp = pathlib.PosixPath
pathlib.PosixPath = pathlib.WindowsPath

I am also loading fastai models and this trick works.

IMPORTANT: Since this might cause issues later, remember to set pathlib.PosixPath = temp when done.

Marie

Since this might cause issues later, remember to set pathlib.PosixPath = temp when done. – Legofmutton

You can also see my suggestion to automatically revert PosixPath (especially if you have an error while doing the import): https://mcmap.net/q/689041/-i-can-39-t-load-my-model-because-i-can-39-t-put-a-posixpath – Portecochere

Sometimes on linux systems i get this error, for that i had to use import platform import pathlib plt = platform.system() if plt == 'Linux': pathlib.WindowsPath = pathlib.PosixPath – Sinuous

Answer 2

When working on Windows, you could temporarily set pathlib.PosixPath to WindowsPath. It is important to revert to original value, especially when there is an exception during the loading of pickle.

A simple way is to do a try / finally:

posix_backup = pathlib.PosixPath
try:
    pathlib.PosixPath = pathlib.WindowsPath
    learn_inf = load_learner(EXPORT_PATH)
finally:
    pathlib.PosixPath = posix_backup

If you do it a lot, you could make the flow smoother as follows:

1. Define a function that could temporarily do the change
2. Use it in a with block

You could add this somewhere (top of a script or dedicated cell if you use Jupyter).

from contextlib import contextmanager
import pathlib

@contextmanager
def set_posix_windows():
    posix_backup = pathlib.PosixPath
    try:
        pathlib.PosixPath = pathlib.WindowsPath
        yield
    finally:
        pathlib.PosixPath = posix_backup

And then, just use it like this:

EXPORT_PATH = pathlib.Path("model.pkl")

with set_posix_windows():
    learn_inf = load_learner(EXPORT_PATH)

... also, check the answer from sophros: https://mcmap.net/q/689041/-i-can-39-t-load-my-model-because-i-can-39-t-put-a-posixpath

Portecochere

using contextmanager is just brilliant! – Laughlin

This solution also works when running the load_learner() method in an Azure Function (running locally on Windows). – Fissionable

Answer 3

According my own question, I find a way using:

from pathlib import Path

folder_path = Path('./models/model1')

---

UPDATE 1

This solution only works on Linux, on Windows I still get an error.

---

Stickweed

Hi, some person have suggested solutions that also work on Windows. Please change the accepted answer. Thanks – Portecochere

Answer 4

The issue here is related to the differences in the way Python handles paths depending on the OS:

• PosixPath - on Linux / Unix

• WindowsPath - on Windows

When persisting objects using pickle on one OS (say Linux - as in this case) information about the type / class is persisted too (here: PosixPath).

Now, when the pickle file is being loaded Python assumes it is going to be able to recreate objects based on the type information it previously persisted. In this case it tries to recreate an object of PosixPath type which is prevented by pathlib library and cannot be instantiated on Windows. On Windows there should be WindowsPath be used instead but pickle module does not handle such OS-dependent logic very well hence it helplessly throws the error.

You could theoretically interfere in the code of pathlib to remove the OS check but there is no easy workaround but avoiding pickling of OS-dependent objects (e.g. storing paths as strings - as os.path does - would certainly workaround this issue).

There is also another possibility - to use a platform-independent PurePosixPath class for path objects.

Dvinsk

Great answer and thanks for finding the "real" problem and how to fix it! – Claus

Answer 5

This usually happens when you might have trained your model in Google Colab, and try accessing on Windows. To overcome this, you just have to paste the below code in your yolov5's detect.py

import pathlib   
temp = pathlib.PosixPath   
pathlib.PosixPath = pathlib.WindowsPath

Have a Great Day!

Netsuke

This is identical to the accepted answer to this question, which has been community-validated. Please check all other answers before positing. – Soluk

Answer 6

According to provided error message you are using pathlib. So you don't need to use + '/' + here: str(path) + '/' + str(fname)

/ as path separator works on Linux/Unix:

state = pickle.load(open(path / fname, 'rb'))

On Windows use .joinpath() instead:

state = pickle.load(open(path.joinpath(fname), 'rb'))

If you are not going to use the pathlib, use os.path.join(). It will automatically select right format for your OS.

Soliz

Even if I use os.path.join() on Windows, I still get this following error: state = pickle.load(open(path/fname, 'rb')) !! TypeError: unsupported operand type(s) for /: 'str' and 'str'. I'm trying to search a method to not change my package script. – Stickweed

Answer 7

For the posix path error: when you train your model on colab/gradient and download it, then do inference on Windows.

Just redirect PosixPath to WindowsPath:

import pathlib

temp = pathlib.PosixPath
pathlib.PosixPath = pathlib.WindowsPath

Wreck

Welcome to stackoverflow. This should be posted as a comment to KumaTea's answer. – Peregrinate

Answer 8

I am working on the same- deploying fastai model as a web server and got the same issue and this is what I did... While exporting the model use joblib or pickle to pickle the model instead of using learn.export() and in server use the below code.

 __model = pickle.load(open(os.path.join('./artifacts/saved_model.pkl'), 'rb'))

By doing this it is able to solve the path issue but since the model is trained using GPU it gives the error which asks to map the storage to CPU

Savino