serialized machine learning model (pickle) on streamlit web app throws an error

Question

I have built an airbnb machine learning model to predict prices for Singapore. Using pickle, I've serialised the trained model in python 3.9 and pushed it to github.

The streamlit web app located here - https://just4jc-singapore-airbnb-new-streamlit-app-khwftb.streamlit.app/ worked for sometime and recently throws an error.

Traceback (most recent call last):

      File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script

        exec(code, module.__dict__)

      File "/app/singapore-airbnb/new.py", line 119, in <module>

        rf = pickle.load(file)

      File "sklearn/tree/_tree.pyx", line 714, in sklearn.tree._tree.Tree.__setstate__

      File "sklearn/tree/_tree.pyx", line 1418, in sklearn.tree._tree._check_node_ndarray

    ValueError: node array from the pickle has an incompatible dtype:

    - expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}

    - got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]

    /home/appuser/venv/lib/python3.9/site-packages/sklearn/base.py:347: InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeRegressor from version 1.2.0 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:

    https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations

      warnings.warn(

    2023-07-03 06:59:06.382 Uncaught app exception

    Traceback (most recent call last):

      File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script

        exec(code, module.__dict__)

      File "/app/singapore-airbnb/new.py", line 119, in <module>

        rf = pickle.load(file)

      File "sklearn/tree/_tree.pyx", line 714, in sklearn.tree._tree.Tree.__setstate__

      File "sklearn/tree/_tree.pyx", line 1418, in sklearn.tree._tree._check_node_ndarray

    ValueError: node array from the pickle has an incompatible dtype:

    - expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}

    - got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]

    /home/appuser/venv/lib/python3.9/site-packages/sklearn/base.py:347: InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeRegressor from version 1.2.0 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:

    https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations

      warnings.warn(

    2023-07-03 06:59:19.079 Uncaught app exception

    Traceback (most recent call last):

      File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script

        exec(code, module.__dict__)

      File "/app/singapore-airbnb/new.py", line 119, in <module>

        rf = pickle.load(file)

      File "sklearn/tree/_tree.pyx", line 714, in sklearn.tree._tree.Tree.__setstate__

      File "sklearn/tree/_tree.pyx", line 1418, in sklearn.tree._tree._check_node_ndarray

    ValueError: node array from the pickle has an incompatible dtype:

    - expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}

    - got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]

    /home/appuser/venv/lib/python3.9/site-packages/sklearn/base.py:347: InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeRegressor from version 1.2.0 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:

    https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations

      warnings.warn(

    2023-07-03 07:21:24.303 Uncaught app exception

    Traceback (most recent call last):

      File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script

        exec(code, module.__dict__)

      File "/app/singapore-airbnb/new.py", line 119, in <module>

        rf = pickle.load(file)

      File "sklearn/tree/_tree.pyx", line 714, in sklearn.tree._tree.Tree.__setstate__

      File "sklearn/tree/_tree.pyx", line 1418, in sklearn.tree._tree._check_node_ndarray

    ValueError: node array from the pickle has an incompatible dtype:

    - expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}

    - got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]

Appreciate any help on this.

Iodine

Answer 1

This warning line within your error messages indicates this is a versioning conflict:

InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeRegressor from version 1.2.0 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk.

This is a versioning conflict between the version of sklearn that you used to generate/pickle the model and the version the application is using to open it. Version 1.3 was released in June 2023.

To solve your problem, you could:

1. Retrain your model to be compatible with the newer version.
2. Switch to a prior version of sklearn.

Negate

I had same issue and I could successfully run the code after I downgrade scikit-learn version from 1.3.0 to 1.2.0. Thank you. – Carlton

This works, but it isn't very future-proof. What happens when the old scikit version isn't compatible with new Python version anymore? Is it possible to update the pickled model so that it can be read by a newer scikit version? – Blackmarketeer

Answer 2

The Same error was appeared with me and I solved it by edit requirement.txt on github

by type

scikit-learn==1.2.2

then the error has been disappeared Many thanks to anusunalia , I just did this solution from discuss.streamlit

Osteotome

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Suppurate