Embedded Python does not work pointing to Python35.zip with NumPy - how to fix?
Asked Answered
R

1

1

Okay here's the basic example from the Python website for a simple runpy.exe to run Python scripts below. It works fine using Visual Studio 2015 on x64 Windows after referencing the Python includes and linking to python35.lib for basic functions (the docs don't mention pyvenv.cfg must be in the EXE directory). However, calling a script that imports NumPy leads to this error ImportError: No module named 'numpy' Failed to load "eig" only when using embedded python35.zip, so how does one include NumPy in an embedded Python EXE? I.e. I want to also "embed" NumPy (as a .zip, directory, .dll, or .pyd etc.). I've tried adding the NumPy includes and also linking to npymath.lib but I get the same import error. I've also dug through some Cython wrapper code but haven't found a solution. Here is the Python embedded sample code:

#include <Python.h>
#include <iostream>

int main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pDict, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr, "Usage: runpy pythonfile funcname [args]\n");
        return 1;
    }

    Py_SetPath(L"python35.zip"); //this is in the current directory
    Py_Initialize();
    pName = PyUnicode_DecodeFSDefault(argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyLong_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr, "Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    Py_Finalize();
    return 0;
}

Embed file is here: https://www.python.org/ftp/python/3.5.2/python-3.5.2-embed-amd64.zip, python35.zip inside the archive. Here is the simple test script (runpy eig eig 10 to test - note if you don't embed Python35.zip and have NumPy / SciPy installed it WILL run):

eig.py

import numpy as np
from scipy import linalg
def eig(a):
    c = np.random.rand(a,a)*100
    c = np.corrcoef(c)
    print('You are taking the eigsh of a ', a, '^2 matrix')
    e, f = linalg.eig(c)
    return print('Eigvals are: ',np.diag(f))

Anyone know how to fix this issue? Much appreciated.

Update: Here's the compiled version x64 Python 3.5 Windows NumPy SciPy and Pandas with Intel MKL included: https://www.dropbox.com/sh/2smbgen2i9ilf2e/AADI8A3pCAFU-EqNLTbOiUwJa?dl=0

Rouleau answered 12/7, 2016 at 15:36 Comment(0)
C
0

This does not work because numpy is not in the zipfile python35.zip. The runpy-program sets the path to python35.zip: It is thus the only path in the Pythonpath for this programs exception... You have to add the parent-folder of your local numpy-folder also to the Pythonpath to make it working.

Conrad answered 14/7, 2016 at 21:58 Comment(10)
I tried that by zipping up NumPy and SciPy from site-packages and placing in the EXE directory: `Py_SetPath(L"python35.zip; numpy.zip; scipy.zip"). This didn't work so I got rid of the .zip and put the whole NumPy and SciPy directories with the EXE. Still gives the same error - anything different you suggest? Thanks.Rouleau
Yes: Add numpy to a subfolder (for example mylibs) and add this one to the path. If you add numpy directly to the path, then it is normal that the interpreter does not find it. This is because the interpreter goes into each folder of the python path to search for numpy -> The interpreter is thus searching for numpy in python35.zip and .... in numpy...Conrad
Hmm it ran 1x then I tried to create Python35.zip with NumPy and Scipy in the root and it just errors out: Traceback (most recent call last): File "eig.py", line 7, in <module> import numpy as np File "\python35.zip\numpy_init_.py", line 180, in <module> File "\python35.zip\numpy\add_newdocs.py", line 13, in <module> File "\python35.zip\numpy\lib_init_.py", line 8, in <module> File "\python35.zip\numpy\lib\type_check.py", line 11, in <module> File "\python35.zip\numpy\core_init_.py", line 14, in <module> ImportError: cannot import name 'multiarray' Failed to load "eig"Rouleau
Seems like that NumPy error is pretty common on frozen local installations, although I can't seem to fix it by placing the multiarray.pyd anywhere in the zip files...Rouleau
Did you try to put the multiarry.pyd in the folder of the python binary instead of in a zip file? (Like the pyd files shipped by Python). Perhaps pyd-files can't be loaded from zip-file...Conrad
Yes just tried that as well - the multiarray.PYD doesn't work in Python35.zip, extension_modules.zip, or just extracted into the EXE directory unfortunately.Rouleau
anything else you can add here to help with Python embedding? You seem to be the only SO user responding. Something maybe showing an example of importing NumPy inside the c++ code? That way I have something to give you bounty (not that you need it!). Appreciate it.Rouleau
Sorry no... I don't use numpy myself.Conrad
gave you bounty, thanks for helping out, even if I didn't get everything working in a zip file (left the accept answer open for that if it ever happens...)Rouleau
Just an update, I did get everything working by putting everything but the embedded Python35.zip in a subfolder for each package under the name "extension_modules" \ [packagename] and changing the C++ code line to Py_SetPath(L"python35.zip;extension_modules"); that means I copied the installed NumPy Scipy etc. under that folder.Rouleau

© 2022 - 2025 — McMap. All rights reserved.