Python extensions with C: staticforward
Asked Answered
D

2

2

So I needed to use the code of the subprocess module to add some functionality I needed. When I was trying to compile the _subprocess.c file, it gives this error message:

Error 1 error C2086: 'PyTypeObject sp_handle_type' : redefinition

This is the code part which is relevant from _subprocess.c file:

typedef struct {
    PyObject_HEAD
    HANDLE handle;
} sp_handle_object;

staticforward PyTypeObject sp_handle_type;

static PyObject*
sp_handle_new(HANDLE handle)
{
    sp_handle_object* self;

    self = PyObject_NEW(sp_handle_object, &sp_handle_type);
    if (self == NULL)
        return NULL;

    self->handle = handle;

    return (PyObject*)self;
}

#if defined(MS_WIN32) && !defined(MS_WIN64)
#define HANDLE_TO_PYNUM(handle) PyInt_FromLong((long) handle)
#define PY_HANDLE_PARAM "l"
#else
#define HANDLE_TO_PYNUM(handle) PyLong_FromLongLong((long long) handle)
#define PY_HANDLE_PARAM "L"
#endif

static PyObject*
sp_handle_detach(sp_handle_object* self, PyObject* args)
{
    HANDLE handle;

    if (!PyArg_ParseTuple(args, ":Detach"))
        return NULL;

    handle = self->handle;

    self->handle = INVALID_HANDLE_VALUE;

    /* note: return the current handle, as an integer */
    return HANDLE_TO_PYNUM(handle);
}

static PyObject*
sp_handle_close(sp_handle_object* self, PyObject* args)
{
    if (!PyArg_ParseTuple(args, ":Close"))
        return NULL;

    if (self->handle != INVALID_HANDLE_VALUE) {
        CloseHandle(self->handle);
        self->handle = INVALID_HANDLE_VALUE;
    }
    Py_INCREF(Py_None);
    return Py_None;
}

static void
sp_handle_dealloc(sp_handle_object* self)
{
    if (self->handle != INVALID_HANDLE_VALUE)
        CloseHandle(self->handle);
    PyObject_FREE(self);
}

static PyMethodDef sp_handle_methods[] = {
    { "Detach", (PyCFunction)sp_handle_detach, METH_VARARGS },
    { "Close", (PyCFunction)sp_handle_close, METH_VARARGS },
    { NULL, NULL }
};

static PyObject*
sp_handle_getattr(sp_handle_object* self, char* name)
{
    return Py_FindMethod(sp_handle_methods, (PyObject*)self, name);
}

static PyObject*
sp_handle_as_int(sp_handle_object* self)
{
    return HANDLE_TO_PYNUM(self->handle);
}

static PyNumberMethods sp_handle_as_number;

statichere PyTypeObject sp_handle_type = {
    PyObject_HEAD_INIT(NULL)
    0,                                  /*ob_size*/
    "_subprocess_handle", sizeof(sp_handle_object), 0,
    (destructor)sp_handle_dealloc, /*tp_dealloc*/
    0, /*tp_print*/
    (getattrfunc)sp_handle_getattr,/*tp_getattr*/
    0,                                  /*tp_setattr*/
    0,                                  /*tp_compare*/
    0,                                  /*tp_repr*/
    &sp_handle_as_number,               /*tp_as_number */
    0,                                  /*tp_as_sequence */
    0,                                  /*tp_as_mapping */
    0                                   /*tp_hash*/
};`

Also I've found that:

#define staticforward static
#define statichere static

I don't understand what am I doing wrong. Any help would be appreciated. Btw (I'm not sure if it's relevant), I'm using Visual Studio Professional 2013 to compile this file.

Demakis answered 25/1, 2017 at 21:31 Comment(0)
M
1

Notes:

  • I'm talking about Python 2.7 here (since in newer versions, the subprocess module no longer has an own C implementation for Win)
  • Python 2.7 is built (officially) using VStudio2008 (9.0) according to [Python.Wiki]: WindowsCompilers. Building it with a newer (or better: different) version, might yield some other (and harder to find) errors. For example, when I built it with VStudio 2010 (10.0) (I used the built version to run a complex set of (.py*) scripts), I had some trouble at runtime when encountering socket related errors because of some mismatches between errno and WSA* codes, that were changed between the 2 versions

When I tested, I couldn't understand why you encountered the issue and I didn't, then for a while I forgot about it, then when you posted the last comment, it started eating me alive again. As I said I was able to successfully compile the file using VStudio 2010 / 2013.

Trying to compile the same (this was only an assumption) code with different results -> the way that code is compiled might differ. Therefore, I started investigating for other possible definition places for staticforward and statichere (besides lines 878 / 879 of object.h) due to conditional macros: #if, #ifdef, ... . But, I couldn't find any. So, I added some simpler statements:

staticforward int i;
statichere int i = 2;

then, I manually replaced the defines:

static int i;
static int i = 2;

in _subprocess.c (by coincidence, I added them at line #137 - just before statichere PyTypeObject sp_handle_type = { - fact that prevented me to figure out the problem at this point), and it still compiled!!!

Next step, I pasted the above lines in another solution that I had open (in a .cpp source file), and I was able to reproduce the error. So, I payed more attention to the compiler flags (I copy/pasted from the x86 Debug settings of the projects automatically converted by VStudio from the ones found in the PCbuild folder):

  • VStudio 2013

    /GS
    /analyze-
    /W3
    /Gy
    /Zc:wchar_t
    /I"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\Python"
    /I"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\Modules\zlib"
    /I"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\Include"
    /I"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\PC"
    /Zi
    /Gm-
    /Od
    /Fd"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\PCbuild\obj\win32_Debug\pythoncore\vc120.pdb"
    /fp:precise
    /D "_USRDLL"
    /D "Py_BUILD_CORE"
    /D "Py_ENABLE_SHARED"
    /D "MS_DLL_ID=\"2.7-32\""
    /D "WIN32"
    /D "_WIN32"
    /D "_DEBUG"
    /D "_WINDLL"
    /errorReport:prompt
    /GF
    /WX-
    /Zc:forScope
    /Gd
    /Oy-
    /Oi
    /MDd
    /Fa"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\PCbuild\obj\win32_Debug\pythoncore\"
    /nologo
    /Fo"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\PCbuild\obj\win32_Debug\pythoncore\"
    /Fp"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.11-vs2k13\PCbuild\obj\win32_Debug\pythoncore\python27_d.pch"
    
  • VStudio 2010

    /I"..\Python"
    /I"..\Modules\zlib"
    /I"..\Include"
    /I"..\PC"
    /Zi
    /nologo
    /W3
    /WX-
    /Od
    /Oy-
    /D "_USRDLL"
    /D "Py_BUILD_CORE"
    /D "Py_ENABLE_SHARED"
    /D "WIN32"
    /D "_DEBUG"
    /D "_WIN32"
    /D "_WINDLL"
    /GF
    /Gm-
    /MDd
    /GS
    /Gy
    /fp:precise
    /Zc:wchar_t
    /Zc:forScope
    /Fp"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.10-vcbuild\PCbuild\Win32-temp-Debug\pythoncore\pythoncore.pch"
    /Fa"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.10-vcbuild\PCbuild\Win32-temp-Debug\pythoncore\"
    /Fo"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.10-vcbuild\PCbuild\Win32-temp-Debug\pythoncore\"
    /Fd"E:\Work\Dev\Fati\WinBuild\OPSWpython27\src\Python-2.7.10-vcbuild\PCbuild\Win32-temp-Debug\pythoncore\vc100.pdb"
    /Gd
    /analyze-
    /errorReport:queue
    

and then it struck me: It's the way of how the file is compiled: C vs. C++ (the [MS.Docs]: /Tc, /Tp, /TC, /TP (Specify Source File Type) flag). Of course, compiling _subprocess.c as C++ would spit your error.

Check [SO]: Creating a dynamically allocated struct with a 2D dynamically allocated string (@CristiFati's answer), for (a little bit) more details, and how the same mistake triggered very different errors.

Mammary answered 14/3, 2017 at 17:56 Comment(1)
Does this qualify as an answer to your question?Mammary
D
-1

Ok I found an answer.

It turns out that the definition of staticforward should have been extern instead of static. My compiler didn't know how to handle it. I guess in other compilers it works ok.

Demakis answered 26/1, 2017 at 9:22 Comment(6)
This file (as all others) should compile, OOTB. I've tried a couple of combinations (VC2k10 vs Py27 as an eg.), but wasn't able to reproduce the issue. You should probably (look for dups), and close the issue. Anyway , I don't see this as a solution.Mammary
In Visual Studio 2013, you can't define a static variable twice. staticforward and statichere were both defiend as "static" so in fact, the error is legit. Changing staticforward to extern tells the compiler that the variable will be defined later.Demakis
"In Visual Studio 2013, you can't define a static variable twice. staticforward and statichere were both defeend as 'static' so in fact, the error is legit". This couldn't be more wrong. In any VStudio (or any VS, or C) compiler version, static means that the varables are only visible in the complation unit (c / c++ file). Please check the internet (Google, MSDN, Wiki ,... and tons of others) for details.Mammary
It's kind of hillarous that (you think that) a whole community did such a mistake (naming staticforward(staic) but meaning extern) and in 2017 you were the one who found it. BTW, on my box, Py2.7.10 compiles OOTB using VS2k13....now, the question: what did you do to subprocess.c (or one of the headers included) to mess things up so bad?Mammary
What were you really trying to do?Mammary
@Mammary I extended the subrocess module to support spawning processes with a given NUMA node and affinity mask (like in the windows 'start' command).Demakis

© 2022 - 2024 — McMap. All rights reserved.