Python extension module with variable number of arguments

Question

I am trying to figure out how in C extension modules to have a variable (and maybe) quite large number of arguments to a function.

Reading about PyArg_ParseTuple it seems you have to know how many to accept, some mandatory and some optional but all with their own variable. I was hoping PyArg_UnpackTuple would be able to handle this but it seems to just give me bus errors when I try and use it in what appears to be the wrong way.

As an example take the following python code that one might want to make into an extension module (in C).

def hypot(*vals):
    if len(vals) !=1 :
        return math.sqrt(sum((v ** 2 for v in vals)))
    else: 
        return math.sqrt(sum((v ** 2 for v in vals[0])))

This can be called with any number of arguments or iterated over, hypot(3,4,5), hypot([3,4,5]), and hypot(*[3,4,5]) all give the same answer.

The start of my C function looks like this

static PyObject *hypot_tb(PyObject *self, PyObject *args) {
// lots of code
// PyArg_ParseTuple or PyArg_UnpackTuple
}

---

Many thinks to yasar11732. Here for the next guy is a fully working extension module (_toolboxmodule.c) that simply takes in any number or integer arguments and returns a list made up of those arguments (with a poor name). A toy but illustrates what needed to be done.

#include <Python.h>

int ParseArguments(long arr[],Py_ssize_t size, PyObject *args) {
    /* Get arbitrary number of positive numbers from Py_Tuple */
    Py_ssize_t i;
    PyObject *temp_p, *temp_p2;

    for (i=0;i<size;i++) {
        temp_p = PyTuple_GetItem(args,i);
        if(temp_p == NULL) {return NULL;}

        /* Check if temp_p is numeric */
        if (PyNumber_Check(temp_p) != 1) {
            PyErr_SetString(PyExc_TypeError,"Non-numeric argument.");
            return NULL;
        }

        /* Convert number to python long and than C unsigned long */
        temp_p2 = PyNumber_Long(temp_p);
        arr[i] = PyLong_AsUnsignedLong(temp_p2);
        Py_DECREF(temp_p2);
    }
    return 1;
}

static PyObject *hypot_tb(PyObject *self, PyObject *args)
{
    Py_ssize_t TupleSize = PyTuple_Size(args);
    long *nums = malloc(TupleSize * sizeof(unsigned long));
    PyObject *list_out;
    int i;

    if(!TupleSize) {
        if(!PyErr_Occurred()) 
            PyErr_SetString(PyExc_TypeError,"You must supply at least one argument.");
        return NULL;
    }
    if (!(ParseArguments(nums, TupleSize, args)) { 
        free(nums);
        return NULL;
    }

    list_out = PyList_New(TupleSize);
    for(i=0;i<TupleSize;i++)
        PyList_SET_ITEM(list_out, i, PyInt_FromLong(nums[i]));
    free(nums);
    return (PyObject *)list_out;
}

static PyMethodDef toolbox_methods[] = {
   { "hypot", (PyCFunction)hypot_tb, METH_VARARGS,
     "Add docs here\n"},
    // NULL terminate Python looking at the object
     { NULL, NULL, 0, NULL }
};

PyMODINIT_FUNC init_toolbox(void) {
    Py_InitModule3("_toolbox", toolbox_methods,
                     "toolbox module");
}

In python then it is:

>>> import _toolbox
>>> _toolbox.hypot(*range(4, 10))
[4, 5, 6, 7, 8, 9]

Superbomb

Why do you tell us that you're getting crashes from / having difficulty with the PyArg_* functions, and then show us everything except how you use the PyArg_* functions? – Madame

You should put your ParseArguments inside an if statement to see if there was errors during parsing (returned null), and make a cleanup and return null if there were errors. Otherwise you will supress errors during argument parsing. – Turning

Yes, yes you are right. I'll edit the post. – Superbomb

Answer 1

I had used something like this earlier. It could be a bad code as I am not an experienced C coder, but it worked for me. The idea is, *args is just a Python tuple, and you can do anything that you could do with a Python tuple. You can check http://docs.python.org/c-api/tuple.html .

int
ParseArguments(unsigned long arr[],Py_ssize_t size, PyObject *args) {
    /* Get arbitrary number of positive numbers from Py_Tuple */
    Py_ssize_t i;
    PyObject *temp_p, *temp_p2;


    for (i=0;i<size;i++) {
        temp_p = PyTuple_GetItem(args,i);
        if(temp_p == NULL) {return NULL;}

        /* Check if temp_p is numeric */
        if (PyNumber_Check(temp_p) != 1) {
            PyErr_SetString(PyExc_TypeError,"Non-numeric argument.");
            return NULL;
        }

        /* Convert number to python long and than C unsigned long */
        temp_p2 = PyNumber_Long(temp_p);
        arr[i] = PyLong_AsUnsignedLong(temp_p2);
        Py_DECREF(temp_p2);
        if (arr[i] == 0) {
            PyErr_SetString(PyExc_ValueError,"Zero doesn't allowed as argument.");
            return NULL;
        }
        if (PyErr_Occurred()) {return NULL; }
    }

    return 1;
}

I was calling this function like this:

static PyObject *
function_name_was_here(PyObject *self, PyObject *args)
{
    Py_ssize_t TupleSize = PyTuple_Size(args);
    Py_ssize_t i;
    struct bigcouples *temp = malloc(sizeof(struct bigcouples));
    unsigned long current;

    if(!TupleSize) {
        if(!PyErr_Occurred()) 
            PyErr_SetString(PyExc_TypeError,"You must supply at least one argument.");
        free(temp);
        return NULL;
    }

    unsigned long *nums = malloc(TupleSize * sizeof(unsigned long));

    if(!ParseArguments(nums, TupleSize, args)){
        /* Make a cleanup and than return null*/
        return null;
    }

Turning

Wow, am I glad I asked; you nailed it. A little tweaking and my issue was solved. I will edit my question with my fully working solution based on yours for the next guy. – Superbomb

Answer 2

Thanks @yasar, the template is much appreciated. And the following adaptation may be too simplified - but minimal is always good, no?

ctst.c:

/*
 *  https://realpython.com/build-python-c-extension-module
 *
 *  gcc -O2 -fPIC -c ctst.c $(python3-config --includes)
 *  gcc -shared ctst.o -o ctst.so
 */

#include <Python.h>

static PyObject *
cargs(PyObject *self, PyObject *args)
{
    Py_ssize_t n_args = PyTuple_Size(args);
    PyObject * lst = PyList_New(0);

    for (int i = 0; i < n_args; i++) {
        PyObject * item = PyTuple_GetItem(args, i);
        // play with items one at a time ...
        // yes, malloc is needed to play with all
        // items outside this loop
        PyList_Append(lst, item);
    }

    return lst;
}

static PyMethodDef MyCtstMethods[] = {
    {"cargs", fcrgs, METH_VARARGS,
    "toss a bunch of Python vars into a C function, return the list (maybe) modified"},
    {NULL, NULL, 0, NULL}
};


static struct PyModuleDef ctst = {
    PyModuleDef_HEAD_INIT, "ctst", "testing C calls", -1, MyCtstMethods
};

PyMODINIT_FUNC
PyInit_ctst(void)
{
    return PyModule_Create(&ctst);
}

tst.py:

import ctst

print('test0 (null):', ctst.cargs())
print('test1 (ints):', ctst.cargs(1, 2, 3))
print('test2 (mix) :', ctst.cargs(1, 2, 3, 'hi'))
print('test3 (flt) :', ctst.cargs(0.1234))
print('test4 (lst) :', ctst.cargs(['a', 'b', 'c']))

Gathering