Verifying compatibility in compiling extension types, and using them with cdef

Question

Standing questions:

• Why do other errors in Cython compilation point to the specific line of error, while this doesn't?

Prior to update:

Due to difficulty in compiling extension types, as referenced in the 'won't compile' link below, it was thought that the AssertionError was related to the extension types (after their instantiation was transferred to the main pyx file).

Someone from this forum post said that “If you actually want extension types, and want to declare their types in your code, you need to declare their C attributes in a .pxd file.”

If that is true in this case, the problem is that the pyx files won’t compile for me when using pxd files. There is no error when compiling the extension type in one pyx file, but then I receive the error extTypeName is not a type identifier when the extension type is used after its import.

I was told in the comments that extension types can be defined in the pyx file with the main function using them. When doing so, I receive an error I don’t know how to trace (AssertionError: unexpected type int and base type tuple object for indexing). The full traceback is posted below.

---

Difficult to trace error

Traceback when trying to compile with the extension types declared in the same pyx file (which I’m not sure is the source of the issue):

Traceback (most recent call last):
  File "setup.py", line 37, in <module>
    ext_modules = [Extension("HintToCRdict", ["HintToCRdict.pyx"])]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 164, in run
    _build_ext.build_ext.run(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 337, in run
    self.build_extensions()
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 324, in cython_sources
    full_module_name=module_name)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Main.py", line 688, in compile
    return compile_single(source, options, full_module_name)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Main.py", line 638, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Main.py", line 495, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Pipeline.py", line 365, in run_pipeline
    data = phase(data)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ModuleNode.py", line 131, in process_implementation
    self.generate_c_code(env, options, result)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ModuleNode.py", line 359, in generate_c_code
    self.body.generate_function_definitions(env, code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 435, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 1944, in generate_function_definitions
    self.generate_function_body(env, code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 1698, in generate_function_body
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5851, in generate_execution_code
    if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5894, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/UtilNodes.py", line 321, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6402, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/UtilNodes.py", line 321, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6402, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5851, in generate_execution_code
    if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5894, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/UtilNodes.py", line 89, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6015, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6192, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 4786, in generate_execution_code
    self.generate_rhs_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5073, in generate_rhs_evaluation_code
    self.rhs.generate_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 699, in generate_evaluation_code
    self.generate_subexpr_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 714, in generate_subexpr_evaluation_code
    node.generate_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 699, in generate_evaluation_code
    self.generate_subexpr_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 714, in generate_subexpr_evaluation_code
    node.generate_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 705, in generate_evaluation_code
    self.generate_result_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 3737, in generate_result_code
    self.type, self.base.type)
AssertionError: unexpected type int and base type tuple object for indexing

Bloodroot

Hi, for further information at this point I would need a minimum working example that triggers the error. The excerpts that you posted in the other question did not reveal any error so they are probably either in the function prototypes are in the actual code. – Helprin

Your suggestion to strip down to a basic example helped to narrow down that the extension types may not be the problem. I may need to post something more comprehensive soon. In the mean time, feel free to post an answer that may help resolve any other issues in this post (e.g. if you know whether the custom extension type with list and dict may be compatible with a cdef function). – Bloodroot

@Phillip You commented on one of my posts suggesting I might be able to help. I'm honestly quite puzzled about the error. Mu guess would be some issue with your Cython installation or possibly you file paths, but I don't know what I'd do to fix it. – Analogue

Thanks for looking. An edit has just been posted with the solution, although the cause of error is still unknown. – Bloodroot

A few points about the other details: a .pxd file enables other Cython modules to use the lowlevel C details of a Cython extension type but any module (Puthon or Cython) can use the extension type through its Python interface. cdef functions are only usable from Cython, not Python (which makes calling them faster). The "internals" of a cdef and def function are mostly treated in the same way by Cython. As a rule I'd make most functions def, and only cdef the ones that are called often but only used internally by Cythom. – Analogue

I know these are novice questions, so please bear with me. The cdef function in question is called from a python script. To confirm, this would disqualify it as being defined with cdef? If not disqualified, my current confusion lies with whether an extension type cdef-defined with list and dict (not standard c types) could validly be used in a cdef function. – Bloodroot

One further thought base on your edit: you won't gain a lot of speed from cdef tuple since it's still basically treated as a Python class. Try replacing it with the more generic cdef object and I suspect the problem will go away. However, I think tuple should work, so it's probably worth reporting it as a bug github.com/cython/cython/issues – Analogue

@Phillip yes, cdef functions cannot be called from a Python script. A def function could be. You can use any Python type in a cdef function (but it may not be much faster than just writing Python). You can also specify and use Cython and C types within def functions. – Analogue

Feel free to post in answer format at your convenience – Bloodroot

@Phillip Having tested it I can't get that error on my computer so I can't really write anything useful as an answer! I'm tempted to create a (few?) self-answered questions to deal with the stuff about cdef functions/classes, just because they're common questions so it'd be good to have a clear definitive question on the subject without confusing tuple errors – Analogue

Answer 1

A reply with what I can take from your message.

1. The use of Python objects in Cython is possible but limited as soon as you cdef some parts. Try by "un"-cdefing the dict.
2. The error you have suggests that you have an expression that is x[y] where x is a tuple and y an int. In principle it should work all right (indexing a tuple with an int shoud be ok). So you can look at square brackets in the Cython to find out where it comes from.
3. You mentioned having removed the pxd file. Have you then move the full declaration of the extension class/other type definitions to the pyx file?

Helprin

1. If there is anything further that can be explained here, that may be helpful; I initially thought that the opposite would be true (that if something is untyped, it will not be compatible with a cdef function) 2. I've downloaded and am installing gdb in attempt to debug Cython; I'm not sure where the error is arising, and hopefully gdb can narrow it down. 3. Yes – Bloodroot

Normally when there is an error in the code, the compiler points directly to the line of error. It doesn't do that in the traceback listed above...not sure why. Hence I'm having trouble finding the line of error (if any). – Bloodroot

Hi, I cannot find anything to add, sorry. The syntax is probably right and the error cannot be detected before cythonization. Re-building your module with less and less cdefs (so more plain Python) should help to reveal the error. – Helprin

Answer 2

The extension types were not an issue in compilation (at least when declared within the same pyx file as the main cdef function); The error is arising elsewhere.

Resolved:

1. The AssertionError was caused by simple index access to this tuple.

Where a tuple was declared as:

cdef:
    tuple curIRs

…and set in nested loops as:

for deriv1 in xrange(len(L1)):
    for deriv2 in xrange(len(L2)):
        curIRs = (deriv2, deriv1)

…and tuple indexes are accessed:

d1[ix] = (curIRs[0], curIRs[1])

It was an apparently innocuous oversight that curIRs wasn’t just stored instead of essentially copying it by index access. However, this is where the error occurred for an unknown reason:

When (curIRs[0], curIRs[1]) is replaced with curIRs, the AssertionError ceases.

2. As stated in the comments, the main function in the pyx file can't be defined with cdef, since it is invoked in a python script.

---

Pierre de Buyl's suggestion in the comments was very helpful: 'remove cdef declarations of potential problem variables until the error is resolved.' i.e. where there is likely no error in Python, an incompatibility may be uncovered in Cython

Bloodroot