Properly discarding ctypes pointers to mmap memory in Python

Question

I have the issue that I cannot properly close mmap-s in Python after I created a pointer to them. My use case is that I open files (usually that is UIO-devices to work with hardware, but the issue also happens with normal files), memory-map them and then use them as buffers for ctypes-structures. Usually structs or arrays of data. A minimal example looks like this:

import ctypes as ct
import mmap
import os

fileno = os.open('/tmp/testfile', os.O_RDWR | os.O_SYNC)
map = mmap.mmap(fileno, 32768, flags=mmap.MAP_SHARED)
memory = (ct.c_uint32 * 8192).from_buffer(map)

# Use the memory object to do things here

del memory
map.close()
os.close(fileno)

Everything is fine at that point.

However, sometimes I need to call some C library functions that also need access to that memory so I have to pass in a pointer to them. I create that pointer using:

ptr = ct.cast(memory, ct.c_void_p)

All of that works very well except one thing. As soon as I create such a pointer, I can no longer close the memory map. Take this slightly extended example:

import ctypes as ct
import mmap
import os

fileno = os.open('/tmp/testfile', os.O_RDWR | os.O_SYNC)
map = mmap.mmap(fileno, 32768, flags=mmap.MAP_SHARED)
memory = (ct.c_uint32 * 8192).from_buffer(map)

# Use the memory object to do things here
ptr = ct.cast(memory, ct.c_void_p)
del ptr

del memory
map.close()
os.close(fileno)

Running this results in the following exception:

Traceback (most recent call last):
  File "pointer_test.py", line 14, in <module>
    map.close()
BufferError: cannot close exported pointers exist

Process finished with exit code 1

I've run some analysis (using gc.get_referrers) on what is referencing the map-instance and it turns out that there is a memoryview-instance still there. In the end, this traces back to the ctypes Array:

[<__main__.c_uint_Array_8192 object at 0x7f954bd1e0>,
 [{547965620704: <__main__.c_uint_Array_8192 object at 0x7f954bd1e0>,
   'ffffffff': <memory at 0x7f95621a08>},
  [<memory at 0x7f95621a08>,
   [<managedbuffer object at 0x7f95746d08>,

However, this does not really help me. I'd like to know how I can get rid of that pointer. I know that this is likely not fully safe since I could, of course, always have a copy of that pointer somewhere. But completely preventing closing the mmap after a pointer was created also does not seem like a good idea. Anyone knows how I can convince Python that I discarded all the pointers and it is now safe to close the mmap?

Teague

Did the same test as you and got the same result. I thought it was just a ref being kept by python somewhere, so I just did: import gc and gc.collect() and tried to close the mem mapped file immediately after that: it worked... I can't see any other way to tell python that there are no refs on the mem map :( – Grater

I'm stuck with this problem, too. In a larger code env where classes and finalizers are involved, it feels near-impossible to correctly track down all views of the mmap. Why can't we just get an "unsafe" mode that allows freeing the mmap even while views are still in existence? – Numbing

After wasting a lot of time, I finally found the causes. One was that a reference to a view ended up as last value of a for loop variable which had to be del'ed explicitly. The other was a reference on class level. We close the mmap in a weakref finalizer of said class. Since the object being finalized is not accessible, I presumed that the reference was gone, but apparently the object still exists while the finalizer is running. That creates a rather serious design problem, since we cannot get rid of the reference in the finalizer. Cf github.com/pypdfium2-team/pypdfium2/pull/237 – Numbing

Possible workaround is to store the attribute in question in a list (or other mutable object) and pass it through the finalizer, where we can clear the reference. To keep access convenient on class level, I guess it should be possible to create a wrapper property returning index 0 of the list attribute. – Numbing

Answer 1

find the memoryview instance and call memoryview-instance.release() first, then you can close the mmap instance.

Burkitt

Answer 2

Use addressof to launder the reference; i.e. ptr = ct.cast(ct.addressof(memory), ct.c_void_p)

Combs