Get address of read-only mmap object

Question

I'm currently working on some code that shares some state between processes using a mmap object. The use case is a bunch of processes that have access to some read-only data in a shared mmap backed array. Part of this involves doing some arithmetic on the underlying memory representation, I'm using ctypes much like in this question to get a the underlying memory address.

I have a situation where want to be able to open this mmap file as read only by the processes that only read data. However when I do that I'm not sure how to get the pointer address in that case. The following is as close to a minimal example of this problem as I can make:

import mmap
import ctypes

filename = "test"

with open(filename, 'rb+') as fd:
    buf = mmap.mmap(fd.fileno(), 0)
    int_pointer = ctypes.c_int.from_buffer(buf)

with open(filename, 'rb') as fd:
    test_mmap_ro = mmap.mmap(
        fd.fileno(), 0, access=mmap.ACCESS_READ,
        )
    int_pointer2 = ctypes.c_int.from_buffer(test_mmap_ro) #fails here

Running on Python3 this fails with this:

TypeError: must be read-write buffer, not mmap.mmap

While Python2 gives this:

TypeError: mmap can't modify a readonly memory map.

Given that I actually want to use a read only memory map how can I do this? I'll change to a writeable mmap if I have to, but I would rather not if there's another way to do this, so any suggestions or workarounds would be appreciated.

Catabasis

I think it's worth mentioning that because I'm already using CFFI in this project I'm open to some sort of solution that leverages that. – Catabasis

"Part of this involves doing some arithmetic on the underlying memory representation" -- Why do you need the absolute address? Could you give an example? – Orobanchaceous

Say you have some code in c that opens the mmap and writes some structs to it. In order to get access to that data in python you might need to be able to find offsets into that struct at the level of individual bytes. – Catabasis

Couldn't you just slice the mmap object? -- When I'm parsing binary data with Python, I usually just call from_buffer_copy() method of a ctypes.Structure subclass on a sub region of the mmap. – Orobanchaceous

Answer 1

Using ctypes I ended up with this:

obj = ctypes.py_object(buf)
address = ctypes.c_void_p()
length = ctypes.c_ssize_t()
ctypes.pythonapi.PyObject_AsReadBuffer(obj, ctypes.byref(address), ctypes.byref(length))
int_pointer = address.value

Catabasis

Answer 2

To obtain a char* pointer to your read-only memory map (mmap.mmap instance), which you can then pass to your C functions, the key is CFFI's ffi.from_buffer() (docs).

So, if you already have:

import mmap
with open("my_huge_file.bin", "rb") as stream:
    buf = mmap.mmap(stream.fileno(), 0, access=mmap.ACCESS_READ)

To call a C function through CFFI, passing a pointer to the mapped memory, do like so:

from ._my_cffi_module import lib as my_cffi_module, ffi
cbuf = ffi.from_buffer(buf)
my_cffi_module.my_c_function(cbuf, 1, 2, 3)  # Whatever your parameters may be

Your C function can now freely read from the buffer, using whatever pointer arithmetic you need, but of course not write to it.

Refreshing