How to allocate memory in a DriverKit system extension and map it to another process?

Question

I have allocated memory in my application and passed its pointer and size to IOConnectCallStructMethod. Using IOMemoryDescriptor::CreateMapping I have then mapped this memory to the DriverKit system extension process, and it is possible to write to this mapped memory location and read the data from my application.

I would now like to do something similar for memory that is allocated in the system extension, and then map it to the application that is using the system extension. I would like to create a set of memory buffers in the system extension, and then write to it from the application and then signal to the system extension with IOConnectCallScalarMethod that a given buffer should be sent to the USB device, using IOUSBHostPipe::AsyncIO. When the CompleteAsyncIO callback then comes as a result of the sending completing, I would notify back to the application that it is now possible to copy data to the first buffer that was sent. The mechanism for this could probably be done using IOConnectCallAsyncStructMethod, and the OSAction object that is created in the system extension. What I don't understand is how to map memory allocated in the system extension to the application.

Kuhn

I have answered your question but note that if you retain the structureOutputDescriptor and/or structureInputDescriptor of your external method's IOUserClientMethodArguments, you can continue to use that descriptor in the driver as you wish. It really depends which approach is more convenient for your use case. There is one downside to the structure descriptor approach: structure inputs and outputs only turn up in the dext/kext as a memory descriptor if they're more than 4096 bytes long. Otherwise they're copied by value into/from the OSData* typed field instead. – Dneprodzerzhinsk

All of the IOMemoryDescriptor that are accessible in the IOUserClientMethodArguments are allocated by the application, right? Can I pass an IOMemoryDescriptor that points to memory allocated by the application to for example IOUSBHostPipe::AsyncIO? Or do I have to allocate any descriptor that is used with IOUSBHostPipe::AsyncIO with IOUSBHostInterface::CreateIOBuffer? – Kuhn

You can safely pass the memory descriptors from IOUserClientMethodArguments to IOUSBHostPipe::AsyncIO. In most cases, this will be a zero-copy operation. The downsides are the awkward size threshold and the fact that you can't subdivide or concatenate the IOMemoryDescriptor in any way. (I think you can drop trailing bytes by passing a smaller length to AsyncIO though.) – Dneprodzerzhinsk

IOUSBHostInterface::CreateIOBuffer should be used for buffers allocated for USB I/O in preference to IOBufferMemoryDescriptor::Create - i.e. if you were already going to be allocating your buffer memory inside the dext anyway. – Dneprodzerzhinsk

Incidentally, in kexts, you have IOSubMemoryDescriptor and IOMultiMemoryDescriptor which you can use to splice memory descriptors from near enough any source before using them for I/O. (In fact you can define your own subclasses which can apply whatever splicing logic you like.) I've reported their absence in DriverKit as a bug to Apple, if those would be helpful to you I recommend you do the same. – Dneprodzerzhinsk

ok, then I will go with the solution of allocating in the application and pass the buffer with for example IOConnectCallAsyncStructMethod. The maximum buffer size I want to send is larger than 4096 so the buffer should appear as an IOMemoryDescriptor. Yes, I am planning to pass the length of the data written to that buffer as the dataBufferLength parameter of AsyncIO. I did not have any need for IOSubMemoryDescriptor or IOMultiMemoryDescriptor yet, but I will keep them in mind and submit a request for them to be added if I find a use case for them. – Kuhn

Just to be clear: Note that it's not about the maximum buffer size, but the actual size passed to IOConnectCallAsyncMethod's struct argument(s). So if your external method can legally be called with a buffer of 4096 bytes or less, you have to special case it and copy to an IOBufferMemoryDescriptor. It's super annoying. For output structs, this means you can't actually asynchronously fill the buffer if it's 4096 bytes or fewer, because the structureOutput field is an OSData* and synchronously returned. – Dneprodzerzhinsk

The 4096 byte limit is defined via the io_struct_inband_t type by the way. (I'd rather use sizeof(io_struct_inband_t) than a magic 4096 in my special-casing code.) – Dneprodzerzhinsk

ok, sound good to use the sizeof(io_struct_inband_t). I just realised that IOSubMemoryDescriptor would be handy. The max packet size of the USB device that I am communicating with is much smaller that the maximum size of the buffers that I want to send so somehow the buffers needs to be split. What would be the best way to ask Apple to add something similar to DriverKit? – Kuhn

Answer 1

This is what IOUserClient::CopyClientMemoryForType in DriverKit is for, which gets invoked when your user process calls IOConnectMapMemory64 from IOKit.framework. The kext equivalent, incidentally, is IOUserClient::clientMemoryForType and essentially works exactly the same.

To make it work, you need to override the CopyClientMemoryForType virtual function in your user client subclass.

In the class definition in .iig:

virtual kern_return_t CopyClientMemoryForType(
    uint64_t type, uint64_t *options, IOMemoryDescriptor **memory) override;

In the implementation .cpp, something along these lines:

kern_return_t IMPL(MyUserClient, CopyClientMemoryForType) //(uint64_t type, uint64_t *options, IOMemoryDescriptor **memory)
{
    kern_return_t res;
    if (type == 0)
    {
        IOBufferMemoryDescriptor* buffer = nullptr;
        res = IOBufferMemoryDescriptor::Create(kIOMemoryDirectionInOut, 128 /* capacity */, 8 /* alignment */, &buffer);
        if (res != kIOReturnSuccess)
        {
            os_log(OS_LOG_DEFAULT, "MyUserClient::CopyClientMemoryForType(): IOBufferMemoryDescriptor::Create failed: 0x%x", res);
        }
        else
        {
            *memory = buffer; // returned with refcount 1
        }
    }
    else
    {
        res = this->CopyClientMemoryForType(type, options, memory, SUPERDISPATCH);
    }
    return res;
}

In user space, you would call:

    mach_vm_address_t address = 0;
    mach_vm_size_t size = 0;
    IOReturn res = IOConnectMapMemory64(connection, 0 /*memoryType*/, mach_task_self(), &address, &size, kIOMapAnywhere);

Some notes on this:

• The value in the type parameter comes from the memoryType parameter to the IOConnectMapMemory64 call that caused this function to be called. Your driver therefore can have some kind of numbering convention; in the simplest case you can treat it similarly to the selector in external methods.
• memory is effectively an output parameter and this is where you're expected to return the memory descriptor you want to map into user space when your function returns kIOReturnSuccess. The function has copy semantics, i.e. the caller expects to take ownership of the memory descriptor, i.e. it will eventually drop the reference count by 1 when it is no longer needed. The returned memory descriptor need not be an IOBufferMemoryDescriptor as I've used in the example, it can also be a PCI BAR or whatever.
• The kIOMapAnywhere option in the IOConnectMapMemory64 call is important and normally what you want: if you don't specify this, the atAddress parameter becomes an in-out parameter, and the caller is expected to select a location in the address space where the driver memory should be mapped. Normally you don't care where this is, and indeed specifying an explicit location can be dangerous if there's already something mapped there.
• If user space must not write to the mapped memory, set the options parameter to CopyClientMemoryForType accordingly: *options = kIOUserClientMemoryReadOnly;

To destroy the mapping, the user space process must call IOConnectUnmapMemory64().

Dneprodzerzhinsk

Hi @pmdj, the problem of memory mapping has also troubled me for a long time. IOConnectMapMemory64/CopyClientMemoryForType, Can this mode map the memory of the application layer to the driver layer? I tried it, CopyClientMemoryForType (ivars->dmaMemDesc = *memory;) crashed after running.Is the method I used wrong? – Dubbin

I have another question. When the memory of the PCIe bar is mapped to the application, if the PCIe device is turned off and the application is operating this memory, it will cause the system to crash. Is there a safer way to exit? Or not crash – Dubbin

@Dubbin You can only map driver memory into user space using IOConnectMapMemory64. For the reverse, use the external method struct argument API. (with some limitations) – Dneprodzerzhinsk

@Dubbin You shouldn't really be memory-mapping BARs at all nowadays; this isn't supported on arm64, and IIRC deprecated on x86-64. On x86-64, accesses to BARs that have disappeared should result in reads returning 0xffffffff. Using the explicit BAR read/write methods is safest, in any case. – Dneprodzerzhinsk

Thanks @pmdj. I understand. Refer to stackoverflow.com/questions/64267540. If I use external method to send kernel space memory less than or equal to 4096 size to user space, problems will occur. args->structureOutputDescriptor and args->structureOutput are both nullptr. Is this normal? How to output data less than or equal to 4096 size? – Dubbin

@Dubbin When there's no memory descriptor, you need to set structureOutput to an OSData object yourself before returning from the external method. OSData objects are immutable in DriverKit, so there's no point in the runtime pre-filling it. – Dneprodzerzhinsk