Windows ring buffer without copying

Question

On Ring Buffer's Wikipedia entry, there's example code showing a hack for UNIX systems whereby the adjacent virtual memory to a piece of memory is mapped to the same phbysical memory, thus implementing a ring buffer without the need for any memcpy, etc. I was wondering if there's a way to so something similar in Windows?

Thanks, Fraser

Promontory

The example in question doesn't eliminate a need for memcpy, it eliminates a need to do a DMA operation in two fragments when the operation would step past the end of the allocation for the buffer. It will only work in the virtual address space since it depends on arranging for a page-sized buffer being mapped to two places in virtual memory. – Gleam

Errr... yeah, that. In my case, I'm passing the pointer to another (library) function that does the writing, so that function can just do its own memcpy. – Promontory

Answer 1

I didn't really follow all the details of the example in wikipedia. With that in mind, you map memory in Windows using CreateFileMapping and MapViewOfFile, however MapViewOfFile does not allow you to specify a base address for the mapping. MapViewOfFileEx can be used to specify a base address so maybe you could use a similar technique.

I don't have any way of telling if this would actually work:

// determine valid buffer size
SYSTEM_INFO info;
GetSystemInfo(&info);

// note that the base address must be a multiple of the allocation granularity
DWORD bufferSize=info.dwAllocationGranularity;

HANDLE hMapFile = CreateFileMapping(
             INVALID_HANDLE_VALUE,
             NULL,
             PAGE_READWRITE,
             0,
             bufferSize*2,
             L"Mapping");

BYTE *pBuf = (BYTE*)MapViewOfFile(hMapFile,
                    FILE_MAP_ALL_ACCESS,
                    0,                   
                    0,                   
                    bufferSize);
MapViewOfFileEx(hMapFile,
                    FILE_MAP_ALL_ACCESS,
                    0,                   
                    0,                   
                    bufferSize,
                    pBuf+bufferSize);

Medusa

Sorry my initial reaction. I didn't really expect that would have worked – Medusa

;-P Actually, I think CreateFileMapping only needs to allocate bufferSize, not bufferSize*2... I'm not getting Access Violations, anyway. – Promontory

Answer 2

Starting with Windows 10, you can use VirtualAlloc2 & MapViewOfFile3 to create a circular buffer, as shown in the Examples section on the VirtualAlloc2 MSDN page.

Contrary to the comment by Ben Voigt, VirtualAlloc can't be used to reserve memory for MapViewOfFile. According to MSDN:

No other memory allocation can take place in the region that is used for mapping, including the use of the VirtualAlloc or VirtualAllocEx function to reserve memory.

As a poor workaround on older Windows versions, you can reserve a virtual memory area with VirtualAlloc, then release it with VirtualFree and attempt to create a mapping in the newly-freed area. Obviously this approach is subject to races against other threads attempting to allocate virtual memory.

Cabretta

Answer 3

Oh hey, this is the topic which worried me a lot lately. I needed posix-optimised ring buffer on Windows, mostly because of its random-access interface, but never had any idea on how to implement it. Now, the code proposed by @1800 INFORMATION works sometimes, sometimes it doesn't, but the idea is great anyway.

The thing is, MapViewOfFileEx sometimes fails with ERROR_INVALID_ADDRESS meaning that it cannot map the view to pBuf+bufferSize. This is because the MapViewOfFile called before selects a free address space of bufferSize length (starting from pBuf), but it doesn't garantee this address space to be bufferSize*2 long. And why would we need bufferSize*2 virtual memory? Because our ring buffer needs to wrap. This is what the second mapping view is for. When the read or write pointer leaves the first view, it enters the second view (because they are contigous in memory), but actually it starts over at the same mapping.

UINT_PTR addr;
HANDLE hMapFile;
LPVOID address, address2;

hMapFile = CreateFileMapping (    // create a mapping backed by a pagefile
    INVALID_HANDLE_VALUE,
    NULL,
    PAGE_EXECUTE_READWRITE,
    0,
    bufferSize*2,
    "Local\\mapping" );
if(hMapFile == NULL) 
    FAIL(CreateFileMapping);

address = MapViewOfFile (    // find a free bufferSize*2 address space
    hMapFile,
    FILE_MAP_ALL_ACCESS,
    0,                   
    0,                   
    bufferSize*2 );
if(address==NULL) 
    FAIL(MapViewOfFile);
UnmapViewOfFile(address);
// found it. hopefully it'll remain free while we map to it

addr = ((UINT_PTR)address);
address = MapViewOfFileEx (
    hMapFile,
    FILE_MAP_ALL_ACCESS,
    0,                   
    0,                   
    bufferSize, 
    (LPVOID)addr );

addr = ((UINT_PTR)address) + bufferSize;        
address2 = MapViewOfFileEx (
    hMapFile,
    FILE_MAP_ALL_ACCESS,
    0,                   
    0,                   
    bufferSize,
    (LPVOID)addr);  

if(address2==NULL)      
    FAIL(MapViewOfFileEx);

// when you're done with your ring buffer, call UnmapViewOfFile for 
// address and address2 and CloseHandle(hMapFile)

Woodsy

"it doesn't guarantee this address space to be bufferSize*2 long" -- this can be fixed by using VirtualAlloc with the MEM_RESERVE flag, then two calls to MapViewOfFileEx, passing the two halves of the single address range found by VirtualAlloc. – Pitre