How would one prevent MMAP from caching values?

Question

I've written a linux driver that ioremaps exports PCI BAR0 for a particular device to a sysfs binary attribute allowing userspace to directly control it.

The problem rears when I attempt to MMAP on top of the attribute to directly access that bit of memory (from a userland program). Reads succeed just fine and return expected values, though when I write to that memory it appears to be cached somewhere between the kernel and memory and not delivered to the GMCH root complex (and therefore the device). What I'd like to do is have an implicit write memory barrier after each access.

• Is there any way to prevent the kernel from caching writes to a mmap-ed bit of memory?

Follow ups:

• Is calling msync() after every access the "accepted" way to do this?

Zwinglian

To answer the follow-up ... you don't need msync() at all here because there is no file backing the mmap. What you might need though are some architecture specific instructions for various things ... for example to ensure correct order of hardware accesses on powerpc you may need the eieio instruction (someone has a sense of humour :-) ... same comment made here. – Spiegleman

Answer 1

Going to go ahead and answer this one myself with my solution.

In the Kernel driver from my sysfs mmap function, there is a macro in /include/asm/pgtable.h that sets the proper flags for a nocache'd pfn remap. It looks like this:

vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
if (io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
               vma->vm_end - vma->vm_start,
               vma->vm_page_prot))
    return -EAGAIN;

Additionally, in the userland mmap, I used the MAP_SHARED flag in the mmap flags argument.

The combination of the two ultimately did the trick.

Zwinglian

Thx so much man ! I had exactly the same problem as u and couldn't find a solution... – Ordination

+1 Worked for me too. Thank you for taking the time to post the solution here once you found it. :-) – Deploy

Answer 2

Might ioremap_nocache() help?

Terrazzo

I'm using io_remap_pfn_range to remap and translate the kernel pages into the userspace addresses. I've looked (and am using) ioremap_nocache() elsewhere and that manually sets the KERNEL_NOCACHE attribute on each page manually. I can't find any mention of that happening (or a flag to set) using io_remap_pfn_range. – Zwinglian

Additional: ioremap_nocache() is used pretty much only for mapping IO memory into kernel space and not down into userspace - that's why io_remap_pfn_range is handy. – Zwinglian