Write from mmapped buffer to `O_DIRECT` output file

Question

I have a device which writes to a video buffer. This buffer is allocated in system memory using CMA and I want to implement streaming write from this buffer to a block device. My application opens video buffer with mmap and I would like to use O_DIRECT write to avoid page cache related overhead. Basically, the pseudo-code of the application looks like this:

f_in = open("/dev/videobuf", O_RDONLY);
f_mmap = mmap(0, BUFFER_SIZE, PROT_READ, MAP_SHARED, f_in, 0);
f_out = open("/dev/sda", O_WRONLY | O_DIRECT);
write(f_out, f_mmap, BLOCK_SIZE);

where BLOCK_SIZE is sector aligned value. f_out is opened without any errors, but write results in EFAULT. I tried to track down this issue and it turned out that mmap implementation in video buffer's driver uses remap_pfn_range(), which sets VM_IO and VM_PFNMAP flags for VMA. The O_DIRECT path in block device drivers checks these flags and returns EFAULT. As far as I understand, O_DIRECT writes need to pin the memory pages, but VMA flags indicate the absence of struct page for underlying memory which causes an error. Am I right here?

And the main question is how to correctly implement O_DIRECT write from mmapped buffer? I have video buffer driver and can modify it appropriately.

I found similar question, but these is no clear answer there.

Ultimatum

Before modifying the video buffer driver, I'd first see if sendfile() works: off_t offset = 0L; ssize_t byteSent = sendfile( f_out, f_in, &offset, BLOCK_SIZE );, assuming the video buffer driver doesn't support lseek() or otherwise track file offset. (Or ssize_t byteSent = sendfile( f_out, f_in, NULL, BLOCK_SIZE );) If it works, that would avoid the mmap() and the need to pin user-space pages entirely. – Acedia

@AndrewHenle I tested sendfile() and it does not work in my situation. If f_in is a regular file than it works as expected, but if f_in is a video buffer than this function returns EINVAL. – Ultimatum

Answer 1

The function remap_pfn_range designates your virtual memory area as special by using pte_mkspecial and includes VM_IO/VM_PFNMAP in the vma. Consequently, it fails specific validations for Direct I/O.

Since your memory comes from CMA which already supports struct page, you can utilize vm_insert_pages with the following steps:

1. Add CMA region using kernel arguments or device tree source (DTS).
2. Get struct pages from CMA:

dma_page = dma_alloc_contiguous(&pdev->dev, size, GFP_KERNEL);
if (!dma_page) {
    pr_err("%s %d, dma_alloc_contiguous fail\n", __func__, __LINE__);
    return -ENOMEM;
}
nr_pages = DIV_ROUND_UP(size, PAGE_SIZE);
pages = kvmalloc_array(nr_pages, sizeof(*pages), GFP_KERNEL);
for (i = 0; i < nr_pages; i++)
    pages[i] = &dma_page[i];

3. Insert pages into vma when calling mmap:

int your_mmap(struct file *file, struct vm_area_struct *vma) {
    int ret = 0;
    unsigned long temp_nr_pages;

    if (vma->vm_end - vma->vm_start > size)
        return -EINVAL;

    /* Duplicate nr_pages as vm_insert_pages can change nr_pages */
    temp_nr_pages = nr_pages;

    ret = vm_insert_pages(vma, vma->vm_start, pages, &temp_nr_pages);
    if (ret < 0)
        pr_err("%s vm_insert_pages fail, error is %d\n", __func__, ret);

    return ret;
}

4. Export dma_alloc_contiguous (the only memory management code change, but it's acceptable).

modified   kernel/dma/contiguous.c
@@ -332,6 +332,7 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)

        return cma_alloc_aligned(dma_contiguous_default_area, size, gfp);  }
+EXPORT_SYMBOL(dma_alloc_contiguous);

 /**
  * dma_free_contiguous() - release allocated pages

Bate

Could you indicate where the direct I/O validation happen in the kernel code, please? I'm trying to do direct I/O from a buffer allocated with this driver (github.com/ikwzm/udmabuf/blob/master/u-dma-buf.c#L1102) which uses the dma_alloc_coherent function and marks the pages as VM_IO | VM_PFNMAP. Thanks. – Pettifogger