Does Python support zero-copy I/O?

Question

I have two open file objects, dest and src. File object dest is opened for writing, with the seek position placed at some offset within the file, and file object src is opened for reading. What I need to do is simply read from the current position in src to EOF and transfer the contents to dest as quickly as possible.

If I were programming in Java, I could utilize the FileChannel#transferTo() method to perform zero-copy file I/O.

Does Python also support zero-copy?

Wavy

Answer 1

Since version 3.3, Python has os.sendfile, which interfaces to various Unix variants' sendfile(2) zero-copy I/O interfaces. It operates on file descriptors, not general file-like objects. For older Pythons, there's py-sendfile.

Etty

Answer 2

Since Python 3.8, you can use shutil.copyfile (and others from shutil) which will internally use zero-copy if possible, such as os.sendfile, and if not possible, fall back to a simple read-write loop.

See the shutil docs for details. Or issue 33671 (Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)). And the corresponding (merged) pull request.

You might also be interested in copy-on-write support or server-side copy support. See here, here. The os.copy_file_range (since Python 3.8) would do that. See issue 37159 (Use copy_file_range() in shutil.copyfile()) (maybe Python 3.9 or 3.10).

Sarnen