Python multiprocessing copy-on-write behaving differently between OSX and Ubuntu

Question

I'm trying to share objects between the parent and child process in Python. To play around with the idea, I've created a simple Python script:

from multiprocessing import Process
from os import getpid

import psutil

shared = list(range(20000000))

def shared_printer():
    mem = psutil.Process(getpid()).memory_info().rss / (1024 ** 2)
    print(getpid(), len(shared), '{}MB'.format(mem))

if __name__ == '__main__':
    p = Process(target=shared_printer)
    p.start()
    shared_printer()
    p.join()

The code snippet uses the excellent psutil library to print the RSS (Resident Set Size). When I run this on OSX with Python 2.7.15, I get the following output:

(33101, 20000000, '1MB')
(33100, 20000000, '626MB')

When I run the exact same snippet on Ubuntu (Linux 4.15.0-1029-aws #30-Ubuntu SMP x86_64 GNU/Linux), I get the following output:

(4077, 20000000, '632MB')
(4078, 20000000, '629MB')

Notice that the child process' RSS is basicall 0MB on OSX and about the same size as the parent process' RSS in Linux. I had assumed that copy-on-write behavior would work the same way in Linux and allow the child process to refer to the parent process' memory for most pages (perhaps except the one storing the head of the object).

So I'm guessing that there's some difference in the copy-on-write behavior in the 2 systems. My question is: is there anything I can do in Linux to get that OSX-like copy-on-write behavior?

Assistance

Have a look at the top answer to that question #1268752 – Macrography

Might just be a matter of how the OS reports RSS values? In addition to looking at the process's memory, check free memory before/after the fork and see if 600MB disappears. – Colettecoleus

see this answer: https://mcmap.net/q/25489/-what-is-rss-and-vsz-in-linux-memory-management – Prudish

All your libs are probably statically linked in CPython for MAC and not on Ubuntu ? Look at superuser.com/questions/1162037/… – Titian

Is the result consistent? Just OOC, what happens if you explicitly import gc and add gc.disable() just before creating the Process? It's possible cycle collection runs are messing with reference counts and forcing more copy-on-write (not likely to be consistent, but worth ruling out). Also try adding print(multiprocessing.get_start_method()) to be sure there isn't some weirdness with one OS changing the default start method from the non-Windows default of fork to forkserver or spawn for some reason. – Spermous

@mat.viguier: Your comment and your link seem unrelated. Static linkage is about C libraries, pre-compiling modules just changes whether the .pyc files are generated at install time or at runtime, but either way, it shouldn't affect the behavior of fork. – Spermous

@Spermous my first thought was also about the GC, but the behavior is consistent regardless. I'm thinking it has to do with how XNU systems reports memory. See this question on host_statistics (which is used by psutil) and the answer as to why its results are inconsistent across platforms. – Kishke

@Spermous : Off course you're right. It's related to the way the CPython on MAC and the CPython on Linux are using the underlying library. If OP is sure the libraries are the same, my comment is completely out of the scope. If not, it is possible that the copy-on-write (as itself, not just a fork) just doesn't work on one of the tow hosts ... – Titian

Answer 1

So I'm guessing that there's some difference in the copy-on-write behavior >in the 2 systems. My question is: is there anything I can do in Linux to >get that OSX-like copy-on-write behavior?

The answer is NO. Behind the command psutil.Process(getpid()).memory_info().rss / (1024 ** 2) the OS uses the UNIX command $top [PID] and search for the field RES. Which contains the non-swapped physical memory a task has used in kb. i.e. RES = CODE + DATA.

IMHO, these means that both OS uses different memory managers. So that, it's almost impossible to constrain how much memory a process uses/needs. This is a intern issue of the OS. In Linux the child process has the same size of the parent process. Indeed, they copy the same stack, code and data. But different PCB (Process Control Block). Therefore, it is impossible to get close to 0 as OSX does. It smells that OSX does not literally copy the code and data. If they are the same code, it will make pointer to the data of the parent process.

PD: I hope that would help you!

Valoniah

Both Linux and OSX should be using fork with the same copy-on-write semantics for the memory in existence at the moment the child forks off from the parent. I'd be more inclined to suspect a difference in how OSX counts memory, not in how much it uses. Both of them should have almost no real memory usage immediately after the fork, but Linux may report the memory use assuming all copy-on-write pages end up copied (thanks to CPython's reference counted+cycle collector design, most probably will), while OSX only reports unique private, non-copy-on-write, pages. – Spermous