What is the current limit on memory in Pyodide, and how do I maximize it? For example, does it depend on the client browser, or does it default to the RAM of the client machine?
Related old question:
Memory limits in webassembly
Memory limits in Pyodide
Asked Answered
The current memory limit in Pyodide is 2 GB due to its use of Emscripten targeting 32-bit WebAssembly architecture.
You can check this experimentally in the Pyodide REPL with the following code,
>>> import numpy as np
>>> import gc
>>> x = np.ones(528535456, dtype=np.float32); print(f'size={x.nbytes/1048576:.2f} MB'); del x; gc.collect()
size=2016.20 MB
practically the largest allocatable array is around 2GB, if you increase the array size more it would fail at the allocation time.
The maximum size addressable with a 32 bit architecture is 4GB, it would be possible to enable it in Emscripten however it has a cost in terms of generated code size, and so far we have not done so in Pyodide. If you need this feature please open an issue in the Pyodide issue tracker.
In the long term, the WASM memory64 proposal aims to extend memory use beyond 2 or 4 GB. It currently has experimental support in Emscripten and is being experimented in browsers. Once it's stabilized, and assuming there are no performance downsides, it's possible that Pyodide would be able switch to it.
© 2022 - 2024 — McMap. All rights reserved.