Prevent a Python-embedded to look in my default path C:\Python38 for modules

Question

I'm using Cython in --embed mode to produce a .exe. I'm evaluating the Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine. To do this, I only copy a minimal number of files from the Python Windows embeddable package.

In order to check this, I need to be sure that the current process I'm testing doesn't in fact use my system default Python install, i.e. C:\Python38.

To do this, I open a new cmd.exe and do set PATH= which temporarily removes everything from the PATH. Then I can test any self-compiled app.exe and make sure it doesn't reuse C:\Python38's files under the hood.

It works, except for the modules. Even after doing set PATH=, my code app.py

import json
print(json.dumps({"a":"b"}))

when Cython---embed-compiled into a .exe works, but it still uses C:\Python38\Lib\json\__init__.py! I know this for sure, because if I temporarily remove this file, my .exe now fails, because it cannot find the json module.

How to completely remove any link to C:\Python38 when debugging a Python program which shouldn't use these files?

Why isn't set PATH= enough? Which other environment variable does it use for modules? I checked all my system variables and I think I don't find any which seems related to Python.

Africa

PATH has to do with where Windows looks for exes. It has nothing to do with where Python looks for modules – Taraxacum

@Taraxacum Ok, but why does an embedded version of python (installed somewhere else) look for modules in my C:\Python38 folder? Is there a Windows registry key that informs about this, or another environment variable? – Africa

Answer 1

Thanks to @ead's answer and his link getpath.c finally redirecting to getpathp.c in the case of Windows, we can learn that the rule for building the path for module etc. is:

• current directory first

• PYTHONPATH env. variable

• registry key HKEY_LOCAL_MACHINE\SOFTWARE\Python or the same in HKCU

• PYTHONHOME env. variable

• finally:

  Iff - we can not locate the Python Home, have not had a PYTHONPATH specified, and can't locate any Registry entries (ie, we have nothing we can assume is a good path), a default path with relative entries is used (eg. .\Lib;.\DLLs, etc)

---

Conclusion: in order to debug an embedded version of Python, without interfering with the default system install (C:\Python38 in my case), I finally solved it by temporarily renaming the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Python to HKEY_LOCAL_MACHINE\SOFTWARE\PythonOld.

Side note: I'm not sure I will ever revert this registry key back to normal: my normal Python install shouldn't need it anyway to find its path, since when I run python.exe from anywhere (it is in the PATH for everyday use), it will automatically look in .\Lib\ and .\DLL\ which is correct. I don't see a single use case in which my normal install python.exe wouldn't find its subdir .\Lib\ or .\DLL\ and requiring the registry for this. In which use case would the registry be necessary? if python.exe is started then its path has been found, and it can take its .\Lib subfolder, without help from registry. I think 99,99% of the time this registry feature is doing more harm than good, preventing a Python install to be really "portable" (i.e. that we can move from one folder to another).

---

Notes:

• To be 100% sure, I also did this in command line, but I don't think it's necessary:

  set PATH=
  set PYTHONPATH=
  set PYTHONHOME=
  

• Might be helpful to do debugging of an embedded Python: import ctypes. If you haven't _ctypes.pyd and libffi-7.dll in your embedded install folder, it should fail. If it doesn't, this means it looks somewhere else (probably in your default system-wide Python install).

Africa

Answer 2

Python has a quite complicated heuristic for finding its "installation" (see for example this SO-question or this description), so probably it doesn't find the installation you are providing but the "default" installation.

Probably the most simple way is to set the environment variable PYTHONPATH pointing to the desired installation prior to start of the embedded interpreter.

By examination of sys.path one can check whether the correct installation was found.

Heth

Thanks. I have tried set PATH= and set PYTHONPATH= in cmd.exe before running the Cython---embed-executable in the same terminal, but this exe still looks in C:\Python38... Indeed in this exe, I added print(sys.path) and it reveals C:\python38 is still in it. Why? How to disable this? – Africa

@Africa this is really hard to tell, without knowing everything about your system, as the algorithm for finding the installation is quite complicated. For example we don't know what can be found inside your installation: if "landmark" site.py isn't there the algorithm will not get the desired output. Sadly, we could only guess what could go wrong. – Heth

Solved thanks to your help here, thanks again @ead! – Africa