1. Intro
I noticed this question (or very similar / duplicate ones) several times lately, but none of them provides a clear path to consistently run into the issue (environment creation, commands run in order to reach this state). Also, considering the frequency this question is viewed, one could only conclude this is a fairly common problem.
Posting some guidelines to prevent users from running into it, and also some investigation techniques meant to let users know how to get out of it once hit.
Notes:
I wasn't able to run into this - well unless purposely messing up the environment
Error is about _ctypes.pyd, but the principle applies to any other .pyd (.dll)
Although I'm conducting my investigation on Win, concepts also apply to Nix
I am using (at answer date, upgraded constantly from older versions) Anaconda:
2. CTypes considerations
[Python.Docs]: ctypes - A foreign function library for Python is a library (initially started by Thomas Heller, then adopted by Python's standard library a long long time ago), designed to interact with (mainly, call functions from) .dlls (.sos) written in other (lower level) languages.
There are tons of examples on the internet (SO included), I'm going to exemplify a few (of mine):
CTypes uses LibFFI ([GitHub]: libffi/libffi - A portable foreign-function interface library - written mostly in C and ASM) under the hood.
It consists of 2 parts:
The core: written in C (which uses LibFFI). Located at [GitHub]: pytgon/cpython - (main) cpython/Modules/_ctypes. This does all the magic, and it's an extension module ([Python.Docs]: Extending Python with C or C++). It's a .dll (.so) file named _ctypes.pyd on Win (this is how I'm going to reference it), or _ctypes*.so on Nix (check [Python.PEPs]: PEP 3149 - ABI version tagged .so files for more naming convention details)
The Python wrappers. Located at [GitHub]: pytgon/cpython - (main) cpython/Lib/ctypes. Provides a Python friendly access to the core
The way _ctypes.pyd depends on LibFFI varies very much on OS and Python distribution / version.
2.1. CPython
On some OSes (Win included) LibFFI code (at least the needed part) was duplicated in Python codebase and built into _ctypes.pyd. Although it was easier at the beginning, it's not a good practice as it becomes very hard to maintain if one needs to keep the referred software up to date, and also licensing might become a problem. So, since Python 3.8 they stopped doing that ([GitHub]: pytgon/cpython - bpo-45022: Pin current libffi build to fixed version in preparation for upcoming update, [GitHub]: pytgon/cpython - bpo-45022: Update libffi to 3.4.2 in Windows build). Differences (check for libffi_msvc):
[GitHub]: pytgon/cpython - (3.7) cpython/Modules/_ctypes
[GitHub]: pytgon/cpython - (3.8) cpython/Modules/_ctypes
This change has some implications:
Build: LibFFI code (API) is required on the build machine and Python code needs to know where it is (similar to Linux world where libffi-dev (or equivalent) is required)
LibFFI is built in a stand alone .dll (libffi*.dll) that _ctypes.pyd depends on (and must be present at runtime (when the module is loaded)).
On Win, that .dll is shipped by the Python installer
Illustrating the differences on various OSes:
Couple of words about Nix (Linux) where a similar error occurs.
Ubuntu:
Running the command ls /usr/lib/x86_64-linux-gnu/libffi*.so* | xargs dpkg -S will output 3 packages owning files of interest: LibFFI 8, LibFFI 7, LibFFI-Dev.
Depending on Python version, the appropriate package should be installed (3rd one isn't required, but adding it just in case):
apt install libffi8 libffi7 libffi-dev
CentOS:
Following the same steps (different tools) yields:
yum install libffi libffi-devel
More details can be found at [SO]: No module named '_ctypes'.
2.2. Anaconda
Naturally, Anaconda's distribution of Python follows CPython, but took things a bit further.
Note: In the following (Win) snippets (command outputs), due to space reasons, the Anaconda's installation base path (f:\Install\pc064\Anaconda\Anaconda\Version) will be replaced by a placeholder: ${ANACONDA_INSTALL_PATH} (Nix style env var).
(base) [cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q073458524]> :: ------- Anaconda Prompt -------
(base) [cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q073458524]> cd /d f:\Install\pc064\Anaconda\Anaconda\Version
(base) [cfati@CFATI-5510-0:f:\Install\pc064\Anaconda\Anaconda\Version]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[prompt]> :: Reactivate base to have active environment in prompt
[prompt]> conda deactivate & conda activate base
(base) [prompt]> :: ------- Anaconda Prompt (still) -------
(base) [prompt]> :: ------- ${ANACONDA_INSTALL_PATH} is a placeholder for f:\Install\pc064\Anaconda\Anaconda\Version -------
(base) [prompt]> conda env list
# conda environments:
#
F:\Install\pc032\Intel\OneAPI\Version\intelpython\python3.7
F:\Install\pc032\Intel\OneAPI\Version\intelpython\python3.7\envs\2021.1.1
base * ${ANACONDA_INSTALL_PATH}
py_pc032_030602_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc032_030602_00
py_pc064_030610_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030610_00
py_pc064_030704_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030704_00
py_pc064_030716_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030716_00
py_pc064_030800_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030800_00
py_pc064_030808_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030808_00
py_pc064_030817_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030817_00
py_pc064_030900_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030900_00
py_pc064_030917_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030917_00
py_pc064_030917_01 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_030917_01
py_pc064_031000_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_031000_00
py_pc064_031006_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_031006_00
py_pc064_031012_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_00
py_pc064_031012_01 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_01
py_pc064_031104_00 ${ANACONDA_INSTALL_PATH}\envs\py_pc064_031104_00
(base) [prompt]>
(base) [prompt]> :: Search environments for _ctypes.pyd
(base) [prompt]> dir /B /S "envs\*_ctypes.pyd"
${ANACONDA_INSTALL_PATH}\envs\py_pc032_030602_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030610_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030610_00\DLLs\instrumented\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030704_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030704_00\DLLs\instrumented\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030716_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030800_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030800_00\DLLs\instrumented\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030808_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030817_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030900_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030917_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030917_01\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031000_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031006_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_00\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_01\DLLs\_ctypes.pyd
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031104_00\DLLs\_ctypes.pyd
(base) [prompt]>
(base) [prompt]> :: Search environments for the FFI dll
(base) [prompt]> dir /B /S "envs\*ffi*.dll"
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030800_00\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030808_00\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030817_00\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030817_00\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030817_00\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030817_00\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030900_00\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030917_00\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_030917_01\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031000_00\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031000_00\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031000_00\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031006_00\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031006_00\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031006_00\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_00\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_00\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_00\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_01\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_01\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031012_01\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031104_00\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031104_00\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\envs\py_pc064_031104_00\Library\bin\ffi.dll
As seen, there are 2 types of .dlls:
libffi*.dll (located in the same directory as _ctypes.pyd)
ffi*.dll (located in ..\Library\bin relative to _ctypes.pyd)
I discovered empirically what I'm stating here (I don't have an official source - at least not yet).
#1. comes from CPython (also, older versions don't have the libffi*.dll), and are distributed by the Python package.
#2. comes from LibFFI package (was split out of Python - it now offers a greater granularity, separate upgrades, ...). There are 3 different .dlls (ffi.dll, ffi-7.dll, ffi-8.dll), but they are just copies (don't know why weren't they SymLinked). This seems to be the newer approach, preferred by newer Python versions (I think starting with v3.10). Worth mentioning v3.8.17 which seems to contain both forms1.
Scanning inside packages, the paths match (once the leading environment part is dropped):
(base) [prompt]> :: ------- Anaconda Prompt (still) -------
(base) [prompt]> :: ------- ${ANACONDA_INSTALL_PATH} is a placeholder for f:\Install\pc064\Anaconda\Anaconda\Version -------
(base) [prompt]> dir /B /S "pkgs\*ffi*.dll"
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.2-hd77b12b_4\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.2-hd77b12b_4\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.2-hd77b12b_4\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.2-hd77b12b_6\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.2-hd77b12b_6\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.2-hd77b12b_6\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.4-hd77b12b_0\Library\bin\ffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.4-hd77b12b_0\Library\bin\ffi-8.dll
${ANACONDA_INSTALL_PATH}\pkgs\libffi-3.4.4-hd77b12b_0\Library\bin\ffi.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.8.0-hff0d562_2\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.8.12-h6244533_0\DLLs\libffi-8.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.8.13-h6244533_0\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.8.17-h1aa4202_0\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.8.5-h5fd99cc_1\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.8.8-hdbf39b2_5\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.9.0-h6244533_2\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.9.12-h6244533_0\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.9.16-h6244533_2\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.9.17-h1aa4202_0\DLLs\libffi-7.dll
${ANACONDA_INSTALL_PATH}\pkgs\python-3.9.17-h6244533_0\DLLs\libffi-7.dll
3. The error
Although, the error mentions _ctypes.pyd, it's not necessary that that .dll is problematic (and often isn't), but one of its dependencies (or dependencies dependencies, ...). Even if it's about a different error, [SO]: Python Ctypes - loading dll throws OSError: [WinError 193] %1 is not a valid Win32 application (@CristiFati's answer) is a thorough investigation, and it's the same principle (this error is mentioned somewhere at the end).
While we're here, it probably also worth reading:
3.1. Avoiding it
There are a bunch of generic (common sense) rules that should keep anyone following them out of trouble. Needless to say that if there's a bug in one of the packages (out of user's control), the error might still pop up:
Be aware of the Python instance being used (active): [SO]: How to install a package for a specific Python version on Windows 10? (@CristiFati's answer)
Work from Conda prompt
Only use Conda managed environments: [Conda.Docs]: Managing environments. Technically things could work if using other types (VEnv, plain CPython, ...), but there's pretty high chance of running into trouble (and if you're reading this, you probably shouldn't)
Activate an environment and only use that one. Avoid mixing them ([SO]: PyCharm doesn't recognize installed module (@CristiFati's answer)). Activation sets some env vars specific to that environment:
(base) [prompt]> :: ------- Anaconda Prompt (still) -------
(base) [prompt]> :: --- PATH in active (base) environment ---
(base) [prompt]> echo off & (for %g in ("%PATH:;=" "%") do (echo %~g)) & echo on
f:\Install\pc064\Anaconda\Anaconda\Version
f:\Install\pc064\Anaconda\Anaconda\Version\Library\mingw-w64\bin
f:\Install\pc064\Anaconda\Anaconda\Version\Library\usr\bin
f:\Install\pc064\Anaconda\Anaconda\Version\Library\bin
f:\Install\pc064\Anaconda\Anaconda\Version\Scripts
f:\Install\pc064\Anaconda\Anaconda\Version\bin
f:\Install\pc064\Anaconda\Anaconda\Version\condabin
C:\WINDOWS\System32\WindowsPowerShell\v1.0
C:\WINDOWS\System32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\Install\pc064\Docker\Docker\Version\Docker\resources\bin
C:\Program Files\dotnet
e:\Work\Dev\Utils\current\Win
e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts
C:\Users\cfati\.dotnet\tools
.
(base) [prompt]>
(base) [prompt]> :: --- Activate Python 3.10.12 environment ---
(base) [prompt]> conda activate py_pc064_031012_00
(py_pc064_031012_00) [prompt]> :: --- PATH in active (py_pc064_031012_00) environment ---
(py_pc064_031012_00) [prompt]> echo off & (for %g in ("%PATH:;=" "%") do (echo %~g)) & echo on
f:\Install\pc064\Anaconda\Anaconda\Version\envs\py_pc064_031012_00
f:\Install\pc064\Anaconda\Anaconda\Version\envs\py_pc064_031012_00\Library\mingw-w64\bin
f:\Install\pc064\Anaconda\Anaconda\Version\envs\py_pc064_031012_00\Library\usr\bin
f:\Install\pc064\Anaconda\Anaconda\Version\envs\py_pc064_031012_00\Library\bin
f:\Install\pc064\Anaconda\Anaconda\Version\envs\py_pc064_031012_00\Scripts
f:\Install\pc064\Anaconda\Anaconda\Version\envs\py_pc064_031012_00\bin
f:\Install\pc064\Anaconda\Anaconda\Version\condabin
C:\WINDOWS\System32\WindowsPowerShell\v1.0
C:\WINDOWS\System32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\Install\pc064\Docker\Docker\Version\Docker\resources\bin
C:\Program Files\dotnet
e:\Work\Dev\Utils\current\Win
e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts
C:\Users\cfati\.dotnet\tools
.
It's easy to see that having variables set for an environment, but using another is a recipe for disaster
Prefer Conda packages (and pay attention to the channels they come from) over PIP. Although the latter will most likely work, they are mainly tested for CPython, and might miss some Anaconda specifics
3.2. Troubleshooting
There are ways of getting out of the error. [SO]: Discover missing module using command-line ("DLL load failed" error) (@CristiFati's answer) contains a number of tools able to help pinpointing a missing / bad dependency. In our case:
(py_pc064_031012_00) [prompt]> :: ------- Anaconda Prompt (still) -------
(py_pc064_031012_00) [prompt]> "f:\Install\pc064\LucasG\DependencyWalkerPolitistTexan\Version\DependenciesGui.exe" "envs\%CONDA_DEFAULT_ENV%\DLLs\_ctypes.pyd"
(py_pc064_031012_00) [prompt]> conda deactivate & conda activate py_pc064_030817_00
(py_pc064_030817_00) [prompt]> "f:\Install\pc064\LucasG\DependencyWalkerPolitistTexan\Version\DependenciesGui.exe" "envs\%CONDA_DEFAULT_ENV%\DLLs\_ctypes.pyd"
Python 3.10.12 (containing and using LibFFI .dlls from LibFFI package (newer)):
![Img0]()
Python 3.8.17 (1) (containing LibFFI .dlls from both packages, but using the ones from Python (older)):
![Img1]()
As seen, the dependency tree (custom dependents at least) is simple. In this case it's a 99% chance that ffi.dll is missing. That can be fixed by executing (in the environment):
conda install libffi
Manually taking care of things (copying the files) is also an option, but it might come to bite in the ass at a later time.
Worth paying attention to the PATH env var, as it might contain directories with the same .dll names (before the "good" ones). Check [SO]: Can't get FontForge to import as a module in a custom Python script (@CristiFati's answer) (Update #0 section) for a similar problem.
In some cases the error might be triggered by some other factors (disk failure, manually deletion, faulty packages, ...), and in those cases recreating the environment could solve it.
Some other times, upgrading the environment or Conda ([Anaconda.Docs]: Updating from older versions) might help.
If the error is stubborn and just won't go away, reinstalling Anaconda altogether might be required, but only use that after lots of considerations and as a last resort.
4. Summary
If reaching this situation, here's what to do in order to get out of it:
Search _ctypes.pyd dependencies and if missing, install (inside the environment) the package that owns them (typically):
conda install libffi
If _ctypes.pyd is missing, reinstall its owning package:
conda install python
One could also copy (unpack) files from the right package, into the right (environment) paths, but that's for experienced users only
Check PATH env var, as it might contain paths with files (.dlls) interfering with the "right" ones
Upgrade / recreate the environment
Upgrade Conda
Reinstall Anaconda (!!! will lose all environments !!!)
dir "C:\Users\shing\anaconda3\DLLs\_ctypes.pyd"? Also the other environment is also Python 3.10.0? – Mockup