Pre-commit not finding python packages

Question

Im using pyenv to support having different versions of python.

In a project using python 3.7 I also want linting with pre-commit to run when doing code changes.

But when the lint rules run, pre-commit seem to be looking in a cache folder under the user for python packages and can thus not find them.. even though they are installed on system level. For example:

>git ci -m'lint test'  
...
...
/Users/[]/.cache/pre-commit/repo180cws4u/py_env-python3.7/lib/python3.7/site-packages/ 
...
ModuleNotFoundError: No module named 'six'


>pip list | grep six
six                 1.11.0

How can I solve this. (A colleague did by running /Users/[]/.cache/pre-commit/repo180cws4u/bin/python install -m six which solved the issue, but feels wrong.. must be possible to configure to use system packages or similar..?

Telegenic

please show your actual configuration and un-doctored output -- you've obscured the useful information – Irrupt

Answer 1

pre-commit installs isolated environments for each of the tools such that they don't interfere with local development. It sounds like you're missing dependencies in that environment

Running pip install inside the cache environments is very not supported and you are likely to break pre-commit by doing that. They are intentionally private implementation detail so you don't mess with them!

You have two main options:

making your scripts work in isolation (the supported way)

set up your tools such that they specify their dependencies properly (you haven't shown your configuration, but my guess is you're calling some script?)

for repositories that get installed, the dependencies of your tool should be listed in your setup metadata (usually setup.py / setup.cfg / pyproject.toml)

For example, pre-commit/pre-commit-hooks specifies these dependencies:

# setup.cfg
install_requires =
    ruamel.yaml>=0.15
    toml

for local repositories with language: python, you should specify your dependencies in additional_dependencies -- for example:

-   repo: local
    hooks:
    -   id: run-the-thing
        name: run the thing
        entry: ./scripts/lint
        additional_dependencies: [six, ...]
        language: python
        types: [python]

repository local hooks (the escape hatch)

with language: system / language: script, pre-commit does not provision an environment. it is the responsibility of each user to set up such an environment. these are the escape hatch from the normal way because they kind of defeat the purpose of the framework in the first place (which is to provision the tools necessary on its own -- without every developer needing to carefully manage a bunch of tools)

For repo: local you'd use language: system and just hope your script produces a useful error message if the user's environment is misconfigured

-   repo: local
    hooks:
    -   id: run-the-thing
        name: run the thing
        entry: ./scripts/lint
        language: system
        types: [python]

---

disclaimer: I created pre-commit

Irrupt

Thank you, a good and precise answer! It feels like this should help quite a bit. However I cannot find our config.. we install pre-commit with pip install. We have some extra linting rules but not any config file(s) for pre-commit that I can find.. neither in repo or my home folder Where should I look? – Telegenic

the pre-commit configuration lives at ./.pre-commit-config.yaml – Irrupt