How can I configure pre-commit tool to rely on a single location for all hook configurations?

Question

Please note that this question is about pre-commit.com tool and not about installing git-hooks.

The tool uses a .pre-commit-config.yml in the root of your repository, one file that defines which hooks are to be used and that also pins them using git tags. The pinning is useful as it avoids introducing random breakages when a newer hook/linter is released.

This raise a new challenge if you have lots of repositories (>40) and your hooks are actively developed. On OpenStack repositories that already adopted pre-commit tool this created a new maintenance burden, a signifiant one, especially as average time spent in CI for each patch is over 5 hours.

Configuring pre-commit to use HEAD is also not an option as this would totally break too often. T

Due to this I am wondering if there is another setup which would allow me to have a centralized pre-commit repository that defines its config and I could configure most projects to use that.

If this is possible it would bump linters from a single place. We can take care of breaking some repositories without any problems because we can trigger builds on other repositories if we want or alternatively we can take-the-risk.

There is also another aspect of this, some linters do need their own configuration files like .ansible-lint, .flake8 files and it would be nice if we could also keep these in a centralized location. That aspect is more problematic because there will be a number of repositories where we would be forced to alter the default configuration. This means that the only way it would work is if centralized linter config would be used only when there is no in-repo config.

How can I achieve that?

Please note that any solution should not change the way users are already linting (tox -e linters which calls pre-commit run -a). Requiring users to manually clone another repository or to run a bash script that does some magic would be a deal breaker because it would introduce another thing that needs to be maintained.

Background

• https://github.com/pre-commit/pre-commit/issues/450 is kinda similar but does not really provide a suitable solution

Enjoyment

Answer 1

pre-commit is intentionally designed against centralized management because it makes it impossible to upgrade the centralized configuration without breaking lots of repositories. For example, we managed hundreds of repositories at yelp and when the systems team would upgrade flake8 a good percentages of them them would break!. At the core, pre-commit is designed for repeatability and per-repository customization. It provides a mechanism to make this way easier: pre-commit autoupdate. You can also use a distributed refactoring tool such as all-repos (which has direct pre-commit support) to make sweeping changes to the configuration (and have each individual repository tested in a repository).

That said, the design gives you quite a few escape hatches to allow unsupported pathways which could accomplish what you're looking for. The issue you linked in fact contains many of these solutions (it's where I've been trying to accumulate them!) -- I'll reiterate them here though.

---

using "pre-commit in pre-commit"

at the end of the day, pre-commit is just a tool which calls executables, why can't that executable be pre-commit? (it can)

layout

$ tree -I .git -a ../testrepo/
../testrepo/
├── .pre-commit-hooks.yaml
├── orghooks.yaml
└── run-org-hooks

0 directories, 3 files

.pre-commit-hooks.yaml

-   id: org-hook
    name: org-wide hooks
    language: script
    entry: ./run-org-hooks
    verbose: true

verbose: true forces the output to always appear whether or not things pass

./run-org-hooks (mode: 0755)

#!/usr/bin/env python
import os
import sys

HERE = os.path.dirname(os.path.realpath(__file__))


def main():
    cfg = os.path.join(HERE, 'orghooks.yaml')
    cmd = ['pre-commit', 'run', '--config', cfg, '--files'] + sys.argv[1:]
    os.execvp(cmd[0], cmd)


if __name__ == '__main__':
    exit(main())

Note here that we're using pre-commit's --config option as a sort of "include". --files is important here such that top-level pre-commit run --files ... is honored.

orghooks.yaml (basically any old .pre-commit-config.yaml!)

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.4.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
-   repo: https://github.com/asottile/add-trailing-comma
    rev: v1.5.0
    hooks:
    -   id: add-trailing-comma

consuming repository

.pre-commit-config.yaml

(note, I'm using local paths and a sha here because I wanted to test the idea, in reality you'd put this in a clonable repository and use tags)

repos:
-   repo: /tmp/wat/testrepo
    rev: 2d76bfbfddde6129c4ec5db31ac08abfbe362114
    hooks:
    -   id: org-hook

running

$ pre-commit run --all-files
org-wide hooks...........................................................Passed
hookid: org-hook

Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Add trailing commas......................................................Passed

$ pre-commit run --files README.md
org-wide hooks...........................................................Passed
hookid: org-hook

Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Add trailing commas..................................(no files to check)Skipped

$ SKIP=trailing-whitespace pre-commit run --files setup.py
org-wide hooks...........................................................Passed
hookid: org-hook

Trim Trailing Whitespace................................................Skipped
Fix End of Files.........................................................Passed
Add trailing commas......................................................Passed

---

symlinks

nothing currently requires that .pre-commit-config.yaml is a regular file. You could make it be a symlink

Some ideas on how a symlink might accomplish this:

1. conventionally cloned repository adjacent to working repositories (.pre-commit-config -> ../convention/pre-commit-config.yaml)
2. configuration management managed file (.pre-commit-config.yaml -> /etc/pre-commit/pre-commit-config.yaml)
3. submodule (.pre-commit-config.yaml -> submodule/pre-commit-config.yaml)

---

Don't use pre-commit install and use your own shell script for .git/hooks/pre-commit / etc.

pre-commit install is a convenience, but not necessary to work. it's ~essentially a wrapper around pre-commit run, you could easily substitute that script for a shell script which calls pre-commit run --config /etc/pre-commit/pre-commit-config.yaml.

---

disclaimers:

• I am the author of pre-commit
• I am the author of all-repos

Wool

Thank you for this! I find for this to work I had to also add require_serial: true to the .pre-commit-hooks.yaml otherwise the process would run multiple times in a row – Sr