Does pip provide a TOML parser for python 3.9?

Question

I want to parse TOML files in python 3.9, and I am wondering if I can do so without installing another package.

Since pip knows how to work with pyproject.toml files, and I already have pip installed, does pip provide a parser that I can import and use in my own code?

Zenobia

It would be more accurate to say that pip has access to a parser; it doesn't really provide it, in the sense that you are welcome to use it. – Consequently

I think I understand. Although one can access it, and one can parse TOML files with it, the TOML parser inside pip is only for its internal use. I still like knowing because (a) MacGyver, and (b) I learn more about how python packages are managed and how "batteries included" is achieved. – Zenobia

Keep in mind: Python 3.11 now has a TOML parser in the standard library, so pip may stop providing a vendored parser in the future. – Consequently

(The above is my opinion; I have no knowledge of what the pip maintainers may be planning.) – Consequently

Answer 1

For 3.9, pip vendors tomli:

from pip._vendor import tomli

For consenting adults, importing the vendored module should work as usual. However, this is an implementation detail of pip, and it could change without a deprecation period at any moment in a future release of pip. Therefore, for anything apart from a quick hack, it would be safer to install tomli (or some other TOML parser) into the site instead.

Neom

And even pip will prefer a properly installed version of tomli to its vendored package if available. – Consequently

@Consequently Hmm, are you sure? I don't see evidence of that. It would seem to defeat the purpose of vendoring. – Neom

Oh, I was looking at a section in pip._vendor which only does the conditional import of the vendored code if a redistribution has explicitly patched pip._vendor.DEBUNDLED to be true. It uses a vendored function that tries to install a package normally before patching the vendored version into place. (I assumed the vendored version was just a backup of last resort.) – Consequently

Answer 2

So the workaround I started with was to try for tomllib and, failing that, import the vendored version as mentioned by wim:

try: import tomllib
except ModuleNotFoundError: import pip._vendor.tomli as tomllib

Since the APIs are the same (or close enough for what I needed), this worked well with no further code changes. I also confirmed that a comment above mentioning that it was pip._vendor.toml in Python 3.8 was incorrect; it's tomli, as it is in all the others.

However, this works with the "modern" version of pip, which is only usable on 3.8 and above. Python 3.7 and below have their own separate versions of getpip.py, and 3.6 does not seem to have a vendored TOML library in it. (3.7 i believe does.)

Since in my particular case I was just using this to do a tiny bit of parsing on pyproject.toml files before installing those packages and their dependencies, and I am always using a virtual environment for this anyway, I decided the best option was to keep things simple: directly install the tomli package from PyPI and use only it, saving myself the hassle of multiple different checks.

But if you're supporting only Python 3.7 and up, the above might be one of the easiest solutions.

Oireachtas

Answer 3

So, it seems that PIP vendors the toml library tomli. Just after searching for "pyproject.toml" in the pip github repo, I found the following function in the source code::

def load_pyproject_toml(
    use_pep517: Optional[bool], pyproject_toml: str, setup_py: str, req_name: str
) -> Optional[BuildSystemDetails]:
    """Load the pyproject.toml file.
    Parameters:
        use_pep517 - Has the user requested PEP 517 processing? None
                     means the user hasn't explicitly specified.
        pyproject_toml - Location of the project's pyproject.toml file
        setup_py - Location of the project's setup.py file
        req_name - The name of the requirement we're processing (for
                   error reporting)
    Returns:
        None if we should use the legacy code path, otherwise a tuple
        (
            requirements from pyproject.toml,
            name of PEP 517 backend,
            requirements we should check are installed after setting
                up the build environment
            directory paths to import the backend from (backend-path),
                relative to the project root.
        )
    """
    has_pyproject = os.path.isfile(pyproject_toml)
    has_setup = os.path.isfile(setup_py)

    if not has_pyproject and not has_setup:
        raise InstallationError(
            f"{req_name} does not appear to be a Python project: "
            f"neither 'setup.py' nor 'pyproject.toml' found."
        )

    if has_pyproject:
        with open(pyproject_toml, encoding="utf-8") as f:
            pp_toml = tomli.loads(f.read())
    ...

The tomli import at the top:

from pip._vendor import tomli

Checking in the REPL

>>> from pip._vendor import tomli
>>> tomli.loads('[foo]\nbar = "baz"')
{'foo': {'bar': 'baz'}}

Of course, this is not part of the public API, and you should really not rely on this.

Gunstock

Thanks for the details, teaching me how to fish. For anyone who has the same question for other versions of python, I went poking around on my pc: versions 3.7, 3.9, 3.10, 3.11 all installed with pip._vendor.tomli while version 3.8 had pip._vendor.toml. The difference brings home your point about not relying on it, in addition to it not being intended. Python 3.11 includes tomllib in the standard library, so no point snooping inside pip for it. – Zenobia