HuggingFace Transformers not getting installed in VS Code

Question

I'm new in VS code and in coding in general, I've been trying to install transformers with the command pip install transformers and pip install transformers[tf-cpu] both didn't work, with the following error:

Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\TevaJ\Desktop\Projects\venv\Scripts\python.exe' 'C:\Users\TevaJ\Desktop\Projects\venv\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' build_wheel 'C:\Users\TevaJ\AppData\Local\Temp\tmps54vamjd'
       cwd: C:\Users\TevaJ\AppData\Local\Temp\pip-install-kolpujzg\tokenizers_befed87ed1a749f8a23369cfa56db8a2
  Complete output (19 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.10
  creating build\lib.win-amd64-3.10\tokenizers
  copying tokenizers\__init__.py -> build\lib.win-amd64-3.10\tokenizers
  running build_ext
  error: can't find Rust compiler

  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

  To update pip, run:

      pip install --upgrade pip

  and then retry package installation.

  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
(venv) PS C:\Users\TevaJ\Desktop\Projects> 

I looked up on google but I can't find a solution. Does anyone know this error and how to fix it?

Stander

There was a similar problem and was supposed solution: https://mcmap.net/q/442432/-could-not-build-wheels-for-tokenizers-which-is-required-to-install-pyproject-toml-based-projects – Vitavitaceous

The solution did not work. I've tried to look deeper with the links, on github huggingface, rustup, wheel, cartography... nothing worked. – Stander

Refer to installation page, try if one helps. or install it without dependencies by the command pip install transformers=="3.1.0" --no-dependencies. – Quintanilla

@MollyWang-MSFT Unfortunately I need the dependencies for my researches... It did work to install transformers with your command tho. – Stander

Answer 1

Solved.

Just had to install it from source without dependencies with PIP 619.

git clone https://github.com/huggingface/transformers.git 
cd transformers
pip install -e .

Stander

Answer 2

Another possible solution - is install Rust compiler, make a restart and try pip install again.

Vitavitaceous

I guess it's not working either way, probably some packages/versions incompatible together. Been trying to solve it since couple days and i can't find a proper way to do so. Scraping the financial news hasn't been easy with the new transformers... – Stander

Answer 3

I try the following command.It works

conda install -c huggingface transformers

Exoteric

Answer 4

you cannot install Transformers version >2.3.0 with pip, try installing using conda instead, after installing rust compiler.

"conda install transformers" or "conda install -c huggingface transformers"

Secco