Huggingface AlBert tokenizer NoneType error with Colab

Question

I simply tried the sample code from hugging face website: https://huggingface.co/albert-base-v2

from transformers import AlbertTokenizer, AlbertModel
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')

then I got the following error at the tokenizer step: encoded_input = tokenizer(text, return_tensors='pt')

TypeError: 'NoneType' object is not callable

I tried the same code on my local machine, it worked no problem. The problem seems within Colab. However, I do need help to run this model on colab GPU.

My python version on colab is Python 3.6.9.

Flee

Can you please add the versions of the transformers library you are using (i.e. transformers.__version__)? – Tauro

Answer 1

I found the answer. After install, import the AlbertTokenizer and Tokenizer=..., I received an error asking me to install SentencePiece package. However, after I install this package and run tokenizer again, I started receiving the error above. So I open a brand new colab session, and install everything including the SentencePiece before creating tokenizer, and this time it worked. The Nonetype error simply means it doesn't know what is albert-base-v2. However if you install the packages in right order colab will recognize better the relationship between AlbertTokenizer and SentencePiece. In short for this to work in colab

0. Open a new Colab session
1. Install Transformers and SentencePiece
2. import AlbertTokenizer
3. create tokenizer.

Flee

Thanks! This helped. Sometimes it's the good old "did you tried to turn it on and off again?" – Tajuanatak

Answer 2

MeiNan Zhu's answer is correct.

Installing or importing SentencePiece before transformers works.

pip install Sentencepiece
!pip install transformers

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)

type(tokenizer)

transformers.models.xlnet.tokenization_xlnet.XLNetTokenizer

Planarian

Answer 3

I just tried restarting the kernel and run again after installing - !pip install sentencepiece - and it worked..

Roseline

Answer 4

If you have installed transformers and sentencepiece library and still face NoneType error, restart your colab runtime by pressing shortcut key CTRL+M .

(note the dot in shortcuts key)

or use runtime menu and rerun all imports.

Note: don't rerun the library installation cells (cells that contain pip install xxx)

Plant

Answer 5

Had the exact same problem, just restarting the Jupyter Notebook kernel after installing the sentencepiece library did the trick for me.

Pigpen

Answer 6

I was having this issue with LlamaTokenizer.from_pretarined(MODEL_NAME).

With every huggingface tokenizer, you need the sentencepiece library to be installed. Here is my code:

!pip install -qqq transformers==4.28.1 --progress-bar off
!pip install -qqq bitsandbytes==0.38.1 --progress-bar off
!pip install -qqq accelerate==0.18.0 --progress-bar off
!pip install -qqq sentencepiece==0.1.99 --progress-bar off

Once you install the libraries, make sure you import them in order:

import textwrap
import torch
import sentencepiece
from transformers import LlamaForCausalLM, LlamaTokenizer, GenerationConfig

And then make sure you restart your kernel, wherever you are writing code (Google-colab, VS Code,etc.). Otherwise, the installed libraries will not be recognized and you will get the weird NoneType error.

Delapaz