How to define special "untokenizable" words for nltk.word_tokenize

Question

I'm using nltk.word_tokenize for tokenizing some sentences which contain programming languages, frameworks, etc., which get incorrectly tokenized.

For example:

>>> tokenize.word_tokenize("I work with C#.")
['I', 'work', 'with', 'C', '#', '.']

Is there a way to enter a list of "exceptions" like this to the tokenizer? I already have compiled a list of all the things (languages, etc.) that I don't want to split.

Juju

possible duplicate of Modify python nltk.word_tokenize to exclude “#” as delimiter – Matron

The difference with that question is that OP is not asking to keep the # always, but instead just in C# and possibly in hundreds of other particular words such as F# and similar technical names. – Plait

Answer 1

The Multi Word Expression Tokenizer should be what you need.

You add the list of exceptions as tuples and pass it the already tokenized sentences:

tokenizer = nltk.tokenize.MWETokenizer()
tokenizer.add_mwe(('C', '#'))
tokenizer.add_mwe(('F', '#'))
tokenizer.tokenize(['I', 'work', 'with', 'C', '#', '.'])
['I', 'work', 'with', 'C_#', '.']
tokenizer.tokenize(['I', 'work', 'with', 'F', '#', '.'])
['I', 'work', 'with', 'F_#', '.']

Mardellmarden