Nice question.
First, we should clarify that spaCy uses the BILUO annotation scheme instead of the BIO annotation scheme you are referring to.
From the spacy documentation the letters denote the following:
- B: The first token of a multi-token entity.
- I: An inner token of a multi-token entity.
- L: The final token of a multi-token entity.
- U: A single-token entity.
- O: A non-entity token.
Then, some definitions:
![definitions]()
Spacy has a built-in class to evaluate NER. It's called scorer.
Scorer uses exact matching to evaluate NER. The precision score is returned as ents_p, the recall as ents_r and the F1 score as ents_f.
The only problem with that is that it returns the score for all the tags together in the document. However, we can call the function only with the TAG we want and get the desired result.
All together, the code should look like this:
import spacy
from spacy.gold import GoldParse
from spacy.scorer import Scorer
def evaluate(nlp, examples, ent='PERSON'):
scorer = Scorer()
for input_, annot in examples:
text_entities = []
for entity in annot.get('entities'):
if ent in entity:
text_entities.append(entity)
doc_gold_text = nlp.make_doc(input_)
gold = GoldParse(doc_gold_text, entities=text_entities)
pred_value = nlp(input_)
scorer.score(pred_value, gold)
return scorer.scores
examples = [
("Trump says he's answered Mueller's Russia inquiry questions \u2013 live",{"entities":[[0,5,"PERSON"],[25,32,"PERSON"],[35,41,"GPE"]]}),
("Alexander Zverev reaches ATP Finals semis then reminds Lendl who is boss",{"entities":[[0,16,"PERSON"],[55,60,"PERSON"]]}),
("Britain's worst landlord to take nine years to pay off string of fines",{"entities":[[0,7,"GPE"]]}),
("Tom Watson: people's vote more likely given weakness of May's position",{"entities":[[0,10,"PERSON"],[56,59,"PERSON"]]}),
]
nlp = spacy.load('en_core_web_sm')
results = evaluate(nlp, examples)
print(results)
Call the evaluate function with the proper ent parameter to get the results for each tag.
Hope it helps :)
