Skip to main content

📝 Named Entity Recognition (NER)

Description​

< What is it? >​

Named Entity Recognition (NER) finds spans of text that name real-world entities and assigns each one a category, such as a person, organization, location, date, or product. It is commonly treated as a sequence-labeling or token-classification task.

  • Example: in “Ava Chen joined Acme Labs in Berlin,” an NER system may extract:

    Text spanEntity type
    Ava Chenperson
    Acme Labsorganization
    Berlinlocation

NER is a task, not one specific model. Rules, conditional random fields, recurrent networks, and Transformers can all perform it.

Key points​

< Entity labels and the BIO scheme >​

NER datasets often label tokens with the BIO scheme:

TagMeaning
B-PERBeginning of a person entity
I-PERInside the same person entity
OOutside every labeled entity
  • Example: “Ava Chen joined Acme Labs” can be labeled as B-PER I-PER O B-ORG I-ORG. The token labels are then combined back into the spans “Ava Chen” and “Acme Labs.”

< How a modern NER model works >​

A Transformer encoder creates a contextual representation hi\mathbf{h}_i for each token ii. A small classification head predicts that token's entity label:

hi=Encoder⁡(x)i\mathbf{h}_i = \operatorname{Encoder}(\mathbf{x})_i p(yi∣x)=softmax⁡(Whi+b)p(y_i\mid\mathbf{x}) = \operatorname{softmax}(W\mathbf{h}_i+\mathbf{b})

The surrounding words matter. For example, “Jordan scored” is likely a person, while “Jordan borders Iraq” is likely a location.

< Example: extract entities with a pretrained model >​

This example uses Hugging Face's token-classification pipeline. The aggregation strategy merges BIO-tagged token pieces into complete entity spans:

Input text: "Apple opened an office in London."

# pip install transformers torch
from transformers import pipeline

ner = pipeline(
    task="token-classification",
    model="dslim/bert-base-NER",
    aggregation_strategy="simple",
)

text = "Apple opened an office in London."

for entity in ner(text):
    print(
        f"{entity['word']}: {entity['entity_group']} "
        f"(score={entity['score']:.2f}, "
        f"span={entity['start']}:{entity['end']})"
    )

An illustrative result is:

wordentity_groupscorestart:end
AppleORGapproximately 0.990:5
LondonLOCapproximately 0.9926:32

The score is the model's confidence for the predicted entity type. The character offsets identify where the entity appears in the original text; exact scores can vary by model and version.

Note: entity_group is not a list of words. It is a category label selected from the model's finite label set. The text span can be many different words—for example, Apple, Microsoft, or OpenAI—but each may receive the same ORG (organization) group. For this checkpoint, the merged entity groups are typically PER, ORG, LOC, and MISC; another NER model may define a different set, such as DATE or PRODUCT.

With aggregation_strategy="simple", the pipeline combines BIO token labels such as B-ORG and I-ORG into one complete span, then reports the shared type as entity_group: "ORG". Tokens labeled O mean “not an entity” and are usually omitted from the output.

< Training data and tokenization >​

Training examples need both the text and annotated entity spans. A tokenizer may split one word into several subword tokens, so labels must be aligned carefully; special tokens are ignored, and a common convention labels only the first subword of each original word.

< Evaluation and uses >​

NER is usually evaluated with entity-level precision, recall, and F1. A predicted entity normally counts as correct only when both its span and entity type match the reference. Token accuracy alone can hide boundary mistakes.

Common uses include search, document extraction, content moderation, medical or legal document processing, and redacting personally identifiable information.

Comparison​

< NER vs text classification >​

Named Entity RecognitionText classification
Prediction unitTokens or text spansA whole document, sentence, or input
OutputB-PER, I-PER, B-ORG, O, and similar labelsOne or more document-level labels
ExampleFind “Ava Chen” as a personClassify a review as positive or negative
Typical challengeCorrect entity boundaries and token-label alignmentChoosing the correct class for the complete input
  • Classification introduces supervised classification more generally.
  • Embeddings explains token representations.
  • Transformer explains the encoder architecture commonly used for modern NER.
  • Softmax explains how the model converts label scores into probabilities.

Reference​