Natural Language Processing

What is NLP?

Natural Language Processing (NLP) is the subfield of ML concerned with enabling computers to understand, generate, and reason about human language. Language is discrete, compositional, context-dependent, and ambiguous — properties that make it fundamentally different from image or tabular data.

Text Preprocessing Pipeline

Raw text → Cleaning → Tokenization → Normalization → Numericalization → Model

Cleaning

• Lowercase (or not — case carries meaning in NER: "Apple" vs. "apple")
• Remove HTML, URLs, special characters (depends on task — keep punctuation for sentiment)
• Normalize whitespace and encoding (UTF-8 issues)

Tokenization

The process of splitting text into tokens (units the model processes):

Subword tokenization (used by BERT, GPT, Llama) is the standard: rare words are split into known subword pieces; common words remain whole. Never unseen tokens.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer("Hello, world!", return_tensors="pt")
# {'input_ids': tensor([[101, 7592, 1010, 2088, 999, 102]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])}

# Byte Pair Encoding (BPE) from scratch
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=30000, special_tokens=["[UNK]","[PAD]"])
tokenizer.train(files=["corpus.txt"], trainer=trainer)

Normalization

• Stemming: reduce to stem — "running" → "run" (crude, rule-based; Porter stemmer)
• Lemmatization: reduce to dictionary form — "ran" → "run" (needs POS tag; spaCy, NLTK)
• For transformer models, generally skip stemming/lemmatization — the model learns it

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The children ran to the stores")
tokens = [(t.text, t.lemma_, t.pos_) for t in doc]

Text Representations

Bag of Words (BoW)

Represent a document as a vector of word counts. Ignores order.

"the cat sat" → {"the": 1, "cat": 1, "sat": 1}

TF-IDF (Term Frequency-Inverse Document Frequency)

Weights word importance: common words (the, is) get downweighted; rare, distinctive words get upweighted.

TF(t,d) = count(t in d) / |d| IDF(t) = log(N / df(t) + 1) TF-IDF(t,d) = TF × IDF

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(max_features=10000, ngram_range=(1, 2),
                         min_df=2, max_df=0.95)
X = tfidf.fit_transform(corpus)   # sparse matrix (n_docs, vocab_size)

Word Embeddings

Map each word to a dense vector where semantic similarity ≈ geometric closeness:

Word2Vec (Mikolov et al., 2013): - CBOW: predict center word from context - Skip-gram: predict context words from center word - Properties: king − man + woman ≈ queen (linear analogies)

GloVe: factorize the word co-occurrence matrix.

FastText: character n-gram embeddings — handles morphology and out-of-vocabulary words.

from gensim.models import Word2Vec

model = Word2Vec(sentences, vector_size=100, window=5,
                 min_count=5, workers=4, sg=1)  # sg=1: Skip-gram
vector = model.wv["king"]
similar = model.wv.most_similar("king", topn=10)
analogy = model.wv.most_similar(positive=["woman","king"], negative=["man"])

Contextual Embeddings (BERT-era)

Unlike Word2Vec (one vector per word), BERT produces context-dependent embeddings — "bank" in "river bank" and "money bank" have different vectors.

Core NLP Tasks

Text Classification

Assign a label to an entire document (sentiment, topic, spam).

With sklearn/traditional:

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer

clf = Pipeline([("tfidf", TfidfVectorizer()), ("lr", LogisticRegression())])
clf.fit(train_texts, train_labels)

With Transformers:

from transformers import pipeline
classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
result = classifier("I love this movie!")
# [{'label': 'POSITIVE', 'score': 0.9998}]

Named Entity Recognition (NER)

Tag spans of text as entities (PERSON, ORG, LOC, DATE, etc.):

"Apple is buying a startup in London for $1B" ORG LOC MONEY

BIO tagging: B-ORG, I-ORG, O, B-LOC, O, O, B-MONEY — sequence labeling problem.

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is buying a startup in London for $1B")
for ent in doc.ents:
    print(ent.text, ent.label_)
# Apple ORG, London GPE, $1B MONEY

Relation Extraction

Given entities, classify the semantic relationship between them: - (Apple, London) → LOCATED_IN? No. - (CEO, Company) → WORKS_AT

Text Generation

Generate text token by token via autoregressive decoding:

P(w₁, w₂, …, wₙ) = Πᵢ P(wᵢ | w₁, …, wᵢ₋₁)

Decoding strategies:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model     = AutoModelForCausalLM.from_pretrained(model_name)

inputs  = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50,
                          do_sample=True, temperature=0.8, top_p=0.9)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)

Machine Translation

Seq2seq with Transformer encoder-decoder. Modern systems (DeepL, Google Translate) fine-tune large multilingual models (mBART, M2M-100, NLLB-200).

BLEU score: measures n-gram overlap between prediction and reference translations.

Question Answering

Extractive QA: find a span in a given context document. Generative QA: generate an answer (no constraints on source span). Open-domain QA: retrieve relevant documents then extract/generate.

from transformers import pipeline
qa = pipeline("question-answering", model="deepset/roberta-base-squad2")
result = qa(question="Who wrote Hamlet?",
             context="Hamlet is a tragedy written by William Shakespeare.")
# {'answer': 'William Shakespeare', 'score': 0.998, 'start': 40, 'end': 58}

Summarization

Extractive: select important sentences from the document. Abstractive: generate new text that paraphrases (may not appear verbatim in source).

Metric: ROUGE (Recall-Oriented Understudy for Gisting Evaluation): - ROUGE-1: unigram overlap - ROUGE-2: bigram overlap - ROUGE-L: longest common subsequence

Semantic Similarity / Sentence Embeddings

Embed sentences into a common space; use cosine similarity to measure how similar they are semantically.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embs  = model.encode(["Paris is in France", "France contains Paris", "Python is fast"])
from sklearn.metrics.pairwise import cosine_similarity
sim = cosine_similarity(embs)

Pre-trained Transformer Models for NLP

BERT (Bidirectional Encoder Representations from Transformers)

• Architecture: Transformer encoder
• Pre-training: Masked Language Modeling (MLM) + Next Sentence Prediction
• Best for: classification, NER, QA, embedding (NOT generation)

GPT Family (Generative Pre-trained Transformer)

• Architecture: Transformer decoder (causal)
• Pre-training: next-token prediction on web text
• Best for: text generation, chat, code completion

T5 / BART

• Architecture: full encoder-decoder
• Pre-training: denoising / text-to-text
• Best for: translation, summarization, classification framed as text-to-text

Domain-Specific Models

Fine-Tuning BERT for Classification

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer
from datasets import Dataset

model_name = "bert-base-uncased"
tokenizer  = AutoTokenizer.from_pretrained(model_name)
model      = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, max_length=512, padding="max_length")

dataset = Dataset.from_dict({"text": texts, "label": labels})
dataset = dataset.map(tokenize, batched=True)
dataset = dataset.train_test_split(test_size=0.2)

args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True
)

trainer = Trainer(model=model, args=args,
                   train_dataset=dataset["train"],
                   eval_dataset=dataset["test"])
trainer.train()

Text Feature Engineering (Classical)

When not using transformers, rich hand-crafted features still matter:

• Character n-gram TF-IDF (captures morphology, misspellings)
• POS tag distributions
• Parse tree depth, syntactic features
• Named entity counts
• Sentiment lexicon scores (VADER, SentiWordNet)
• Readability indices (Flesch-Kincaid)
• Word count, sentence count, avg word length

NLP Evaluation Metrics

Perplexity for language models:

PP(W) = exp(−(1/N) Σ log P(wᵢ|w₁,…,wᵢ₋₁))

Lower perplexity = model assigns higher probability to the test text = better language model.