Large Language Models

What Are Large Language Models?

Large Language Models (LLMs) are neural networks trained on massive text corpora to predict the probability of the next token. Despite this simple objective, scale unlocks emergent capabilities: reasoning, code generation, in-context learning, and instruction following.

Scale = data + compute + parameters. Scaling laws (Chinchilla, 2022) show that performance improves predictably with these three axes.

Architecture: The Decoder-Only Transformer

Modern LLMs (GPT, Llama, Mistral, Gemini) are decoder-only Transformers — stacked causal self-attention blocks:

Input tokens → Token Embedding + Positional Encoding
→ [Causal Self-Attention → FFN] × L layers
→ Language Model Head (Linear + Softmax over vocabulary)
→ Next-token probabilities

Causal masking: each token can attend only to prior tokens. This enables autoregressive generation and efficient training on entire sequences simultaneously.

Key Components

RMSNorm (modern variant of LayerNorm, no mean subtraction): RMSNorm(x) = x / RMS(x) · γ, RMS(x) = √(mean(x²) + ε)

Rotary Positional Embedding (RoPE): encode position by rotating query/key vectors: q_m = R(mθ) · q where R(mθ) is a rotation matrix depending on position m

RoPE enables extrapolation to longer contexts and is used in Llama, Mistral, Qwen, DeepSeek.

SwiGLU Feed-Forward: FFN(x) = (SiLU(W₁x) ⊙ W₃x) · W₂

Used in Llama. The gating mechanism adds expressiveness compared to the original ReLU FFN.

Grouped Query Attention (GQA): multiple query heads share one key/value head — reduces KV cache size at inference: - Multi-Head: Q heads = K heads = V heads = h - Multi-Query (MQA): 1 K/V head, h Q heads - Grouped Query (GQA): g K/V heads, h Q heads (h/g Q per K/V group)

Pre-Training

LLMs are pre-trained on next-token prediction (autoregressive language modeling):

L(θ) = −Σₜ log P(wₜ | w₁, …, wₜ₋₁; θ)

Data: web crawl (Common Crawl, FineWeb), books (Books3, Gutenberg), code (GitHub, The Stack), Wikipedia, academic papers.

Scale: modern LLMs train on 1–15 trillion tokens. - Chinchilla scaling law: optimal tokens ≈ 20× parameters - Llama 3 70B: 70B parameters, 15T tokens - GPT-4: ~1.8T parameters (estimated MoE)

Infrastructure: distributed training across thousands of GPUs using: - Data parallelism: same model on multiple GPUs, each sees different data - Tensor parallelism: split matrix operations across GPUs - Pipeline parallelism: different layers on different GPUs - ZeRO (DeepSpeed): shard optimizer states, gradients, parameters

Post-Training: Alignment Pipeline

Raw pre-trained models generate text but don't follow instructions. Alignment makes them helpful and safe:

1. Supervised Fine-Tuning (SFT)

Fine-tune on high-quality demonstration data (prompt → response pairs):

from transformers import AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
trainer = SFTTrainer(
    model=model,
    train_dataset=sft_dataset,       # dataset with a "text" column
    args=SFTConfig(                  # TRL >= 0.13: SFTConfig, not TrainingArguments
        output_dir="./sft_output",
        max_length=2048,             # was max_seq_length in older TRL
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,
        learning_rate=2e-5,
        num_train_epochs=1,
    )
)
trainer.train()

2. RLHF (Reinforcement Learning from Human Feedback)

Reward modeling: train a model to predict which of two responses humans prefer.

PPO fine-tuning: use the reward model as a signal to RL-optimize the LLM policy.

KL-penalized objective: reward(x,y) = r_θ(x,y) − β · KL(π_θ(y|x) ‖ π_ref(y|x))

3. DPO (Direct Preference Optimization)

Eliminates the separate reward model. Directly optimize on preference pairs (y_w preferred over y_l):

L_DPO = −E log σ(β log(π_θ(y_w|x)/π_ref(y_w|x)) − β log(π_θ(y_l|x)/π_ref(y_l|x)))

Simpler, more stable, and often as good as PPO. Most modern open models use DPO or a variant.

from trl import DPOTrainer, DPOConfig

dpo_config = DPOConfig(beta=0.1, learning_rate=5e-7, num_train_epochs=1)
trainer = DPOTrainer(
    model=model, ref_model=ref_model,
    args=dpo_config, train_dataset=preference_dataset
)
trainer.train()

Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning of 70B models requires hundreds of GPUs. PEFT methods adapt models with far fewer trainable parameters.

LoRA (Low-Rank Adaptation)

Freeze original weights W; add a low-rank decomposition:

W' = W + ΔW = W + B·A

where B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, rank r ≪ min(d,k).

Only A and B are trained. This reduces trainable parameters by ~10,000× for large models.

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    r=16,               # rank
    lora_alpha=32,      # scaling: ΔW weight = alpha/r
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.062

QLoRA: quantize the base model to 4-bit (NF4), train LoRA adapters in 16-bit. Fine-tune 70B models on a single A100 80GB GPU.

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)

Other PEFT Methods

Inference and Decoding

KV Cache

At inference, attention requires Q, K, V for all prior tokens. The KV cache stores computed key and value tensors, avoiding recomputation:

• Prefill phase: process the entire prompt; fill KV cache
• Decode phase: generate one token at a time; append to KV cache

KV cache size = 2 · L · n_heads · d_head · seq_len · dtype_bytes

For Llama-3 70B (L=80, h=8 GQA K/V heads, d_head=128, seq_len=8192, BF16): 2·80·8·128·8192·2 B ≈ 2.7 GB per sequence.

Speculative Decoding

Use a small "draft" model to propose multiple tokens; the large model verifies them in parallel. Achieves 2–3× speedup with identical output distribution.

Quantization for Inference

Reduce memory and speed up computation by using lower-precision weights:

# GPTQ quantized inference
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-7B-GPTQ",
    device_map="auto",
    torch_dtype=torch.float16
)

Prompting Techniques

Zero-Shot

Ask the model directly with no examples.

Classify the sentiment: "I love this product!" → Positive

Few-Shot

Provide k examples in the prompt context.

Q: 2+2 = ? A: 4
Q: 3+5 = ? A: 8
Q: 7+4 = ? A:

Chain-of-Thought (CoT)

Elicit step-by-step reasoning before the final answer:

"Let's think step by step..."

Dramatically improves performance on math, logic, and multi-step reasoning.

Self-Consistency

Sample multiple CoT reasoning paths; take the majority answer. More reliable than greedy single-path.

ReAct (Reasoning + Acting)

Interleave reasoning steps and tool calls:

Thought: I need to find the population of France.
Action: search("France population 2026")
Observation: 68 million
Thought: Now I can answer.
Final Answer: France has approximately 68 million people.

Evaluation of LLMs

Landmark Models Timeline

The reasoning-model shift (2024–2025): o1 showed that RL on chain-of-thought plus test-time compute beats scale alone on math/code; DeepSeek-R1 reproduced it with open weights, and every frontier line now ships a "thinking" mode with a controllable reasoning budget.