Information Theory for Artificial Intelligence
Cross-entropy loss, Softmax, KL in VAEs, and perplexity in LLMs all come from one idea: how surprised should you be by an event? This intermediate lesson builds entropy, divergences, and mutual information into the language of modern neural networks.
Why Information Theory Matters in AI
Information theory, founded by Claude Shannon (1948), quantifies uncertainty and information content. In AI, almost every probabilistic objective is an information-theoretic quantity in disguise: classification uses cross-entropy, generative models minimize KL terms, transformers are scored with perplexity, and representation learning often maximizes mutual information.
You should know discrete probability, logarithms, expectations, and the idea of a loss from Lesson 3. Linear algebra from Lesson 2 helps when Softmax acts on logit vectors.
Where you already use it
- Training classifiers —
CrossEntropyLossis \(H(p,q)\) with one-hot \(p\) - Language models — next-token NLL and perplexity are entropy rates
- VAEs / diffusion — KL terms keep latents near a prior
- Distillation & RLHF — teacher–student or policy–reference KL
| Concept (this lesson) | Typical AI artifact |
|---|---|
| Self-information | Per-token / per-class \(-\log q\) |
| Entropy \(H(p)\) | Label uncertainty; irreducible CE floor |
| Cross-entropy | Classification & LM training loss |
| KL divergence | VAE ELBO, distillation, RLHF penalty |
| Mutual information | Contrastive SSL, IB-style reps |
| Perplexity | LLM leaderboard / eval metric |
Self-Information (Surprise)
The self-information (or surprisal) of an outcome \(x\) with probability \(p(x)\) answers: how surprising was this event?
\[I(x) = -\log p(x)\]
Rare events (small \(p\)) → large \(I\). Certain events (\(p=1\)) → \(I=0\).
Why the negative log?
- Independent events multiply in probability → their informations add (logs turn products into sums)
- Monotone: less probable ⇒ more information
- Matches coding length: optimal codeword length for \(x\) is about \(-\log_2 p(x)\) bits
Fair coin: \(p(H)=0.5\), \(I(H)=1\) bit. Biased coin \(p(H)=0.9\): \(I(H)\approx 0.15\) bits — heads is almost expected. A tokenizer emitting a rare token with \(p=2^{-10}\) contributes 10 bits of surprise to the sequence score.
Negative log-likelihood (NLL) of a dataset is the sum of self-informations under the model. Training a classifier or LM is literally minimizing total surprise on the training data.
Entropy
Entropy is the expected self-information — the average surprise of drawing from distribution \(p\).
\[H(p) = H(X) = -\sum_x p(x)\,\log p(x) = \mathbb{E}_{x\sim p}\bigl[-\log p(x)\bigr]\]
Continuous analogue (differential entropy): \(h(X)=-\int p(x)\log p(x)\,dx\).
Key properties
- \(H(X)\ge 0\) for discrete \(X\); \(H=0\) iff deterministic
- Maximum entropy for a fixed alphabet: uniform distribution
- For binary: \(H(p)=-p\log p-(1-p)\log(1-p)\) (binary entropy), max at \(p=0.5\)
Three-class label prior \(p=(0.7,0.2,0.1)\): \(H(p)\approx 1.16\) bits (base 2). Uniform \(p=(1/3,1/3,1/3)\): \(H=\log_2 3\approx 1.58\) bits — harder to predict a priori.
Label entropy sets a lower bound on cross-entropy loss: even a perfect model cannot beat \(H(p)\) on average. Class imbalance (low entropy) makes accuracy look easy while minority classes remain hard.
Entropy of Softmax outputs
During training, the entropy of \(q_\theta(y|x)\) often starts near \(\log K\) (almost uniform) and falls as the model becomes confident. Monitoring prediction entropy is a cheap diagnostic for overconfidence and calibration.
Joint & Conditional Entropy
For two random variables, we ask how much uncertainty remains in one after observing the other.
Joint entropy: \(H(X,Y)=-\sum_{x,y}p(x,y)\log p(x,y)\)
Conditional entropy:
\[H(Y\mid X)=\sum_x p(x)\,H(Y\mid X=x)=-\sum_{x,y}p(x,y)\log p(y\mid x)\]
Chain rule: \(H(X,Y)=H(X)+H(Y\mid X)=H(Y)+H(X\mid Y)\)
A supervised model aims to drive \(H(Y\mid X)\) down — after seeing features \(X\), labels \(Y\) should be predictable. Conditional entropy of next-token given context is exactly what language models minimize.
Cross-Entropy
Cross-entropy \(H(p,q)\) is the expected surprise when events come from \(p\) but you score them with \(q\).
\[H(p,q) = -\sum_x p(x)\,\log q(x) = \mathbb{E}_{x\sim p}\bigl[-\log q(x)\bigr]\]
Fundamental identity:
\[H(p,q) = H(p) + D_{\mathrm{KL}}(p\,\|\,q)\]
Why it dominates ML
When \(p\) is a one-hot label \(y\), cross-entropy collapses to \(-\log q(y)\) — the familiar Softmax loss. Minimizing \(H(p,q)\) over model parameters is equivalent to minimizing KL when \(p\) is fixed (because \(H(p)\) does not depend on \(\theta\)).
| Quantity | Depends on | Role in training |
|---|---|---|
| \(H(p)\) | data only | Constant — ignore for optimization |
| \(H(p,q_\theta)\) | data + model | Standard loss to minimize |
| \(D_{\mathrm{KL}}(p\|q_\theta)\) | data + model | Same optimum as CE |
True class = cat. Model Softmax: \(q(\text{cat})=0.6\), \(q(\text{dog})=0.3\), \(q(\text{bird})=0.1\). CE loss \(=-\log 0.6\approx 0.511\) nats. If \(q(\text{cat})=0.95\), loss \(\approx 0.051\) — much less surprise.
nn.CrossEntropyLoss in PyTorch combines LogSoftmax + NLL. Label smoothing replaces hard \(p\) with a softer target, which changes \(H(p)\) and can improve calibration.
KL Divergence
The Kullback–Leibler divergence measures how one distribution diverges from another — the expected extra surprise from using \(q\) when the truth is \(p\).
\[D_{\mathrm{KL}}(p\,\|\,q) = \sum_x p(x)\log\dfrac{p(x)}{q(x)} = H(p,q)-H(p)\]
Properties: \(D_{\mathrm{KL}}\ge 0\), equals 0 iff \(p=q\) almost everywhere; not symmetric and does not satisfy the triangle inequality (not a metric).
Forward vs reverse KL (intuition)
- \(D_{\mathrm{KL}}(p\|q)\) — zero-forcing / mass-covering when fitting \(q\) to \(p\)
- \(D_{\mathrm{KL}}(q\|p)\) — mode-seeking (typical in mean-field VI)
VAE training maximizes the ELBO, which includes \(-D_{\mathrm{KL}}(q_\phi(z|x)\|p(z))\). Knowledge distillation uses KL between teacher and student Softmax. RLHF / preference models often use KL penalties against a reference policy.
Let \(p=(0.5,0.5)\) and \(q=(0.9,0.1)\). Then \(D_{\mathrm{KL}}(p\|q)=0.5\log(0.5/0.9)+0.5\log(0.5/0.1)\approx 0.511\) nats, while \(D_{\mathrm{KL}}(q\|p)=0.9\log(0.9/0.5)+0.1\log(0.1/0.5)\approx 0.369\) nats — same pair, different numbers.
For \(q=\mathcal{N}(\mu,\sigma^2)\) and \(p=\mathcal{N}(0,1)\):
\[D_{\mathrm{KL}}(q\|p)=\tfrac{1}{2}\bigl(\mu^2+\sigma^2-1-\log\sigma^2\bigr)\]
This closed form is why diagonal-Gaussian VAEs are popular — the KL term is cheap and exact.
Mutual Information
Mutual information \(I(X;Y)\) measures how much knowing \(X\) reduces uncertainty about \(Y\) (and vice versa).
\[ \begin{aligned} I(X;Y) &= H(X)-H(X\mid Y) = H(Y)-H(Y\mid X)\\ &= H(X)+H(Y)-H(X,Y)\\ &= D_{\mathrm{KL}}\bigl(p(x,y)\,\|\,p(x)p(y)\bigr) \end{aligned} \]
Contrastive learning (InfoNCE) lower-bounds mutual information between views. Feature selection and IB theories phrase "good representations" as high \(I(Z;Y)\) with constrained \(I(Z;X)\).
Softmax & the Maximum-Entropy Principle
Given logits \(z\in\mathbb{R}^K\), Softmax produces a distribution over classes. It is not arbitrary: it is the maximum-entropy distribution consistent with those scores as expected energy constraints.
\[q_i = \dfrac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}\qquad i=1,\ldots,K\]
Equivalently, \(q = \arg\max_{r\in\Delta} \bigl\{ H(r) + \langle z, r\rangle \bigr\}\) over the probability simplex.
Temperature
With temperature \(T>0\): \(q_i \propto \exp(z_i/T)\). As \(T\to 0\), \(q\) approaches one-hot (argmax). As \(T\to\infty\), \(q\) approaches uniform. Distillation and sampling both exploit \(T\).
Logits \(z=(2.0, 1.0, 0.1)\). Softmax ≈ \((0.659, 0.242, 0.099)\). With \(T=2\): softer ≈ \((0.506, 0.307, 0.187)\).
Attention weights are Softmax over query–key scores. Gumbel-Softmax / concrete distributions extend this idea for discrete latent variables. Numerically, subtract \(\max z_i\) before exp for stability.
| Component | Role of Softmax |
|---|---|
| Classifier head | Map logits → class probabilities |
| Transformer attention | Map scores → mixing weights over keys |
| LM head | Map vocab logits → next-token distribution |
| Mixture / gating | Soft assignment over experts or components |
Perplexity for LLMs
Perplexity turns average NLL into an interpretable "effective branching factor" — how many equally likely choices the model feels it has at each step.
For a sequence \(w_1,\ldots,w_N\) under model \(q\):
\[\mathrm{PPL} = \exp\Bigl(\tfrac{1}{N}\sum_{t=1}^{N}-\log q(w_t\mid w_{<t})\Bigr) = e^{H(p,q)}\]
(Using natural log → PPL in "nats-as-exp"; with \(\log_2\), report \(2^{H}\) for bit-based perplexity.)
Compare PPL only under the same tokenizer and evaluation set. Bits-per-byte (BPB) is preferred when vocabularies differ. PPL correlates with likelihood but not always with human preference — hence RLHF and win-rate metrics.
Bits vs Nats
The base of the logarithm sets the unit of information.
| Base | Unit | Common in |
|---|---|---|
| \(\log_2\) | bit (shannon) | Coding theory, compression |
| \(\ln = \log_e\) | nat | ML libraries, continuous math |
| \(\log_{10}\) | hartley / dit | Rare in AI |
\[1\ \mathrm{nat} = \log_2 e \approx 1.4427\ \mathrm{bits}\qquad 1\ \mathrm{bit} = \ln 2 \approx 0.6931\ \mathrm{nats}\]
PyTorch CE loss reports nats. A loss of \(0.693\) nats ≈ \(1\) bit ≈ random binary guess. Multiply NLL by \(\log_2 e\) to convert to bits before computing bit-based perplexity \(2^{\mathrm{NLL}_{\mathrm{bits}}}\).
Papers sometimes write \(\log\) without a base — in information theory that usually means \(\log_2\); in ML it usually means \(\ln\). Always check the surrounding equations.
Deriving Cross-Entropy Loss from MLE
Assume i.i.d. samples \((x^{(i)}, y^{(i)})\) and a categorical model \(q_\theta(y|x)=\mathrm{Softmax}(f_\theta(x))\). Maximum likelihood maximization is equivalent to minimizing cross-entropy.
Likelihood: \(\mathcal{L}(\theta)=\prod_i q_\theta(y^{(i)}|x^{(i)})\)
Log-likelihood: \(\ell(\theta)=\sum_i \log q_\theta(y^{(i)}|x^{(i)})\)
Minimize NLL / empirical CE:
\[J(\theta)= -\dfrac{1}{m}\sum_{i=1}^{m}\log q_\theta(y^{(i)}|x^{(i)}) = \widehat{H}(p_{\mathrm{data}}, q_\theta)\]
For one-hot \(y\): \(J=-\log q_{y}\). Softmax+CE gradient w.r.t. logits is simply \(q - y\) (derived fully in Lesson 6).
Binary case (sigmoid)
For two classes with \(q=\sigma(z)=1/(1+e^{-z})\) and label \(y\in\{0,1\}\), CE becomes binary cross-entropy:
\[L = -\bigl(y\log q + (1-y)\log(1-q)\bigr)\]
This is still \(H(p,q)\) with \(p=\mathrm{Bernoulli}(y)\). Multiclass Softmax CE is the natural generalization.
Batch of 2: labels \([1,0]\), model probs \([0.8, 0.3]\). Mean CE \(= \tfrac{1}{2}\bigl(-\log 0.8 - \log(1-0.3)\bigr)\approx 0.290\) nats. Gradient on the probability vector pulls mass toward the true class.
This is why "training with CE" and "doing MLE under a Softmax model" are the same statement. Changing the output family (Gaussian, Bernoulli) changes the matching NLL — MSE for Gaussian mean estimation is also an MLE story.
Information Bottleneck Intuition
The information bottleneck (IB) formalizes compression of \(X\) into representation \(Z\) that remains predictive of \(Y\).
\[\min_{p(z|x)} \; I(X;Z) - \beta\, I(Z;Y)\]
Small \(I(X;Z)\): discard irrelevant detail. Large \(I(Z;Y)\): keep what predicts the target. \(\beta\) trades off compression vs prediction.
Deep learning reading
- Early layers may retain more \(I(X;Z)\); later layers become more task-specific
- Dropout, bottlenecks, and sparse codes act as soft information constraints
- IB is a lens, not always a literal training loss — estimating MI is hard in high dimensions
Variational IB replaces MI terms with tractable bounds (similar spirit to VAEs). In transformers, the residual stream carries high-capacity information; attention and MLPs selectively route task-relevant bits.
Applications: Classification, LMs, and VAEs
| Setting | Information quantity | What you optimize / report |
|---|---|---|
| Classification | Cross-entropy \(H(p,q)\) | CE / NLL loss |
| Language models | Token NLL → perplexity | PPL, BPB |
| VAEs | ELBO = recon − KL | Reconstruction + \(D_{\mathrm{KL}}(q\|p)\) |
| Distillation | KL(teacher ‖ student) | Soft targets at temperature \(T\) |
| Contrastive SSL | MI lower bound (InfoNCE) | Contrastive loss |
| RLHF | KL to reference policy | Reward − \(\beta\) KL |
\(\log p(x) \ge \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{\mathrm{KL}}(q(z|x)\|p(z))\). The KL term is an information penalty keeping the approximate posterior near the prior.
Reading a training log through information theory
A decoder-only LM is a huge conditional distribution \(q(w_t|w_{<t})\). Causal masking enforces the chain rule of probability — exactly the expansion behind sequence cross-entropy and perplexity.
Summary
Information theory gives a precise language for uncertainty and prediction error in AI systems.