Lesson 2: Improving Deep Neural Network Performance
After learning how to build and train a neural network, the next question is: how do I make it better? This guide covers 9 practical sections — from weight initialization to regularization, optimizers, and learning rate scheduling — explained in plain language with real-life analogies, numerical examples, and PyTorch code.
Initialization — Setting Starting Weights
When you create a new neural network, every weight and every bias needs a starting value. The question is: where do these values come from?
Imagine starting a race from the starting line. If you begin too close to the finish line (large weights), you'll overshoot and get lost. If you start too far away (zero or identical weights), you won't move — or everyone moves in the same direction. Good initialization = a smart starting point.
Why not start with zeros?
If you set all weights = 0, every neuron in the same layer computes the same value, receives the same gradient, and updates the same way — they stay identical forever (the symmetry problem). That's why we need randomness.
Random Initialization
The simple idea: pick small random numbers for each weight.
Simple network: input layer (3 features) → hidden layer (2 neurons).
- Weight matrix W is 3×2 — fill it with random values like: [[0.005, -0.008], [0.003, 0.012], [-0.007, 0.001]]
- Each neuron starts differently → they learn different things ✓
The problem: In deep networks (10+ layers), very small random values cause vanishing gradients; very large ones cause exploding gradients. That's where Xavier and He come in.
Xavier Initialization (Glorot)
Proposed by Xavier Glorot (2010): choose variance so that signal variance stays constant across layers — it neither grows nor shrinks.
where n_in = number of layer inputs, n_out = number of layer outputs.
Layer: 100 inputs → 50 outputs. Bound = √(6/(100+50)) = √(6/150) ≈ 0.2
Each weight is drawn randomly from [-0.2, +0.2]. Best for Sigmoid and Tanh.
He Initialization
Proposed by Kaiming He (2015) for ReLU activations: ReLU zeros out negative values, so variance is roughly halved. We need larger variance:
Layer: 256 inputs. Standard deviation = √(2/256) = √(0.0078) ≈ 0.088
Random weights around 0 with std ~0.088 — larger than Xavier because ReLU "kills" half the signals.
| Method | When to use | Activation |
|---|---|---|
| Small random | Shallow networks (1–2 layers) only | Any |
| Xavier / Glorot | Medium-depth networks | Sigmoid, Tanh |
| He / Kaiming | Default choice today | ReLU, LeakyReLU, GELU |
Regularization
A model that's too powerful memorizes training data word for word (like a student who memorizes answers instead of understanding the material) — this is called overfitting. Regularization = techniques that make the model "simpler" and generalize better to new data.
A student memorizes 1,000 questions verbatim → aces the practice exam but fails a new exam with different wording. Regularization means learning general rules instead of memorizing.
L1 Regularization
We add the sum of absolute values of weights to the loss function:
λ (lambda) = regularization strength. The larger λ, the smaller the weights become.
Result: Many weights become exactly zero → the model selects only important features (automatic feature selection).
Weights before L1: [0.8, -0.3, 0.05, 0.02, -0.7]. After training with strong L1: [0.6, -0.2, 0, 0, -0.5] — small weights disappear.
L2 Regularization
We add the sum of squared weights:
It doesn't make weights exactly zero, but keeps them small — preventing a single weight from dominating the decision.
One weight = 10. Its L2 contribution = 10² = 100 — a big penalty! The model is forced to shrink it to, say, 0.5 (contribution = 0.25 only).
Weight Decay
In modern optimizers, L2 is often applied as weight decay directly in the update step:
Each step, weights "shrink" slightly toward zero. In PyTorch:
Important note: With standard Adam, weight_decay ≠ L2 exactly. That's why AdamW (covered in Section 3) correctly decouples them.
Dropout
During training, we randomly disable a fraction of neurons (e.g., 50%) at each step. Each neuron has probability p of being turned off.
A soccer team: in every practice match, half the players have "fake injuries" — the rest learn to play without them. Result: the team doesn't depend on one star player.
Early Stopping
Monitor validation loss during training. If it starts rising (while train loss keeps falling) = overfitting. Stop early and save the best model.
- Epoch 10: val_loss = 0.35 ← best so far ✓
- Epoch 15: val_loss = 0.38 ← starting to rise
- Epoch 20: val_loss = 0.42 ← rising more
- Decision: stop at Epoch 10 and save those weights (patience = 5 epochs)
L1
Weights disappear (sparse). Useful for feature selection.
L2 / Weight Decay
Small weights. Most common in practice.
Dropout
Randomly disables neurons. Strong for large layers.
Early Stopping
Stop when val_loss worsens. Free and effective!
Optimization Algorithms
After computing the gradient, how do we update the weights? Plain SGD is the foundation — but advanced optimizers make training faster and more stable.
You're on a mountain peak (high loss) and want to reach the valley (low loss). Gradient = steepest descent direction. The optimizer = how you walk: small fixed steps? with momentum? adaptive steps based on terrain?
SGD — Stochastic Gradient Descent
At each step, compute the gradient on a small batch (e.g., 32 examples) instead of all data:
Weight w = 2.0, gradient = 0.5, learning rate = 0.1
w_new = 2.0 - 0.1 × 0.5 = 1.95
Problem: The path zigzags, especially in narrow valleys. Solution: Momentum.
Momentum
Add "momentum" — like a rolling ball: it accelerates in consistent directions and slows on oscillations:
w_new = w_old - lr · v
Without Momentum: gradients oscillate [+0.5, -0.4, +0.6, -0.5] → slow progress.
With Momentum (β=0.9): opposing vectors cancel, consistent ones accumulate → faster, smoother path to the solution.
RMSProp
Adapts the learning rate per weight based on gradient history. Weights with large gradients → smaller steps. Small gradients → larger steps:
w_new = w_old - lr / √(s + ε) · ∇Loss
Excellent for deep networks and RNNs — solves the problem of gradients with very different magnitudes.
Adam — Adaptive Moment Estimation
Combines Momentum (gradient average) + RMSProp (gradient² average). The default choice for most projects:
v = β₂·v + (1-β₂)·(∇Loss)² (adaptation)
w_new = w_old - lr · m̂ / (√v̂ + ε)
β₁=0.9, β₂=0.999, ε=1e-8 — default values that work well in most cases.
AdamW
Fixes Adam's issue with weight decay: it separates weight updates from weight shrinkage (decoupled weight decay). Best for regularization with Adam — the standard in Transformers and LLMs.
Nadam — Nesterov + Adam
Combines Adam with Nesterov Momentum — "look ahead" before jumping: computes the gradient at the point expected after a momentum step. Sometimes converges faster than Adam early in training.
| Optimizer | When? | Note |
|---|---|---|
| SGD + Momentum | Large CNNs, when you want max accuracy | Needs careful lr tuning |
| Adam | Default starting point, most projects | Fast and works out of the box |
| AdamW | Transformers, NLP, with weight decay | Best with regularization |
| RMSProp | RNNs, sequential data | Less common than Adam today |
| Nadam | Experiments — Adam alternative | Not default in most frameworks |
Hyperparameter Tuning
Parameters = weights (learned by the model). Hyperparameters = settings you choose before training. Wrong choices = slow or failed model.
Learning Rate
The most important hyperparameter. It controls the step size at each update.
On MNIST with Adam:
- lr = 0.1 → Loss jumps: 2.3, 0.8, 5.1, 1.2... (unstable)
- lr = 0.001 → Loss: 2.3 → 0.5 → 0.15 → 0.08 (excellent)
- lr = 0.00001 → Loss: 2.3 → 2.1 → 2.0 → 1.95 (too slow)
Batch Size
How many examples do we process before updating weights?
Small (16–32)
More updates = more variety. Slower but sometimes generalizes better. Works on smaller GPUs.
Medium (64–128)
Good balance — most common choice.
Large (256–1024)
Faster on GPU. Less noisy gradients. May need larger lr.
Rule: When you double batch size, try doubling learning rate (linear scaling rule).
Epochs
How many times do we pass through all training data?
- Few (5–10): Possible underfitting
- Many (100+): Possible overfitting — use Early Stopping
- Best: Train for many epochs with Early Stopping (e.g., max 100 epochs, patience 10)
Number of Layers
Each layer = a higher level of abstraction. Simple MNIST: 2–3 layers suffice. ImageNet: dozens of layers (ResNet-50 = 50 layers).
- 1 hidden layer (128): ~95% accuracy
- 2 layers (256 → 128): ~97% accuracy
- 5 layers without regularization: train 99% but val 94% (overfitting!)
Hidden Units
Width of each hidden layer. More = higher capacity + overfitting risk + computational cost.
Start small (64, 128) and increase gradually. Watch: if train accuracy is high but val is low → reduce neurons or add Dropout.
- Start with known-good settings (Adam, lr=1e-3, batch=64)
- Tune learning rate first (most important)
- Then batch size and architecture
- Finally regularization (dropout, weight decay)
- Change one parameter per experiment — otherwise you won't know what helped!
Batch Normalization
A breakthrough technique (2015) applied after the linear layer and before the activation function. It normalizes the output of each layer.
Internal Covariate Shift — what does it mean?
During training, weights in the first layer change → its outputs change → the second layer sees a "different distribution" at every step. Like trying to hit a target that keeps moving.
A chef learning a recipe: every day ingredient quality changes (previous layer) — hard to learn. Batch Norm = standardizing ingredient quality before each chef.
How does it work?
For each mini-batch, for each neuron:
x̂ = (x - μ) / √(σ² + ε)
y = γ · x̂ + β (γ, β are learnable)
γ and β let the network "undo" normalization if needed — flexibility is preserved.
Benefits
Faster training
Allows larger learning rates — converges faster.
Stability
Reduces sensitivity to initialization — He matters less with BN.
Mild regularization
Slightly reduces overfitting (not a Dropout replacement).
Deeper networks
Enables training more layers without gradient collapse.
Training vs Inference — the critical difference
| Mode | μ and σ | Behavior |
|---|---|---|
| Training | From current batch | Computed fresh each step |
| Inference (test/production) | Running mean/variance | Fixed — doesn't depend on batch |
In PyTorch: model.train() vs model.eval() control this automatically.
Learning Rate Scheduling
Instead of a fixed lr throughout training, we change it over time. The idea: large steps early (far from solution) → small steps at the end (higher precision).
Searching for a house in a city: at first you walk quickly between neighborhoods. As you get closer, you slow down between streets. At the end, tiny steps in front of the door.
Step Decay
Every N epochs, divide lr by e.g. 10:
initial_lr = 0.01, step_size = 30
- Epoch 0–29: lr = 0.01
- Epoch 30–59: lr = 0.001
- Epoch 60+: lr = 0.0001
Cosine Annealing
lr decreases smoothly along a cosine curve from maximum to zero:
Warmup
In the first K steps, lr starts near zero and gradually rises to the target value. Critical for training large Transformers and LLMs — prevents gradient explosion at the start.
target_lr = 0.001, warmup_steps = 1000
- Step 0: lr = 0
- Step 500: lr = 0.0005
- Step 1000: lr = 0.001 (then Cosine decay, for example)
Cyclic LR
lr oscillates between a minimum and maximum periodically. The idea: sometimes a large lr helps "jump" out of local minima. Less common than Cosine but useful in some experiments.
Debugging Deep Networks
Every deep learning practitioner faces these problems. The key: watch the curves (train vs validation loss/accuracy) and know what they mean.
Underfitting
The model is too simple — it doesn't even learn from training data.
- Train loss stays high and doesn't drop much
- Train accuracy is low (e.g., 60% on MNIST)
- Val loss is close to train loss (both bad)
Fixes: Add layers/neurons, reduce regularization, increase epochs, raise learning rate slightly, try a stronger architecture.
Overfitting
The model memorized training data — it fails on new data.
- Train loss is very low (near 0)
- Val loss rises while train keeps falling
- Train accuracy 99% but val accuracy 85%
Fixes: Dropout, Weight Decay, Early Stopping, more data, Data Augmentation, smaller model.
Vanishing Gradient
In deep networks, the gradient is multiplied across many layers → becomes near zero → early layers don't learn.
Causes: Sigmoid/Tanh (small derivatives), bad initialization, network too deep.
Fixes: ReLU, He Initialization, Batch Norm, Residual Connections (skip connections).
Exploding Gradient
Gradient grows exponentially → weights become NaN or Inf.
Loss = 2.3 → 1.5 → 0.8 → NaN. Or weights = [1e15, -3e20, ...]
Fixes: Gradient Clipping (clip gradient if it exceeds a threshold), reduce lr, better initialization.
Dead ReLU
If ReLU receives a large negative input, gradient = 0 → the neuron never learns again (dead).
Fixes: Leaky ReLU (small gradient for negatives: 0.01x), He Initialization, Batch Norm, smaller lr.
- Is data normalized (0–1 or standardized)?
- Are loss.backward() and optimizer.step() being called?
- Is model.train() / model.eval() set correctly?
- Are input shapes correct?
- Plot train vs val loss — what do they say?
Experiment Tracking
When you try 20 different setups, how do you remember which was best? Tracking tools log everything automatically: hyperparameters, loss, accuracy, weights, and charts.
TensorBoard
Built into TensorFlow — also works with PyTorch. Displays loss and accuracy curves, weight histograms, images, and network graphs.
Pros: Free, local, easy to start. Cons: Hard to share across a team.
Weights & Biases (W&B)
Popular cloud platform in research and industry. Auto-logging, experiment comparison, tables, and beautiful dashboards.
Pros: Easy sharing, experiment comparison, artifacts. Cons: Requires an account (free for personal use).
MLflow
Open source — tracks ML experiments broadly (not just deep learning). Logs parameters, metrics, models, and runtime environments.
Pros: Open source, self-hosted, supports model deployment. Cons: Less polished UI than W&B.
| Tool | Start with it if... |
|---|---|
| TensorBoard | Learning alone, want something free and local |
| W&B | Working in a team, want easy experiment comparison |
| MLflow | Company wants self-hosted full MLOps |
Practical Labs
Theory without practice fades away. Here are three hands-on experiments you can run on MNIST or CIFAR-10 in a few hours.
Experiment 1: Improve a Real Model — Baseline → Improved
Start with a weak baseline model, then apply improvements one by one:
| Step | Change | Expected result (MNIST) |
|---|---|---|
| Baseline | One layer 64, SGD lr=0.01, no regularization | ~92% accuracy |
| + He Init | Kaiming initialization | ~93% |
| + Adam | optimizer = Adam(lr=1e-3) | ~96% |
| + Second layer | 128 → 64 | ~97% |
| + Dropout 0.3 | After each ReLU | ~97.5% (better val) |
| + Batch Norm | After each Linear | ~98% |
| + LR Schedule | Cosine Annealing | ~98.2% |
Experiment 2: Compare Optimizers
Same model, same data — change only the optimizer:
- Fix: architecture, batch_size=64, epochs=20, seed=42
- Try: SGD (lr=0.01), SGD+Momentum (lr=0.01), Adam (lr=1e-3), AdamW (lr=1e-3, wd=1e-4), RMSprop (lr=1e-3)
- Record: training time, lowest val loss, highest val accuracy
- Plot loss curves for each optimizer on the same chart
Expected result: Adam/AdamW converge faster. SGD+Momentum may reach similar accuracy but needs lr tuning. Plain SGD is slowest.
Experiment 3: Compare Regularization
Use a deliberately large model (3 layers × 512 neurons) to force overfitting, then try:
No regularization
Train 99.5%, Val 94% — clear overfitting
Dropout 0.5
Train 98%, Val 97% — big generalization gain
Weight Decay 1e-3
Train 98.5%, Val 96.5% — moderate improvement
Dropout + WD + Early Stop
Best balance — Val ~97.5%
For each experiment, log in a table or W&B:
- Name: exp_03_adam_dropout05
- Change: Added Dropout=0.5 after the second layer
- Result: val_acc improved from 94% → 97%
- Conclusion: Dropout is effective for this model size
- Day 1: Baseline on MNIST + TensorBoard. Log the curves.
- Day 2: Optimizer experiment (5 optimizers). Compare in a table.
- Day 3: Regularization experiment. Write a short report of what you learned.