Training Neural Networks — How Does the Machine Learn?
Part 2 dives into the heart of deep learning: how weights update, what backpropagation is, how to fight overfitting, and how CNNs handle images and RNNs handle sequences — with step-by-step examples.
The training loop
Training a neural network means repeating the same cycle thousands of times until the model improves. One epoch = one full pass over all training data.
- Forward pass: feed a batch → compute prediction.
- Compute loss: compare prediction to truth → error number.
- Backward pass (backprop): compute how each weight contributed to error.
- Update weights: adjust weights slightly to reduce loss.
- Repeat for the next batch… next epoch… until convergence.
Batch: small group (32, 64, 128 examples) processed together. Epoch: one full sweep of training data. Iteration: one update step = one batch. 60,000 images, batch=128 → ~469 iterations/epoch.
Loss functions
Loss measures “how wrong were we?” — a single number to minimize.
| Loss | Task | Idea |
|---|---|---|
| MSE | Regression | Mean squared difference between prediction and truth |
| Cross-entropy | Classification | Heavily penalizes confident wrong predictions |
| Binary cross-entropy | Yes/no | Special case for 2 classes |
True label “cat” [1,0,0]. Prediction [0.7, 0.2, 0.1] → loss ≈ 0.36.
Wrong but confident [0.01, 0.98, 0.01] → loss ≈ 4.6 — harsh penalty.
Gradient descent
Imagine fog on a mountain — you want the lowest point (minimum loss). Gradient points steepest uphill. To go down: move opposite the gradient.
Learning rate: step size. Too large → overshoot. Too small → painfully slow. Common values: 0.001, 0.0001.
Batch GD
Uses all data each step — accurate but slow.
SGD
One example per step — fast, noisy.
Mini-batch
Small groups — most common in practice.
Backpropagation
With millions of weights — which to change, and by how much? Backpropagation (1986) answers via the chain rule. Error starts at the output and flows backward; each layer computes its contribution and passes gradients to the layer before.
Forward: x → layer1 → layer2 → y_pred=0.8. Truth=1.0. Loss=0.04.
Backward: output layer reports error → hidden layers adjust w5, w6, w1… → each weight gets a gradient: increase or decrease slightly.
In very deep old nets, gradients shrink (vanish) or blow up (explode). Fixes: ReLU, batch normalization, residual connections (ResNet), gradient clipping.
Optimizers — smarter than plain SGD
| Optimizer | Idea | When |
|---|---|---|
| SGD | Basic gradient × lr | Baseline |
| SGD + momentum | Remembers previous direction | Faster in valleys |
| Adam | Momentum + adaptive lr per weight | Default for most projects |
| AdamW | Adam + proper weight decay | Transformers, LLM fine-tuning |
Hyperparameters — what you choose
- Learning rate: 0.1, 0.01, 0.001 — often the most important.
- Batch size: 32–256 — larger = faster but more memory.
- Epochs: use early stopping instead of guessing.
- Layers/neurons: start small, grow gradually.
Fighting overfitting
Overfitting: model memorizes training data — 99% train, 60% test. Goal: generalization.
Dropout
Randomly “turn off” 20–50% of neurons during training.
L2 regularization
Penalize large weights → simpler model.
Batch normalization
Normalize layer outputs — faster, stabler training.
Early stopping
Stop when validation loss rises.
Data augmentation
Flip, crop, noise — free extra data for images.
More data
Best regularizer when possible.
Convolutional neural networks (CNN)
A 224×224 image = 50,176 pixels. Fully connected 50K→1000 = 50 million weights — impractical. CNNs exploit spatial structure: nearby pixels correlate.
- Convolution: small filters (3×3) slide over the image — detect edges, textures.
- Pooling: shrink spatial size — fewer parameters.
- Fully connected: final classification layers.
128×128 RGB → Conv 32 filters → ReLU → MaxPool → Conv 64 → Pool → Flatten → Dense 128 → Dropout 0.5 → Dense 2 → Softmax.
Train 20 epochs on 10,000 images → ~92% test accuracy. Inference: [0.08, 0.92] → “dog” 92%.
Famous architectures: LeNet, AlexNet, VGG, ResNet, EfficientNet — use pre-trained versions via transfer learning (Part 3).
Recurrent neural networks (RNN)
MLPs and CNNs assume fixed-size input. Text and audio are sequences — order matters. RNNs pass a hidden state from one time step to the next. LSTM and GRU handle long sequences better.
“The movie is amazing!” — word by word, hidden state accumulates positive signal → final layer → Positive 94%.
Full project walkthrough — MNIST
- Data: 60,000 grayscale 28×28 digits. Split 50K train / 10K test.
- Preprocess: normalize pixels to 0–1.
- Architecture: Flatten 784 → Dense 128 ReLU → Dropout 0.2 → Dense 64 → Dense 10 Softmax.
- Loss: cross-entropy. Optimizer: Adam, lr=0.001.
- Train: batch=128, epochs=15, 10% validation.
- Result: ~97–98% test accuracy. Simple CNN → 99%+.
In Part 3: Transformers, LLMs, GANs, transfer learning, deployment, ethics, and your learning roadmap.