Deep Learning Fundamentals & Building Neural Networks
A detailed beginner-friendly English guide covering 9 sections from zero: from “What is AI?” to “How do I build my first model with PyTorch?” Every concept comes with plain explanations, real-life analogies, and step-by-step numeric examples — with no assumed prior background.
Introduction to Artificial Intelligence
Before we build a neural network, we need to understand where deep learning sits in the broader world of artificial intelligence — and why it is almost everywhere today.
AI vs ML vs Deep Learning
Three terms used constantly — and often confused. Picture them as nested circles:
Artificial Intelligence (AI)
The broadest umbrella: any technology that makes machines “smart.” Includes chess rules, expert systems, ML, and DL.
Machine Learning (ML)
The model learns from examples. Includes Decision Trees, SVM, Random Forest, and neural networks.
Deep Learning (DL)
ML using deep neural networks (3+ hidden layers). Strongest on images, text, and audio.
Spam filtering:
- Traditional AI: manual rules — “if the subject contains FREE or WIN → spam.”
- ML: Naive Bayes learns from thousands of spam/ham messages.
- DL: an LSTM or Transformer understands the full context — “this offer is free but from my trusted bank.”
All three are AI — but DL wins when data is abundant and patterns are complex.
History of Deep Learning — from idea to revolution
| Year | Event | Why it matters |
|---|---|---|
| 1943 | McCulloch & Pitts — first mathematical neuron model | Foundation: neuron = sum inputs + threshold |
| 1958 | Perceptron — Frank Rosenblatt | First trainable network — then criticized for its limits |
| 1969 | Minsky & Papert book | Proved Perceptron limits (XOR problem) → “AI winter” |
| 1986 | Backpropagation — Rumelhart, Hinton | Method to train multi-layer networks |
| 2006 | Hinton — Deep Belief Networks | Renewed interest in deep networks |
| 2012 | AlexNet — ImageNet | Revolution: CNN + GPU beat rivals by 10%+ |
| 2016 | AlphaGo | DL + Reinforcement Learning beats Go champion |
| 2017 | Transformer | Foundation of GPT and every modern LLM |
| 2022–2026 | ChatGPT, GPT-4, multimodal models | DL accessible to everyone |
ImageNet competition: 1.2 million images, 1000 categories. AlexNet achieved 15.3% error — second place: 26%. A huge gap. Companies realized: massive data + GPU + deep networks = the future of AI.
Major applications today
| Domain | Application | Technique |
|---|---|---|
| Translation & chat | Google Translate, ChatGPT, Claude | Transformer / LLM |
| Computer vision | Face ID, object detection, self-driving cars | CNN |
| Audio | Siri, Alexa, speech-to-text | RNN / Transformer |
| Medicine | Diagnosis from scans, drug discovery | CNN, GNN |
| Creative generation | DALL-E, Midjourney, Stable Diffusion | Diffusion / GAN |
| Recommendations | Netflix, YouTube, Amazon | Embeddings + Deep Models |
| Games | AlphaGo, AlphaStar | DL + Reinforcement Learning |
Why did Deep Learning succeed?
DL failed in the 1970s and 1980s — then surged after 2012. The reason: three factors came together:
1. Big Data
The internet, phones, cameras — millions of images, texts, and recordings. DL needs lots of data to learn.
2. Compute power (GPU)
NVIDIA GPUs run millions of operations in parallel — training on GPU is 10–100× faster than CPU.
3. Better algorithms
ReLU, Dropout, BatchNorm, Adam, ResNet — solved vanishing gradient and overfitting.
Learning to cook: you need many recipes (data), a well-equipped kitchen (GPU), and refined techniques (algorithms). Without any of the three — cooking (training) fails or takes years.
In traditional ML, an expert hand-picks features. In DL, the network discovers features on its own from raw data — edges → shapes → parts → whole object. This “Representation Learning” is the secret of DL’s power.
Essential Math — in the Simplest Form Possible
You do not need a math degree to start deep learning — but understanding the core ideas makes everything clearer. This section explains what you actually need, with examples tied to neural networks.
Linear Algebra
Vectors
A vector is an ordered list of numbers. Think of it as an arrow in space, or a row of data.
A student has 3 attributes: GPA=3.5, study hours=25, attendance=90%. We represent them as a vector:
In a neural network, each input sample = one vector. An MNIST image 28×28 = a vector of length 784 (one number per pixel).
Matrices
A matrix is a table of numbers (rows × columns). In DL, weights are stored as matrices.
A layer with 3 inputs and 2 neurons. Weights form a 3×2 matrix:
Row i = weights of neuron i from all inputs. Column j = weights of all neurons from input j.
Matrix Multiplication
This operation is the heart of forward propagation. When you multiply a weight matrix W by an input vector x, you get the layer outputs.
Input x = [2, 1], weight w = [0.5, 0.3], bias b = 0.1:
z = (0.5 × 2) + (0.3 × 1) + 0.1 = 1.0 + 0.3 + 0.1 = 1.4
This is exactly what one neuron does: a linear combination of weighted inputs.
Instead of computing each neuron separately, we multiply one matrix by an entire batch — GPUs do this in parallel. 32 images × 784 pixels = a 32×784 matrix multiplied by W — massive speed.
Eigenvalues — simplified idea
A slightly advanced concept, but useful for some techniques. When you multiply matrix A by vector v, the result is usually a new vector. But sometimes v stays in the same direction — only its length changes. That v is an eigenvector, and the scaling factor is the eigenvalue.
- PCA (Principal Component Analysis): dimensionality reduction — keep the most important directions.
- Understanding training: Hessian matrix and optimization stability analysis.
- Graph Neural Networks: analyzing network structure.
As a beginner: know they are “special directions” in data — you do not need to compute them by hand; libraries handle that.
Calculus Basics
Partial Derivatives
A derivative is the “rate of change.” If you change x slightly, how much does y change? A partial derivative is the same idea when a function has multiple variables — differentiate with respect to one variable while holding the others fixed.
Loss function depending on two weights: L(w₁, w₂) = (w₁ - 3)² + (w₂ - 5)²
∂L/∂w₁ = 2(w₁ - 3) — at w₁=3, the derivative = 0 (minimum point).
∂L/∂w₂ = 2(w₂ - 5) — at w₂=5, the derivative = 0.
During training: we compute ∂Loss/∂w for every weight — “in which direction should I adjust this weight to reduce error?”
The Gradient
The gradient is a vector collecting all partial derivatives. It points in the direction of steepest ascent. To descend (reduce Loss) we go opposite the gradient.
You are on a mountain and cannot see the bottom (minimum Loss). The gradient tells you: “descend in this direction.” You take a small step opposite the gradient — repeat until you reach the bottom. That is exactly Gradient Descent.
The Chain Rule
If y depends on z, and z depends on x — how does y change when x changes?
y = z², z = 3x. So y = (3x)² = 9x².
dy/dx = dy/dz × dz/dx = 2z × 3 = 2(3x) × 3 = 18x ✓
The network is a chain: Input → layer1 → layer2 → ... → Loss. Error flows backward through each layer by multiplying derivatives — Chain Rule. Without it, we cannot know how to update weights in early layers of a deep network.
Neural Networks
A neural network is a group of small “computers” (neurons) connected together. Each neuron takes inputs, computes, and passes the result to the next.
The Biological Neuron
In your brain, a neuron receives electrical signals through dendrites, sums them in the cell body, and if they exceed a threshold it fires a pulse through the axon to the next neurons.
Inputs ≈ Dendrites | Summation ≈ Cell Body | Threshold ≈ Bias | Fire/don't fire ≈ Activation | Output ≈ Axon | Weights ≈ strength of each synapse connection
The Artificial Neuron
A simplified mathematical version: takes numbers, multiplies each by a weight, sums them, adds bias, and passes through an activation function.
Perceptron — the simplest network
Perceptron (1958) = one neuron. It classifies points into two categories with a separating line.
Two inputs: GPA (0–4) and study hours (0–40). A Perceptron learns weights like w₁=0.6, w₂=0.3, b=-2.5.
Student GPA=3.5, study=25: 0.6×3.5 + 0.3×25 + (-2.5) = 2.1+7.5-2.5 = 7.1 → activation → “accepted.”
Perceptron limit: cannot solve XOR (data not linearly separable). Solution: multiple layers — MLP.
Multi-Layer Perceptron (MLP)
A network of layers: input → hidden layers → output. Each layer is fully connected (every neuron connects to every neuron in the next layer).
Input layer
Receives raw data — 784 neurons for an MNIST image. Does not “learn” — just passes data through.
Hidden layers
Where the “depth” lives. They extract features: edges → shapes → parts.
Output layer
The result: 10 neurons for 10 digits, 2 for cat/dog, or 1 for regression.
Weights, Bias, and Parameters
- Weights (w): importance of each input — “how much does this pixel affect the decision?” Learned from data.
- Bias (b): shifts the activation threshold — lets a neuron fire even without strong inputs. Like a “default tendency to approve.”
- Parameters: all weights + all biases. A 784→128→10 network ≈ 101,000 parameters.
Layer 784 → 128: weights = 784×128 = 100,352 + 128 bias = 100,480
Layer 128 → 10: 128×10 + 10 = 1,290
Total ≈ 101,770 numbers the model learns!
Forward Propagation
When you feed data into a trained network, it flows from input to output — that is forward propagation. Three steps per layer: linear combination → activation → pass to next layer.
Linear Combination
Each neuron computes: z = w·x + b — multiply each input by its weight, sum, add bias.
x = [2, -1, 3], w = [0.5, 0.8, -0.2], b = 0.1
z = (0.5×2) + (0.8×(-1)) + (-0.2×3) + 0.1 = 1.0 - 0.8 - 0.6 + 0.1 = -0.3
This z is before activation — the “raw” result.
For a full layer: do this for every neuron — or efficiently: z = W·x + b (matrix multiplication).
Activation
We pass z through a nonlinear activation function: a = activation(z). Without it, 100 layers collapse to one straight line — useless.
z = -0.3 → ReLU(-0.3) = max(0, -0.3) = 0
If z = 2.5 → ReLU(2.5) = 2.5
Prediction
After passing through all layers, the output layer gives the final prediction:
- Classification: Softmax converts numbers to probabilities — highest = prediction.
- Regression: one neuron outputs a number — house price, temperature.
- Binary: Sigmoid outputs a 0–1 probability — spam or not.
- Input: 28×28 image = 784 pixels (values 0–1 after normalization).
- Layer 1: 784 → 128 neurons. z₁ = W₁·x + b₁, a₁ = ReLU(z₁).
- Layer 2: 128 → 64. z₂ = W₂·a₁ + b₂, a₂ = ReLU(z₂).
- Output layer: 64 → 10. z₃ = W₃·a₂ + b₃.
- Softmax: [0.01, 0.02, 0.85, 0.03, 0.02, 0.01, 0.02, 0.02, 0.01, 0.01]
- Prediction: digit 2 with 85% probability.
All of this happens in fractions of a second on a GPU.
Activation Functions
Activation functions add nonlinearity — without them the network cannot learn complex patterns. Each function has strengths and weaknesses.
Sigmoid
Maps any number to a value between 0 and 1 — like a probability.
σ(0) = 0.5 | σ(2) ≈ 0.88 | σ(-2) ≈ 0.12
Use: binary classification (single output layer), gates in LSTM.
Drawback: vanishing gradient in deep networks — gradient shrinks exponentially.
Tanh
Outputs between -1 and +1 — centered around zero (sometimes better than Sigmoid).
tanh(0) = 0 | tanh(1) ≈ 0.76 | tanh(-1) ≈ -0.76
Use: RNNs, some older hidden layers.
ReLU (Rectified Linear Unit)
If negative → 0. If positive → unchanged. Most common in hidden layers.
ReLU(-3) = 0 | ReLU(0) = 0 | ReLU(5) = 5
Advantage: fast, gradient = 1 for positive values — fixes vanishing gradient.
Drawback: “Dead ReLU” — neurons stuck at 0 forever if gradient is always 0.
Leaky ReLU
Instead of zeroing negatives, pass a small fraction — the neuron does not “die.”
LeakyReLU(-10) = -0.1 | LeakyReLU(5) = 5
ELU (Exponential Linear Unit)
Smooth negative values (not linear like Leaky). Mean outputs closer to zero — sometimes faster training.
GELU (Gaussian Error Linear Unit)
Smooth function used in Transformers (BERT, GPT). Blends ReLU with probabilistic properties.
Transformers are sensitive to fine details in representations. GELU is smoother than ReLU — small input changes affect the output gradually, which helps with text.
Softmax
Converts n numbers into n probabilities that sum to 1 — for multi-class classification.
Raw outputs: [2.0, 1.0, 0.1]
Softmax: [0.659, 0.242, 0.099] → prediction: cat (65.9%)
Raw outputs: [0.5, 3.0, 0.2] → Softmax: [0.06, 0.90, 0.04] → dog (90%)
| Function | Range | Where used |
|---|---|---|
| ReLU / Leaky / ELU | 0 to ∞ (approx.) | Hidden layers in CNN and MLP |
| Sigmoid | 0 to 1 | Binary classification, LSTM gates |
| Tanh | -1 to 1 | RNN, hidden layers |
| GELU | approx. -0.17 to ∞ | Transformers (BERT, GPT) |
| Softmax | 0 to 1 (sum = 1) | Output layer — multi-class classification |
Loss Functions
A loss function measures “how wrong were we?” — a single number. Training = minimizing this number. The choice of loss depends on the task type.
MSE — Mean Squared Error
Average of the squared difference between prediction and truth. For regression (predicting a number).
Truth: $200,000. Predictions: [195,000, 210,000, 198,000]
Errors: [-5000, +10000, -2000] → squares: [25M, 100M, 4M]
MSE = (25+100+4)/3 = 43 million — large errors are penalized heavily (because they are squared).
MAE — Mean Absolute Error
Average of the absolute error — less sensitive to outliers than MSE.
MAE = (5000 + 10000 + 2000) / 3 = $5,667 — easier to interpret: “on average we are off by $5,667.”
Binary Cross Entropy — binary classification
For two classes (spam/ham, sick/healthy). y = 0 or 1, p = predicted probability (from Sigmoid).
Truth y=1 (spam). Prediction p=0.9 (confident and correct): BCE = -log(0.9) ≈ 0.105 — small error.
Truth y=1. Prediction p=0.1 (confident but wrong!): BCE = -log(0.1) ≈ 2.30 — heavy penalty.
Categorical Cross Entropy — multi-class classification
For 3+ classes (digits 0–9, animal types). y = one-hot [0,0,1,0,...], p = Softmax probabilities.
Truth: cat → y = [1, 0, 0]
Good prediction p = [0.8, 0.15, 0.05]: CCE = -log(0.8) ≈ 0.22
Bad prediction p = [0.1, 0.85, 0.05]: CCE = -log(0.1) ≈ 2.30
Hinge Loss — SVM and margin classification
Heavily penalizes wrong predictions, but if the prediction is correct with enough margin — Loss = 0.
y=+1 (positive class), f(x)=0.8: Hinge = max(0, 1-0.8) = 0.2
y=+1, f(x)=2.0: Hinge = max(0, 1-2) = 0 — confident enough, no penalty.
Use: SVM, some classification models. Less common than Cross-Entropy in deep networks.
KL Divergence — measuring “distance between distributions”
Measures how much distribution P (truth) differs from distribution Q (model prediction). KL = 0 means perfect match.
- VAE (Variational Autoencoders): approximating latent distribution.
- Knowledge Distillation: transferring knowledge from a large model to a small one.
- Language Models: comparing word distributions.
| Loss function | Task | Final layer output |
|---|---|---|
| MSE / MAE | Regression (continuous number) | Linear (no activation or ReLU) |
| Binary Cross Entropy | Two classes | Sigmoid |
| Categorical Cross Entropy | 3+ classes | Softmax |
| Hinge Loss | Margin classification | Linear |
| KL Divergence | Comparing distributions | Softmax / Logits |
Gradient Descent
After computing Loss, we need to update weights to reduce it. Gradient Descent is the core algorithm — we walk opposite the gradient direction.
Cost Function
Cost Function = Loss but over all training data (or a batch). Goal: find weights that minimize Cost.
Loss: error for one example or small batch. Cost: average Loss over a larger set. In practice they are used interchangeably — what matters is: a number we want to minimize.
Optimization
Optimization = finding the best parameter values. In DL, “best” = lowest Cost on validation data, not just training data.
Learning Rate
Step size for each update. The most important hyperparameter of all.
Too large (lr = 1.0)
Jumps over the minimum — Loss oscillates or explodes (NaN). Training fails.
Too small (lr = 0.00001)
Very slow — may take days. But stable.
Just right (lr = 0.001)
Good balance — common starting point with Adam.
Weight w = 2.0, gradient ∂L/∂w = 4.0 (increasing w increases Loss), lr = 0.1
w_new = 2.0 - 0.1 × 4.0 = 1.6 — we decreased w because gradient is positive.
If gradient = -3.0: w_new = 2.0 - 0.1 × (-3) = 2.3 — we increased w because gradient is negative.
Batch Gradient Descent
Uses all training data in one update step.
- Advantage: accurate, stable gradient.
- Drawback: slow — 60,000 MNIST images in one step = huge memory and slowness.
Stochastic Gradient Descent (SGD)
Uses one example per update step.
- Advantage: very fast, escapes local minima (noise helps).
- Drawback: noisy gradient — Loss fluctuates heavily.
Mini-Batch Gradient Descent
The golden middle: a small group (32, 64, 128 examples) per step. Most used in DL.
60,000 training images, batch size = 128:
Iterations per epoch = 60,000 / 128 ≈ 469 update steps.
15 epochs = 469 × 15 ≈ 7,035 total update steps.
| Type | Data per step | Speed | Stability |
|---|---|---|---|
| Batch GD | All data | Slow | Very stable |
| SGD | 1 example | Very fast | Fluctuating |
| Mini-Batch | 32–256 | Fast + GPU-friendly | Excellent balance ✓ |
Adam (most popular): combines Momentum + adaptive learning rate per weight. Start with lr=0.001. SGD + Momentum and AdamW (for Transformers) are also common choices.
Backpropagation
A network with 100,000 weights — how do we know which weight to adjust and by how much? Backpropagation (1986) is the answer: it computes the gradient for every weight efficiently using the Chain Rule.
Chain Rule in Backprop
The network is a chain of functions: x → layer1 → layer2 → ... → Loss. To adjust a weight in layer 1, we need: how much does this weight affect Loss? = multiply derivatives across the chain.
Computing Gradients
Forward:
- Input x = 2.0
- Hidden layer: z = w₁·x + b₁ = 0.5×2 + 0.1 = 1.1
- Activation: a = ReLU(1.1) = 1.1
- Output: y_pred = w₂·a + b₂ = 0.8×1.1 + 0.2 = 1.08
- Truth y_true = 1.5. Loss = (1.08 - 1.5)² = 0.176
Backward:
- ∂Loss/∂y_pred = 2(1.08 - 1.5) = -0.84
- ∂Loss/∂w₂ = ∂Loss/∂y_pred × a = -0.84 × 1.1 = -0.924
- ∂Loss/∂a = ∂Loss/∂y_pred × w₂ = -0.84 × 0.8 = -0.672
- ∂Loss/∂w₁ = ∂Loss/∂a × x = -0.672 × 2 = -1.344 (because ReLU is active)
Each weight now knows: “increase me” (negative gradient) or “decrease me” (positive gradient).
Weight Updates
w₂_new = 0.8 - 0.1 × (-0.924) = 0.8 + 0.092 = 0.892
w₁_new = 0.5 - 0.1 × (-1.344) = 0.5 + 0.134 = 0.634
Next time, the prediction will be closer to 1.5 — Loss smaller.
Automatic Differentiation
Computing gradients by hand for a large network is impossible. Autodiff in PyTorch and TensorFlow does it automatically:
loss.backward() # automatic Backprop — computes all gradients
optimizer.step() # update weights
optimizer.zero_grad() # clear gradients for next step
- Forward: data → prediction
- Loss: compare to truth
- Backward: loss.backward() — gradients for every weight
- Update: optimizer.step() — adjust weights
- Repeat for every batch and every epoch
Building Your First Model — TensorFlow, Keras, PyTorch
Now we connect everything we learned to real code. We will build a simple model to classify MNIST digits — the classic beginner project.
TensorFlow and Keras
TensorFlow = Google's deep learning library. Keras = high-level interface on top of TensorFlow — simple for beginners.
from tensorflow import keras
model = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(64, activation='relu'),
keras.layers.Dense(10, activation='softmax')
])
model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train / 255.0 # normalize to 0-1
model.fit(x_train, y_train, epochs=15, batch_size=128, validation_split=0.1)
test_loss, test_acc = model.evaluate(x_test, y_test)
print(f"Test accuracy: {test_acc:.2%}")
- Flatten: converts 28×28 to 784.
- Dense(128, relu): hidden layer with 128 neurons + ReLU.
- Dropout(0.2): randomly turns off 20% of neurons — fights overfitting.
- Dense(10, softmax): 10 classes (digits 0–9).
- compile: sets optimizer (Adam), Loss, and metric (accuracy).
- fit: training — 15 epochs, batch 128.
- evaluate: test on data the model has never seen.
PyTorch
PyTorch = Facebook/Meta library — most popular in research and education. More “Pythonic” and flexible than TensorFlow.
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
class MNISTNet(nn.Module):
def __init__(self):
super().__init__()
self.layers = nn.Sequential(
nn.Flatten(),
nn.Linear(784, 128),
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 10)
)
def forward(self, x):
return self.layers(x)
model = MNISTNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
transform = transforms.Compose([transforms.ToTensor()])
train_data = datasets.MNIST('./data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=128, shuffle=True)
for epoch in range(15):
for images, labels in train_loader:
optimizer.zero_grad()
outputs = model(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
| Framework | Strengths | When to use |
|---|---|---|
| TensorFlow + Keras | Easy, production-ready, TFLite for mobile | Production projects, Google Cloud, mobile |
| PyTorch | Flexible, Pythonic, research-friendly | Beginners, research, LLMs |
Training
Practical training steps:
- Load data: MNIST, CIFAR, or your own dataset.
- Preprocessing: normalize (0–1), resize, augmentation.
- Split: train (80%) / validation (10%) / test (10%).
- Build model: layers, activation, dropout.
- Train: watch train loss and validation loss.
- Early Stopping: stop if validation loss rises (overfitting).
- Train Loss ↓: model is learning — good.
- Val Loss ↓: generalizing — excellent.
- Train ↓ but Val ↑: overfitting — add Dropout or more data.
- Both not decreasing: underfitting — add layers or increase lr.
Evaluation
After training, evaluate on test data the model has never seen:
Accuracy
Percentage of correct classifications. MNIST: 97–99% is excellent for a simple MLP.
Precision / Recall
Important when classes are imbalanced — medicine, fraud detection.
Confusion Matrix
Table showing which classes get confused — reveals weak spots.
Test Loss
Should be close to validation loss — if much higher = overfitting.
- Day 1–2: install Python + PyTorch. Run MNIST from the official tutorial.
- Day 3: change layer count and neuron count — observe the effect.
- Day 4: try different learning rates (0.1, 0.01, 0.001).
- Day 5: add Dropout — compare overfitting.
- Day 6: plot the Loss curve (train vs val).
- Day 7: write a README on GitHub — document what you learned.
After MNIST: try CIFAR-10 (color images) with a CNN — you will learn Conv layers. Then explore Part 3 on Transformers and LLMs. The Math for AI and Linear Algebra guides deepen your understanding.