If you completed Lesson 1, you already know numbers, functions, and equations. This lesson adds the geometric and structural tools that make large-scale AI computation possible on a GPU.

The loss at the end of training is a scalar — one number telling you how wrong the model is. The learning rate \(\eta\) is a scalar controlling step size. Even when everything else is huge matrices, you often care about a few key scalars: accuracy, loss, learning rate.

Each training example is a feature vector. A 28×28 grayscale image flattened becomes a vector of 784 numbers. Word embeddings (Word2Vec, BERT) map each word to a dense vector where similar words sit close together.

A fully connected layer with 784 inputs and 128 neurons stores its weights in a \(128 \times 784\) matrix. A batch of 32 images is stored as a \(32 \times 784\) matrix — 32 rows, one per image. GPUs are fast precisely because they multiply huge matrices in parallel.

Residual connections in ResNet add the input of a layer back to its output: \(\mathbf{y} = F(\mathbf{x}) + \mathbf{x}\). Dropout scales surviving activations by \(\frac{1}{1-p}\) after randomly zeroing some entries. Batch normalization subtracts a mean vector and divides by a standard-deviation vector — all element-wise vector operations on batches stored as matrices.

Almost every layer in a transformer, CNN (after im2col), or MLP is implemented as GEMM — General Matrix Multiply. Libraries like cuBLAS and PyTorch's torch.matmul exist solely to make this operation fast. Understanding shapes prevents the #1 beginner bug: dimension mismatch errors.

In attention mechanisms, an identity-style mask (diagonal allowed, rest blocked) prevents a token from "seeing" future tokens. Weight initialization sometimes starts near identity (e.g., orthogonal init) to preserve signal magnitude at the start of training. Skip connections effectively add an identity path: output = layer(x) + x.

Computing \(A^{-1}\) explicitly is rare in deep learning (too expensive for huge matrices). But the idea of inversion appears in solving linear systems, normal equations in linear regression \(\mathbf{w} = (X^TX)^{-1}X^T\mathbf{y}\), and whitening transforms in preprocessing. Optimization prefers iterative methods over direct inversion.

A zero determinant means redundant features — the matrix is singular. In PCA, eigenvalues relate to determinants of submatrices. Jacobian determinants appear in normalizing flows (generative models) to track how density changes under a transformation.

Low-rank approximation compresses huge weight matrices (LoRA fine-tuning adds low-rank updates \(W + BA\)). Rank collapse in transformers can make attention maps degenerate. Understanding rank helps you see when embeddings or activations are truly high-dimensional vs. lying on a smaller subspace.

PCA finds eigenvalues of the covariance matrix — largest eigenvalues = directions of maximum variance in data. Spectral analysis of graph Laplacians powers graph neural networks. Stability of recurrent networks can be studied via eigenvalues of the weight matrix (values > 1 can cause exploding gradients).

PCA principal components are eigenvectors of the covariance matrix. In PageRank and GNNs, the dominant eigenvector of a matrix reveals the most important node scores. Eigendecomposition turns covariance structure into interpretable axes for visualization (t-SNE builds on similar ideas).

Every neuron computes a dot product between its weight vector and the input vector (plus bias). Attention scores in transformers are scaled dot products \(\frac{QK^T}{\sqrt{d_k}}\). Cosine similarity between embeddings is the dot product of normalized vectors — used in search, RAG, and recommendation systems.

Cross products are less central than dot products in standard deep learning, but they appear in 3D vision (surface normals, camera geometry), robotics, and physics-informed neural networks. The 2D scalar cross product (determinant of two 2D vectors) is used in computational geometry for orientation tests.

CNN filters apply small linear transformations locally across an image. Data augmentation uses rotation/scaling matrices on images. In transformers, positional encodings add structured vectors so the model knows token order. Understanding transforms helps you debug why augmented data looks "wrong" or why features collapse.

You do not need to derive every theorem by hand. You do need to recognize when a paper says \(W \in \mathbb{R}^{d \times k}\) what shape of object that is, why \(AB \neq BA\) matters for composition order, and why attention is a softmax over dot products. That fluency — reading linear algebra as the operating manual of AI — is what this lesson builds.