Autoencoders

So far we have trained networks to classify or predict. But what if we want to compress information and then rebuild it? That is exactly what an Autoencoder does — and it is the first step toward understanding generation in AI.

Encoder

The Encoder takes input (an image, audio, text as vectors…) and turns it into a much smaller representation called a latent representation. Like taking a high-resolution photo and saving a compressed JPEG — the file is smaller, but the essence of the image remains.

Decoder

The Decoder takes the compressed vector and tries to reconstruct the original input. The training goal: make the difference between the original and the rebuilt version as small as possible (reconstruction loss).

Latent Space

Latent space is a “map” of hidden numbers. Each point represents a kind of data. Similar items (two 7s) sit close together; different ones (7 vs 3) are far apart. This space matters because later we generate new data by picking points from it.

Denoising Autoencoder

We feed the network a corrupted image (noise, blur, random pixels) and ask it to restore the clean image. It learns what matters in the data — not random junk — very useful for image and audio processing.

Variational Autoencoders (VAE)

A VAE pushes the autoencoder idea further: instead of one point in latent space, it learns a probability distribution. That lets you generate new data by sampling randomly from that distribution.

Latent Variables

Latent variables (z) are hidden numbers that capture the “essence” of the data. In a VAE, the Encoder does not output z directly — it outputs:

• μ (mu): the mean — where is the center of the distribution?
• σ (sigma): spread — how far values scatter around the center?

Then we sample: z = μ + σ × ε where ε is random from a normal distribution.

Sampling

During training we sample from the distribution (reparameterization trick) so gradients can flow. At generation time: pick random z from a standard N(0,1) distribution, pass it to the Decoder — out comes a new face, digit, or shape you have never seen exactly before.

KL Loss

A VAE has two losses:

1. Reconstruction loss: did we rebuild the input well?
2. KL divergence: is the Encoder’s learned distribution close to a simple “normal” prior?

KL loss stops the Encoder from stuffing everything into one point — it forces latent space to stay smooth and organized so you can wander it and generate diverse samples.

GANs — Generative Adversarial Networks

A GAN is like a game between two teams: one forges art, the other tries to spot fakes. Over time the forger gets very good — and the result is stunningly realistic images.

Generator

Takes random noise — a vector of random numbers — and turns it into an image (or sound, or any data). At first it outputs garbage; with training it learns convincing shapes.

Discriminator

A classifier network: “Is this image real or fake?” It sees real training images plus Generator outputs. If it catches fakes easily, the Generator must improve. If it gets fooled, the Discriminator must improve.

DCGAN — Deep Convolutional GAN

DCGAN uses convolution layers instead of fully connected ones — suited for images. Practical rules helped stability: BatchNorm, stride instead of pooling, ConvTranspose in the Generator.

CycleGAN — Unpaired domain translation

Turn a horse into a zebra without paired “this horse = this zebra” images. Cycle consistency: horse → zebra → horse should land close to the original. Uses: summer/winter, photo/painting, MRI modalities.

StyleGAN — Style control

StyleGAN (NVIDIA) generates high-quality faces with fine control: face shape, hair color, expression… via style mixing — blend styles from different latent vectors. Powers many famous AI face sites and academic deepfake research.

Diffusion Models

Diffusion models power Stable Diffusion, Midjourney, and DALL·E 3 (partly). The idea: gradually corrupt an image with noise, then teach a network to reverse the process — from noise back to image.

Forward Process

Start with a real image x₀. At each step t add a bit of Gaussian noise:

After T steps (e.g. 1000), the image is pure noise. This process is fixed — we do not learn it; we only apply it to training data.

Reverse Process

A U-Net (usually) learns: “Given a noisy image at step t, what noise was added?” — it predicts ε. Repeat from t=T down to t=0, remove noise step by step, and you get a new image.

Stable Diffusion

Instead of full pixels (slow), Stable Diffusion works in a compressed latent space (VAE encoder). Faster and lighter. It combines:

• Text encoder (CLIP): turns your prompt into a vector.
• U-Net + cross-attention: links text to the image during denoising.
• VAE decoder: turns latent back into the final image.

Multimodal AI

Multimodal means the model understands more than one data type together: image + text, audio + video, document + table… The real world is multimodal — and modern models follow.

Vision + Language

Common tasks:

• Image captioning: image → text description.
• VQA: question about an image → answer.
• Text-to-image: description → image (Stable Diffusion).
• Image search: search by image or by sentence.

CLIP — Contrastive Language-Image Pre-training

OpenAI CLIP trains on millions of (image, text) pairs. It learns that “cat” sits near cat images in a shared embedding space. Uses:

• Zero-shot classification (no training on fixed classes).
• Search engine: “dog playing ball” → finds similar images.
• Foundation for Stable Diffusion (text encoder).

BLIP — Bootstrapping Language-Image Pre-training

BLIP / BLIP-2 combines image understanding and text generation. It captions images, answers questions, and connects to an LLM (e.g. Flan-T5 or Llama) for stronger abilities. Good for VQA and image-aware chatbots.

Florence — Microsoft’s vision foundation model

Florence (and Florence-2) is a vision foundation model: captioning, detection, segmentation, OCR… from one text prompt. “Natural language commands for images” — like GPT but for vision tasks.

RLHF — Reinforcement Learning from Human Feedback

After internet pre-training, an LLM knows “text completion” — but it may be rude, unsafe, or unhelpful. RLHF teaches it what humans prefer: useful, safe, clear answers — like ChatGPT.

Reward Model

Collect human comparisons: “Response A is better than B” for the same question. Train a network to assign a score to each reply — higher when it matches human taste. That reward model becomes the automated coach.

PPO — Proximal Policy Optimization

PPO is an RL algorithm that updates the LLM carefully: generate replies, reward model scores them, adjust weights to increase reward — without breaking what was learned (KL penalty keeps it near the original model).

Preference Optimization

Instead of a heavy RL loop, newer methods optimize the model directly from preference data (A better than B) without long generation cycles.

DPO — Direct Preference Optimization

DPO (Stanford) turns RLHF into a simple loss like classification: raise probability of the preferred reply, lower the other — sometimes without a separate reward model. Faster and easier — used in Llama 2/3 and Mistral tuning.

Fine-Tuning LLMs

A general model (GPT, Llama, Mistral) knows a lot — but you want it expert in your domain: medicine, law, customer support, your tone… Fine-tuning = continue training on your data.

Full Fine-Tuning

Update every weight in the model. Strongest fit to your data, but:

• Needs huge GPUs (70B = tens of GB VRAM).
• Risk of catastrophic forgetting.
• High cost.

PEFT — Parameter-Efficient Fine-Tuning

PEFT updates only a small slice of parameters. The base model stays frozen; you add tiny layers or adapters. Cheaper and faster — right for most projects.

LoRA — Low-Rank Adaptation

LoRA adds two small matrices (A and B) beside an attention layer: W' = W + A×B. Small rank (r=8 or 16) — millions of parameters instead of billions. A LoRA file may be only 50–200 MB!

QLoRA — LoRA + Quantization

QLoRA quantizes the base model to 4-bit (NF4) — so Llama 7B fits in a 16 GB GPU. LoRA adapters stay fp16. This made LLM fine-tuning on Colab accessible to many people.

Model Deployment

After training, a model in Jupyter is not enough — you need to run it in production: fast, stable, on CPU, GPU, or edge. Each tool has a role.

ONNX — Open format

Open Neural Network Exchange: convert PyTorch/TensorFlow to a .onnx file for ONNX Runtime — cross-platform, sometimes faster, CPU/GPU/edge.

TensorRT — NVIDIA acceleration

NVIDIA optimizer for GPU inference: fusion, FP16/INT8 precision, low latency — ideal for production on NVIDIA servers.

TorchScript — PyTorch for production

Export PyTorch to TorchScript (trace or script) — runs with less Python overhead, or via LibTorch in C++.

GGUF — LLMs on CPU

llama.cpp format: quantized models (Q4_K_M, Q8…) — run Llama/Mistral on a laptop without GPU. Single file, easy to share.

llama.cpp

C++ engine for local LLMs. Command: ./main -m model.gguf -p "Hello". Powers Ollama, KoboldCpp, and others.

vLLM — fast LLM inference

Serving library with PagedAttention — high throughput for APIs: many parallel requests on GPU. For cloud production (chatbot API).

Ollama — easy local run

ollama run llama3 — downloads and runs the model. Great for developers on Mac/Windows/Linux.

TensorFlow Serving

Google’s system for TensorFlow models: REST/gRPC, versioning, A/B testing — for companies on TF stack.

Capstone Projects — From Idea to Application

Here we tie everything together into real projects you can start as a beginner — with a clear learning path for each.

Build Local ChatGPT

Idea: LLM + chat UI on your machine.

1. Install Ollama or LM Studio.
2. Download Llama 3 or Mistral (7B Q4).
3. UI: Open WebUI or a simple Streamlit app.
4. Optional: LoRA adapter for your domain.

Image Generator

Stable Diffusion via Diffusers or Automatic1111/ComfyUI. Start with simple prompts, try negative prompts, add ControlNet later for pose control.

RAG System — Retrieve then Generate

RAG = Retrieval-Augmented Generation. Instead of relying only on the LLM’s memory, it searches your documents (PDF, wiki, knowledge base) then answers.

1. Split documents → chunks.
2. Embed (OpenAI / sentence-transformers).
3. Vector DB (Chroma, FAISS, Pinecone).
4. On question: find relevant chunks → attach to prompt → LLM answers.

AI Agent

LLM + tools: search, calculator, APIs, file writes. It plans steps and executes — LangChain, LlamaIndex, or OpenAI Assistants API.

Vision Language Model — See and Speak

LLaVA, GPT-4V, Qwen-VL: upload an image + ask questions. Project idea: “kitchen assistant” — photo of fridge → suggested recipes.

Medical AI (with caution)

X-ray classification (CNN), tumor segmentation (U-Net), or LLM summaries (not a substitute for a doctor). Start with public sets: ChestX-ray, ISIC skin lesions — with ethics and privacy in mind.

Recommendation System

Collaborative filtering + embeddings, or two-tower models (user/item). Netflix/Spotify spirit: “users who watched X also watched Y.” You can add an LLM to describe content.

Time Series Forecasting

LSTM/Transformer for prices, demand, energy. Project: forecast monthly sales from CSV — or ready models: Prophet, Temporal Fusion Transformer.

FAQ on Generative AI