Sequence Models
Language Processing & Time Series
A beginner-friendly guide to how machines understand text, audio, and time-ordered data: from sequential data, through RNN, LSTM, and GRU, to Attention, Transformers, large language models (BERT, GPT, Llama), and everyday apps — with plain-language analogies.
Sequential Data
One simple idea first: some data has order. Change the order, and the meaning changes. That is sequential data.
Imagine five photos: open door → see gift → open gift → smile → thank mom. Shuffle them and the story breaks. Text, audio, and stock prices work the same way: order is part of the meaning.
Text
“The cat ate the mouse” is not “The mouse ate the cat.” Same words, different order, different meaning. Models must track what came before and after.
“I did not like the movie” ≠ “I liked the movie.” The word “not” flips the meaning of what follows.
Audio
Sound is a wave over time. The computer cuts it into tiny consecutive frames. The word “hello” is a sequence of sounds, not a single point.
Time Series
Numbers measured through time: gold price each day, temperature each hour. The goal is often: predict tomorrow from the past.
DNA
DNA is a sequence of letters A, T, C, G. Order decides what a gene does — same “order matters” idea, in biology.
| Type | Unit | Example question |
|---|---|---|
| Text | char / word / sentence | Is this review positive? |
| Audio | frame / phoneme | What did the speaker say? |
| Time series | value at time t | What is tomorrow’s price? |
| DNA | base (A/T/C/G) | Is this region important? |
Recurrent Neural Networks
A normal feedforward net sees inputs once and stops. Text is long: word after word. We need a net that reads step by step and remembers. That is the RNN idea.
RNN — the core idea
At each time step t, the network takes the current input and a summary of past steps (hidden state), then produces a new summary for the next step — like reading a book and updating margin notes.
Hidden State
The hidden state is a vector of numbers meaning “what I have understood so far.”
You do not remember every word. You keep a compressed summary and update it. The hidden state is that notebook inside the network.
Sequence Learning
Many → one
Full sentence → sentiment class.
One → many
Image → multi-word caption.
Many → many
Translate sentence word by word.
Next-step prediction
“Once upon a…” → “time”.
LSTM — Long Short-Term Memory
Vanilla RNNs forget. Name a person early, ask 50 words later “who did that?” — the name may be gone. LSTM adds gates that control what to keep and what to drop.
A warehouse (cell) gets new deliveries. The manager decides: what old stock to throw away, what new stock to store, what to show customers today. Those decisions are the gates.
Gates
Forget Gate
What to erase from memory.
Input Gate
What new info deserves to enter.
Output Gate
What memory to expose as the output now.
Cell State
Think of a smooth conveyor belt carrying memory across steps. Gates gently add or remove information so important facts last longer.
Long-Term Memory
Names, tense, and links from sentence start to end stay available — which made LSTM dominant for years in translation, sentiment, and forecasting.
“Ahmed went to the market, bought apples and oranges, then went home because it started raining. Who bought the fruit?” LSTM keeps “Ahmed” available until the question.
GRU — Gated Recurrent Unit
GRU is close to LSTM but simpler and often faster: two gates instead of three, one state instead of two.
Reset Gate
How much past to ignore when computing the new candidate. Helpful when the topic suddenly changes.
Update Gate
Mixes old and new memory in one knob — from “keep everything” to “replace with new.”
Today, most language tasks move to Transformers. LSTM/GRU remain essential foundations.
NLP Basics
Tokenization
Split text into tokens: words, subwords, or characters. Subwords help rare words by reusing known pieces.
tokens = text.split()
print(tokens) # ['I', 'love', 'deep', 'learning']
Embeddings
Each token becomes a vector of numbers. Similar meanings live near each other in that space.
“King,” “queen,” and “prince” cluster together. “Banana” and “apple” sit in fruit territory. Distance reflects relatedness.
Word2Vec · GloVe · FastText
| Method | Idea | When useful |
|---|---|---|
| Word2Vec | Learn from local context | Similarity search, NLP starters |
| GloVe | Global co-occurrence stats | Ready static vectors |
| FastText | Subword pieces | Rare words, typos, rich morphology |
Modern Transformers learn contextual embeddings: “bank” in “river bank” differs from “bank account.”
Attention
When translating a long sentence, you do not weight every word equally. You focus on what matters now. Attention gives the network that spotlight.
The stage is dimly lit, but the spotlight follows the actor. Translating “ate,” the light focuses more on “apple” than on “yesterday.”
Self Attention
Every word looks at every other word in the same sentence and asks: “who helps me most?”
“The blue bird sang on the branch because it was happy.” The word “it” should link to “bird,” not “branch.” Self-attention learns stronger weights between “it” and “bird.”
Query / Key / Value: like asking a library question (Query), matching book titles (Keys), then taking useful content (Values).
Multi Head Attention
Several heads in parallel learn different relation types: subject–verb, adjective–noun, pronoun–referent — then merge their views.
Transformers
“Attention Is All You Need” (2017) changed the game: process sequences with attention instead of only step-by-step recurrence. Faster GPU training, stronger long-range links.
Encoder
Reads the input and builds a rich contextual representation for each token via stacked Self-Attention + feed-forward layers.
Decoder
Generates the output step by step, looking at its own past tokens and at the encoder’s understanding of the source.
Positional Encoding
Attention alone does not know who is first or second. We add position signals so order is preserved.
Students have faces (meaning) plus seat numbers (position). Without seat numbers, seating order is lost. Positional encoding is the seat ticket for words.
Large Language Models
An LLM is a huge Transformer trained on vast text — often by predicting the next token millions of times until language patterns become fluent.
BERT
Bidirectional encoder. Great for classification, NER, and reading comprehension. Classic pretraining: fill in [MASK]ed words.
GPT
Generative decoder-style models. Basis of many chat assistants: writing, dialogue, summarization, coding (after instruction tuning).
T5
Casts every task as text-to-text: “translate: …”, “summarize: …”, “sentiment: …”.
Llama · Qwen · DeepSeek
Llama
Meta’s open family — popular for research and local customization.
Qwen
Strong multilingual family from Alibaba.
DeepSeek
Modern models noted for training efficiency and strong reasoning/coding.
| Model | Style | Common use |
|---|---|---|
| BERT | Understand (Encoder) | Classify, search, NER |
| GPT | Generate (Decoder) | Chat, write, code |
| T5 | Text→text | Translate, summarize |
| Llama / Qwen / DeepSeek | Modern LLMs | Assistants, apps, fine-tuning |
Applications
Translation
Source → Encoder understanding → Decoder target text. Modern systems are Transformer-based.
Chatbots
Read your messages as a sequence, keep dialogue context within a token limit, generate a reply token by token.
Summarization
Long article → short summary. Extractive (pick sentences) or abstractive (rewrite).
Sentiment Analysis
Product review → positive / negative / neutral. A small model or fine-tuned BERT often suffices.
Speech Recognition
Audio waveform → time features → sequence model → text. Your voice keyboard uses these ideas.
- Days 1–2: Hand-tokenize short sentences; understand sequences.
- Days 3–4: Tiny RNN for next-character prediction.
- Days 5–6: LSTM/GRU sentiment on short texts.
- Days 7–8: Study Attention; visualize weights if possible.
- Days 9–11: Hugging Face pipeline (translate or classify).
- Days 12–14: Small project + GitHub README.