Self-Attention

Language is **sequential** — the order and context of words matter. "The cat sat on the mat" is meaningful because each word relates to others in the sentence. To process language, a model needs to understand **relationships between words**: - "sat" relates to "cat" (who sat?) and "down" (how?) - "The" modifies "cat" (which cat?) - "cat" is the subject that governs the verb "sat" The challenge: how do we build a model that captures these relationships? We need a mechanism that lets each word "look at" other words in the sequence to gather context. This is the core problem that **attention** was designed to solve. Before attention, models processed words one at a time and tried to compress everything into a single fixed-size vector. As we'll see, this created a serious bottleneck.

Before transformers, **Recurrent Neural Networks (RNNs)** were the standard for sequence processing. An RNN reads one word at a time, updating a hidden state at each step: h₁ = f("The", h₀) → h₂ = f("cat", h₁) → h₃ = f("sat", h₂) → h₄ = f("down", h₃) This creates a **bottleneck**: by the time we reach "down," all information about "The" must be compressed into the hidden state vector. For long sequences (hundreds of words), early information gets washed out. **Three critical problems:** **1. Information bottleneck.** The hidden state has a fixed size (e.g., 512 dims) but must encode the entire sequence. Information from early tokens degrades as the sequence grows. **2. Sequential processing.** Each step depends on the previous one — you can't process "cat" until "The" is done. This means **no parallelization**, making training painfully slow on modern GPUs. **3. Vanishing gradients.** During backpropagation, gradients must flow through every time step. Over long sequences, they shrink exponentially, making it hard to learn long-range dependencies. The attention mechanism solves all three problems simultaneously.

The core idea of attention is beautifully simple: > **Every token looks at every other token** and decides how much to "pay attention" to each one. Imagine you're reading "The cat sat down" and you're currently processing the word "sat." Instead of relying on a compressed summary of previous words, you can directly look at every word in the sentence and decide which ones are relevant: - "cat" is very relevant (it's the subject — who sat?) - "down" is relevant (it modifies the sitting) - "The" is somewhat relevant (it's part of the subject phrase) The attention mechanism assigns a **weight** to each word based on its relevance. High weight = pay more attention. Low weight = mostly ignore. This is fundamentally different from the RNN approach. Instead of information flowing through a chain (The → cat → sat → down), every word has a **direct connection** to every other word. The path length between any two words is always 1, regardless of how far apart they are in the sequence. This direct connectivity is why transformers excel at capturing long-range dependencies that RNNs struggle with.

Self-attention can be understood as an **information retrieval system** operating within a single sequence. The word "self" means the sequence attends to itself — each token queries the same sequence it belongs to. Think of it like a library: - **Query (Q):** "I'm looking for information about who performed an action" - **Key (K):** Each word advertises what information it contains: "I'm a noun," "I'm a verb," "I'm a determiner" - **Value (V):** The actual content each word provides when retrieved For "sat" looking for its subject: 1. "sat" generates a Query: "Who performed this action?" 2. Every word generates a Key: "The" → "I'm a determiner", "cat" → "I'm an animate noun", etc. 3. The Query is compared to all Keys to get relevance scores 4. These scores weight the Values to produce the output This Query-Key-Value framework is the heart of the transformer. In the next lessons, we'll see exactly how Q, K, and V are computed and combined. For now, the key insight is: **Self-attention = each token retrieves relevant information from all other tokens in the same sequence.**

Let's see self-attention in action on our sentence "The cat sat down." Below is the actual attention weight matrix computed from the embeddings. Each row shows how one token distributes its attention across all tokens. The weights in each row sum to 1.0 (thanks to softmax normalization, which we'll cover in detail later). **Reading the attention matrix:** - Row = the token that is "looking" (the query) - Column = the token being "looked at" (the key) - Value = how much attention the query pays to that key For example, look at the row for "sat": - It assigns weights to each of "The", "cat", "sat", and "down" - Higher weights mean "sat" considers that token more relevant These attention weights are then used to create a weighted combination of all token representations. Each token's output is an **attention-weighted average** of all tokens' values — enriched with contextual information from the entire sequence. In the next lessons, we'll break down exactly how these weights are computed using Query, Key, and Value projections.

Self-attention gives transformers three superpowers that previous architectures lacked: **1. Massive Parallelism** All attention scores between all pairs of tokens can be computed simultaneously as a single matrix multiplication. On a GPU with thousands of cores, this is orders of magnitude faster than sequential RNN processing. A 512-token sequence that took an RNN 512 steps takes a transformer just one parallel step. **2. Constant-Length Paths** In an RNN, information from token 1 must travel through tokens 2, 3, ..., n to reach token n. Each step loses some information. In self-attention, every token has a **direct connection** to every other token — the path length is always 1. This makes it trivially easy to capture long-range dependencies. **3. Interpretable Attention Patterns** The attention weights form a human-readable matrix showing which words the model considers related. This provides a window into the model's "reasoning" — you can literally see which tokens influence which outputs. (Though attention weights aren't a perfect explanation of model behavior, they're more interpretable than hidden states.) These advantages are why the 2017 paper was titled **"Attention Is All You Need"** — self-attention alone is powerful enough to replace recurrence entirely.

You've learned the intuition behind self-attention — how it lets every token attend to every other token, solving the RNN bottleneck. Let's check your understanding!