A single block does two things, both introduced on the previous page. First an attention step, where every position reads every other position. Then a feed-forward MLP step, applied to each position on its own. A residual connection wraps each step, which means the step's input is added back to its output, so the block adjusts the running representation instead of replacing it. Frontier models stack 32 to 120 of these blocks.

Follow one token's representation up the stack to see why depth matters. The first block runs attention, so the token absorbs a weighted blend of every other token. Then the MLP reshapes that token on its own. The result is handed to the second block, which runs the same two steps again — but now on representations that already carry context from the first pass. Repeat for every block. By the top of a 100-block stack, each token has been informed by every other token dozens of times over, each pass working at a slightly more abstract level than the last.

What ends up living at each depth is not designed. It emerges from training. In practice, early blocks tend to resolve local grammar, middle blocks build meaning and relationships, and late blocks assemble the final prediction. Nobody assigns those roles; they emerge from the same next-token objective.

All three architectures use the same attention operation. What separates them is what each one is allowed to read and what it produces. An encoder reads the whole sequence at once, with every position free to look both left and right, and outputs a single summary vector, a fingerprint of the input. A decoder reads strictly left to right and writes one token at a time, each new token predicted only from the tokens before it. An encoder-decoder does both in sequence: it encodes the input, then writes a fresh output from that encoding.

Each shape maps onto products you know. Every chat model in 2026 — GPT, Llama, Gemini, Claude — is decoder-only, because holding a conversation is exactly that left-to-right writing task. Every image retriever, such as CLIP or SigLIP, is encoder-only, because retrieval needs one fingerprint per item to compare.

One misconception is worth naming. People often call GPT an encoder-decoder model. It is not — a decoder-only model has no separate reading stage; it simply continues the sequence it is handed. The encoder-decoder shape survives mostly in older machine-translation systems.

A model never sees letters or whole words. Before any text reaches the network, a tokenizer chops it into tokens. A token is a chunk of text: often a common word, often a fragment of a rarer one. The tools that do the chopping have names you will meet in code — BPE, SentencePiece, tiktoken — and each is a recipe for deciding which chunks are frequent enough to deserve their own slot. Common words usually become one token. Rare words break into several pieces. Each token is then converted into an integer ID, which is a position in a fixed vocabulary of roughly 30,000 to 200,000 entries.

The token is also the unit of money and memory. You are billed per token. The context window — the amount of text a model can hold in view at once — is measured in tokens. The cost of running the model scales with tokens. So every question about "how much fits in the context window" is really a question about token counts, not word counts.

The trap is to treat one token as one word. They are not the same. The string "GPT" is one token in one tokenizer, three in another, and four if the tokenizer falls all the way back to raw bytes. When the exact count matters — for a cost estimate, or for fitting a long document — run the real tokenizer instead of guessing from the word count.

Recall that each token is an integer ID. The embedding table turns that ID into a vector. The table holds one row per token in the vocabulary, and each row is a list of 768 to 4096 numbers. Looking a token up means reading off its row. That row of numbers is what actually enters the transformer; the integer ID was only an address pointing at it.

Those rows begin as random numbers. During pretraining they are nudged, again and again, so that tokens used in similar contexts end up close together in this space of numbers. Treat each row as coordinates: after training, king and queen land near each other, and far from banana. Nothing in a vector "knows" what a king is; the closeness is only the residue of training. You read it with cosine similarity, the angle between two vectors — a small angle means similar, a near-right angle unrelated.

This single geometry does a great deal of downstream work. It powers retrieval (the R in RAG, Page 20), clustering (grouping similar items, as in the taste-graph), and cross-modal alignment: CLIP and SigLIP are trained so that an image and its caption land near each other in one shared space. Firth put it more plainly in 1957: you shall know a word by the company it keeps.

The whole of pretraining is one task, repeated across trillions of tokens. Show the model a prefix of text, ask it for a probability distribution over the next token, then compare its answer against the token that actually came next. The penalty for being wrong is the cross-entropy loss: the less probability it placed on the correct token, the larger the penalty. No human writes an answer key — the document is its own, because the next word is whatever the author wrote.

A narrow task, but it forces broad ability. To predict the next token well, the model has to absorb whatever makes text predictable: grammar, meaning, facts, the rules of code, the steps of arithmetic, the shape of a dialogue. None of it is taught directly; each emerges because it is the cheapest way to lower the loss.

This same objective is where hallucination begins. The model is rewarded for a plausible continuation, not a true one — and plausible and true often diverge. A confident, well-formed, false statement still scores well in training, as long as it resembles something the data might contain. The model did not memorize the internet; it learned the statistical structure of it, and structure can be convincingly faked.

Two papers established the result. Kaplan et al. 2020 (arXiv:2001.08361) and Hoffmann et al. 2022 (arXiv:2203.15556, the Chinchilla paper) showed that training loss falls predictably as you increase three things: compute (the arithmetic spent training, in FLOPs), parameters (the model's adjustable weights), and training data. The relationship is a power law: plot loss against compute on logarithmic axes and the points fall along a straight line. Chinchilla added the correction in the figure — for a fixed compute budget, a smaller model trained on more data wins (~20 tokens per parameter).

The practical payoff is foresight. You do not have to guess whether a model ten times larger will be better. You fit the curve on smaller runs and extrapolate it. This is why a lab will commit a hundred million dollars to a single training run with confidence: the outcome is a forecast with error bars, not a gamble.

One caution keeps this honest. The laws predict loss, not capability. Loss falls smoothly, but specific abilities sometimes appear in jumps the smooth curve never foreshadowed. So the claim that "scaling laws mean AGI by 2027" reads more into the curve than it actually says. The curve forecasts a number going down, not a particular skill arriving on a date.

Page 12 left the model holding a probability distribution over the entire vocabulary at each step. Sampling is the rule that turns that distribution into one chosen token. Three knobs shape the choice.

Temperature rescales the raw scores (the logits) before the softmax turns them into probabilities. At temperature zero the model is deterministic: it always takes the single highest-probability token, the argmax. At temperature one it samples honestly from the distribution as the model reported it. Above one, the distribution flattens toward uniform, so unlikely tokens get a real chance. Top-p (nucleus sampling; Holtzman et al. 2019, arXiv:1904.09751) instead keeps the smallest set of top tokens whose probabilities sum to p, then samples within it. Top-k is the blunter cousin — keep the k most likely.

Sampling is the largest behavioral lever you have without retraining anything. The same model at temperature 0.2 and at temperature 1.2 can feel like two different collaborators. CoWriter's peer-pushback voice is part system prompt, part temperature. The figure makes one warning visible: higher temperature is more random, which is not the same as more creative. As temperature rises, so does the rate of hallucination, because you are deliberately giving low-probability, often wrong, tokens more room to win.

Put Page 13's lesson in one line and keep it in view: the model is doing exactly what it was trained to do. Training rewards a plausible next token. The person reading the output wants a true one. Hallucination is not a malfunction; it is the name for the distance between those two targets. A model with no access to a source of truth has no way to prefer the true completion over the merely plausible one.

Grounding is the general fix. It means giving the model an external source of truth to lean on at the moment it answers: text retrieved from a database, the result of a tool it ran, a document quoted with citations, a structured record. Without grounding, the model is a fluent autocompleter running past the edge of what it reliably knows. With grounding, its job changes — from inventing a plausible answer to composing one over context it has actually been handed.

This is why every shipping AI product has a grounding layer of some kind. Mechanically that layer is usually RAG (Page 20); in the interface it usually appears as citations. A bigger model lowers the hallucination rate but never removes it: the training objective underneath has not changed. The durable fix is architectural, not a question of scale.

A freshly pretrained model is fluent but not yet helpful. Hand it a question and it is as likely to continue or rephrase it as to answer — continuing text is all it has ever been trained to do. Supervised fine-tuning, or SFT, closes that gap with examples: people write thousands of paired items, an instruction and its ideal response, and the model trains on those pairs until answering, rather than continuing, becomes natural.

Who writes the pairs depends on the domain. General instructions can come from contractors. Specialized ones need experts — lawyers drafting legal questions and answers, doctors writing clinical examples. The training itself uses the very same next-token loss from pretraining; only the data has changed, from raw web text to this curated set of instruction-and-response pairs. (Fine-tuning simply means continuing to train an already-trained model on new data.)

SFT is step one of every commercial model's post-training stack, and the personality you experience starts to form here. A product like CoWriter does not perform SFT — that happens inside a frontier lab on thousands of GPUs. CoWriter works at a different layer: prompt-level steering on top of an already fine-tuned model.

Christiano et al. 2017 (arXiv:1706.03741) first scaled this idea: show human raters two of the model's outputs and ask which is better — thousands of raters, millions of comparisons. A separate network, the reward model, is trained to predict which output a human would prefer. The language model is then adjusted to produce outputs the reward model scores highly. That adjustment uses reinforcement learning (typically PPO), held in check by a KL penalty that punishes drifting too far from the SFT starting point — so it improves on preferences without forgetting how to write.

Rafailov et al. 2023 (arXiv:2305.18290) showed you can reach the same place more directly. Their method, DPO, uses the identical preference data — the same pairs of chosen-over-rejected outputs — but skips building a separate reward model, optimizing the language model straight from the comparisons. Cheaper, simpler, and increasingly the default at smaller labs.

This stage, not pretraining, is where "Claude's personality" or "GPT's voice" lives. The base model speaks in the averaged voice of its training text; post-training is what gives the assistant a consistent, recognizable character. It also carries a bias by construction: the model is tuned toward whichever rater pool produced the rankings — whose preferences count is itself an alignment decision.

Constitutional AI (Bai et al. 2022, arXiv:2212.08073) replaces some of that human ranking labor with model self-critique. The fixed reference is a constitution: a short written list of principles the answer should satisfy, such as being helpful, honest, and harmless. The loop runs like this. The model drafts a response. It then grades that draft against the constitution and writes a revision that scores better. The pair — weaker draft, stronger revision — becomes a new piece of the preference training data Page 18 needed, but generated without a human in the loop. RLAIF carries the same substitution into the ranking step, letting the model stand in for the human rater.

Humans write the constitution once; the repeated critique-and-revise work is done by the model. That shifts the cost of alignment from "thousands of raters" toward "compute," and compute scales on a very different curve than hiring does.

Claude is the most visible product built this way. The constitution is not a safety guarantee but a steering document, and a model can follow it imperfectly. Steering also has side effects: it cuts some failure modes, such as overt harmful output, and creates others, such as over-refusing harmless requests or echoing the constitution's tone too eagerly.

The whole language section reduces to four stages, in order. Pretraining gives the model fluency, by predicting the next token across the internet at scale. Post-training gives it judgment, by ranking pairs of its own outputs and tuning toward the better one. Sampling is the runtime knob that makes one fixed model behave like a cautious clerk at low temperature or a loose brainstormer at high. Grounding is what makes the output trustworthy, by tying it to a real source.

The split that matters for a builder runs down the middle of those four. The first two, pretraining and post-training, happen inside a lab on thousands of GPUs and are effectively fixed for you. The last two, sampling and grounding, are yours. When you ship a product on top of someone else's model, the surface you actually control is the sampling settings you choose and the grounding pipeline you build around them.

Four things to carry out of this section. Hallucination is the default behavior, not a defect to patch later. Personality is installed in post-training; the base model has none of its own. Trustworthy knowledge comes from retrieval, since the weights will otherwise improvise. And capability tracks scale, the one lever a builder does not hold.

Before the diffusion model sees anything, a separate text encoder reads your prompt. It is an encoder in the sense of Page 08: it takes in the whole prompt and outputs a sequence of vectors, one per text token. The encoder is frozen, meaning its weights were trained earlier and are held fixed while the diffusion model trains around it. Those output vectors are the form the diffusion model can actually use, because attention operates on vectors, not on letters.

SD3 and Flux ship three encoders at once because each was trained differently and is strong at something different. The CLIP encoders were trained on image-and-caption pairs, so they are strong on visual concepts but weak on grammar and counting. T5 was trained on language alone, so it is strong on composition, counting, and rendering text inside the image, at the cost of speed. Concatenated — their output vectors stacked into one sequence — they follow a complex prompt more faithfully than any single encoder can.

This reframes a common shorthand. "The model reads the prompt" hides where the language understanding lives: the text encoder reads the prompt and turns it into vectors; the diffusion model only ever attends to those vectors and has no grasp of English on its own. So when a prompt is misunderstood, the encoder is often where it went wrong.

The older design connects text to image with cross-attention. The Query/Key/Value roles from Page 06 are split across the two modalities: the image patches supply the Queries, the text tokens supply the Keys and Values. But the flow runs one way only — the image reads the text, and the text never updates in return.

MM-DiT, the design in SD3 and Flux, removes that asymmetry. It places text tokens and image patches in a single sequence and runs ordinary self-attention over the whole thing. Each modality keeps its own Q/K/V weights, but they share one attention pass. In the SD3 paper's words, it "enables a bidirectional flow of information between image and text tokens."

Because information moves both ways at every layer, text and image sharpen under each other. The payoff is closer prompt adherence, more legible text inside images, and steadier multi-subject handling — structural, not just a larger training run.

MM-DiT is the multimodal version of the DiT from Page 28. DiT ran a transformer over image patches alone; MM-DiT puts the text tokens into the same sequence as the image patches and runs one transformer over both. Each modality keeps its own weights for producing Queries, Keys, and Values, but they share a single self-attention pass. The bidirectional text-and-image flow described on Page 26 is the direct consequence of that one shared sequence — this page is simply its proper name and home.

SD3, Flux, and a growing list of 2025-2026 models are built on MM-DiT. What you see from it is closer prompt adherence, markedly better text rendered inside images (logos, signs, captions), and steadier handling of prompts that name several subjects at once. Those gains trace to the architecture change, not to a larger training run.

This is why SD3 and Flux are not simply "a newer Stable Diffusion 1." Architecturally they belong to a different family: the DiT lineage that runs a transformer over patches, rather than the U-Net lineage they replaced. Same diffusion idea, different engine underneath.

Lipman et al. 2022 (arXiv:2210.02747) reframed what the network is trained to predict. Classic diffusion predicts the noise to remove at each step. Flow matching instead trains the model to predict a velocity: at any point between pure noise and real data, it answers "which direction, and how fast, should this point move to get closer to the data?" Collect those answers across the whole space and you have a vector field, a set of direction arrows like the ones in the figure. The paper describes it as "a simulation-free approach for training CNFs based on regressing vector fields of fixed conditional probability paths."

The practical wins follow from that choice. There is no hand-tuned noise schedule to get right, because the model learns the trajectory directly. Training is more stable. And because the result is a velocity field, generating a sample becomes a standard calculus problem — follow the arrows from noise to data with an off-the-shelf ODE solver, the same kind of routine used to integrate any system of motion over time.

Flow matching and diffusion are not rival paradigms. They are close cousins that reach the same place by different math. The field now uses "diffusion" loosely to cover both, so when you read "a diffusion model" in 2026, it may well have been trained this way.

Rectified flow is a particular kind of flow matching. Flow matching (Page 30) learns a velocity field and samples by following its arrows. If those arrows trace a curved path from noise to image, you have to follow it in many small steps to stay accurate, the way you would trace a winding line with short pen strokes. Rectified flow adds a training term that penalizes curvature, pushing the path toward a straight line. The SD3 abstract states it directly: "Rectified flow is a recent generative model formulation that connects data and noise in a straight line."

A straight path is cheap to follow. Each step of the ODE solver can be large without drifting off course, so reaching the same quality needs far fewer steps. Flux reaches a high-quality image in about 20 steps where SD1 needed roughly 50, a 4× cut. The straighter geometry lets the math do more work per step.

SD3, Flux, and a lengthening list of 2025-2026 models train with rectified flow. The noticeably faster image generation of the past year is partly faster hardware, partly better samplers (Page 33), and partly this move to a straighter path.

Classifier-Free Guidance (Ho & Salimans 2022, arXiv:2207.12598) is a trick for making the model follow the prompt more strongly. It begins in training: about 10% of the time the prompt is dropped, so the model learns to denoise both with conditioning (the prompt) and without it. At generation time, every denoising step is then run twice — once given the prompt, once given nothing. Call those two predictions cond and uncond. The model follows an exaggerated version of their difference, as the figure's formula shows: prediction = uncond + scale × (cond − uncond). The larger the scale, the harder the result is pushed in the direction the prompt added.

That scale is the dial. At scale 1 the prompt barely pulls and you get a generic image. Scale 7 to 12 is the sweet spot most chat UIs default to: clear adherence with natural texture. Past 20 the image is overcooked — oversaturated, harsh, weirdly literal. This dial is exactly the "CFG scale" slider you see in image-generation interfaces.

CFG is the single largest prompt-adherence lever in image and video generation. The cost, noted at the bottom of the figure, is compute: running the model twice per step roughly doubles the work. Veo and other video models use the same trick.

The path a sample takes from noise to image is described by a differential equation, an equation that says how the point should change at each instant. A sampler is a numerical solver for that equation: a recipe for taking discrete steps along the path. Different samplers step differently, trading accuracy against speed in different ways. Karras et al. 2022 (arXiv:2206.00364) unified the many competing proposals into one framework, and modern interfaces expose a handful of the common ones in a dropdown.

How much the choice matters depends entirely on your step budget. At a low number of steps, samplers visibly disagree: same prompt, same seed, different result. At a high number of steps they converge toward the same image, because the path is being traced finely enough that the stepping rule stops mattering. So the "best" sampler is not absolute; it depends on the model and on how many steps you are willing to spend. Flux ships with a flow-matching ODE solver native to how it was trained; SD1 ships with DDIM.

Sampler choice does matter, precisely at the step counts people run in production, around 20 to 30, low enough that the samplers have not yet converged.

Every modern image generator is the same five pieces, with different choices plugged into each slot. The pieces: a VAE to compress and decompress (Page 24), diffusion to turn noise into a latent (Page 23), a backbone to do the denoising (U-Net or DiT), a conditioning stack to read the prompt (Page 25), and a sampler to walk the path (Page 33). SD1 fills those slots with VAE + U-Net + CLIP + cross-attention + DDIM. Flux fills the same slots with VAE + MM-DiT + CLIP-L + CLIP-G + T5-XXL + joint attention + a rectified-flow ODE. The same five slots, different parts in each.

Two of those slots changed enough to define the 2024-2026 era. The backbone moved from U-Net to MM-DiT, joining the DiT lineage. The training objective moved from classical noise-prediction diffusion to rectified flow. Both shifts originated in papers, not products. The faster, sharper image and video tools you have used this year are the downstream consequence of those two changes.

Sora shows behaviors nobody wrote into it: move the camera and objects stay consistent in 3D; let one pass behind another and it survives the occlusion; cut between shots and a character mostly keeps its face and clothes. None of this was coded as a rule. In the report's words, "these properties emerge without any explicit inductive biases for 3D, objects, etc. — they are purely phenomena of scale." An inductive bias is a built-in assumption a designer bakes in; Sora got none of these, and developed them anyway.

The mechanism is the striking part. The team did not build a 3D engine, an object tracker, or a physics simulator. They built one thing: a model that predicts spacetime patches, trained at very large scale. Something that behaves like a physics engine emerged inside it — not because anyone asked for physics, but because crudely tracking how the world moves is the cheapest way to predict what the next patches will look like.

This is also where to stay careful. Saying "Sora understands physics" claims too much. It models physics well enough to predict patches most of the time, and the team is explicit about where that breaks: "it does not accurately model the physics of many basic interactions, like glass shattering." The figure lists more of these failures. Emergent is not the same as correct: the behavior appeared because of scale, but appearing is not the same as being reliable.

Rich Sutton named the pattern in 2019: across the history of AI, general methods that scale with compute have repeatedly beaten clever, hand-crafted approaches built on human insight (his blunt version is in the figure). It is called bitter because the hand-crafted approaches are the ones researchers are proudest of, and scale keeps winning anyway. What makes this deck's moment notable is that the past decade ran the same experiment in three domains — language, image, and video — and got the same result in each.

The figure puts three curves side by side. Language models follow the Kaplan and Chinchilla scaling laws from Page 14. DiT, the image backbone, follows the same curve: more compute, lower FID. Sora's published comparison shows the same shape for video — scale compute from a base run to 4× to 32×, and sample quality climbs the whole way. Three independent fields, one relationship.

Both extreme readings are wrong. "Scaling is hitting a wall" overstates it: specific benchmarks saturate, but the underlying loss curves keep falling. The open question is whether data is the binding constraint — whether we run out of high-quality training data before the curves give out.

Line up every system in this deck and one thing repeats. A language model runs self-attention over text tokens. CLIP and SigLIP run it over image patches. DiT and MM-DiT run it over latent image patches. Sora runs it over spacetime patches. The multimodal models, Gemini and GPT-4o, run it over a single mixed sequence of text tokens and image tokens projected into the same space. The operation from Page 06 — every element attending to every other — sits inside all of them.

What changes from one system to the next is never that operation. It is the two things on either side of it. Upstream sits the tokenization and conditioning: how the world is cut into a sequence, and how a prompt is fed in. Downstream sits the decoder: how the output sequence is turned back into text, pixels, or motion. All the architectural creativity lives at those two edges. The middle stays the same.

If you build products on these models — CoWriter on LLMs, Mosaic on SigLIP plus Gemini Vision — the choices that age well respect how the model actually works. Use embeddings for retrieval (the geometry from Page 11). Use the post-training surface (system prompts, constitutions where available) for steering, the layer personality lives on. Use grounding (RAG, tools, citations) for truth, because the model cannot supply it on its own. The model has a grain; these choices run along it, not against it.

Each model comes down to one job, and your task is to feed it well — the figure's two columns say it: fluent autocompleter, fingerprint matcher, denoiser, each wanting good context, good queries, good conditioning. Supply the input the model is built to use, and stay out of its way.

It is a division of labor. The model is shared infrastructure: you did not train it and will not redesign it. What you own is everything around it — retrieval, prompts, tools, checks, interface. That scaffolding is where a product is won or lost, and it is the part that is yours to build.