SESSION 1 · ~45M
How models are trained
A model's behavior is not magic — it is the residue of three training stages, and knowing them tells you exactly why models behave the way they do.
Frontier models are built in stages, each of which leaves a visible mark on how the model behaves. Understanding the pipeline demystifies responses that otherwise feel arbitrary — why a model hedges, refuses, or suddenly becomes cooperative.
Pretraining: learning to predict
The model is shown enormous quantities of text and learns to predict the next token. This is where raw language ability, world knowledge, and reasoning patterns come from — but the result is a document-completer, not an assistant. Ask it a question and it may answer with another question, because that is what documents do.
Instruction tuning & RLHF
- Instruction tuning: humans write thousands of ideal question-and-answer pairs so the model treats an instruction as a thing to fulfill, not text to continue.
- RLHF: humans rank pairs of responses by helpfulness, honesty, and safety; a reward model learns those preferences and nudges the base model toward them.
- The result is a model that hedges appropriately, declines harmful requests, and explains its reasoning.
When a model refuses a reasonable request, over-apologizes, or sounds conspicuously careful, you are feeling RLHF — the steering layer applied on top of raw capability. The skill is recognizing this and re-specifying your request so the helpful behavior is clearly the right answer.
Behavior is the residue of all three stages: capability from pretraining, compliance from instruction tuning, and temperament from RLHF. When a model surprises you, ask which stage the surprise came from — and adjust your prompt to work with that layer rather than against it.
TRY IT
Ask a model a mildly ambiguous question and notice the hedging language. Now rephrase the same question with explicit context and constraints. Watch the hedge shrink — you are working with the training, not against it.
CHECK YOUR UNDERSTANDING
After pretraining alone, why might a model respond to a question with another question?
What is the role of RLHF in the training pipeline?
When a model over-apologizes or sounds conspicuously careful, which training stage are you noticing?
NOTEBOOKLM
Connect your notebook
Paste a NotebookLM share link to embed your notebook for this module.