Inside GPT-2: A Small Model, Understood

1. What are we actually building?

Imagine that we are going to build a language model, and that every component has to earn its place. No mysterious intelligence box. No experts yet. Just a sequence of understandable operations that turns some text into a prediction about what comes next.

Our guide is the smallest released GPT-2: about 124 million parameters. GPT-2 is not the simplest language model ever invented. It is, however, a manageable example of the decoder-only transformer, the family that many much larger language models belong to.

By the end, you should be able to explain where text enters, how earlier words affect later predictions, where most of the parameters live, and why copying an architecture does not copy its knowledge. We will finish by changing a few parts of this machine to understand GPT-OSS.

First, separate four quantities. Parameters are the learned numbers inside the model. Training tokens are the examples it has processed. Vocabulary size is the number of token identities its tokenizer can represent. Context length is how much token history a particular run can consider. A model with a billion parameters has not necessarily seen a billion tokens, and a large vocabulary does not give it a large context window.

Here is our first mental experiment. Suppose a machine has read a hundred million tokens, but its next-token predictions are still poor. Increasing the maximum context setting does not repair those predictions. Increasing its vocabulary does not automatically repair them either. Those changes affect different parts of the problem.

We will follow an imaginary prompt: The server returned an error because the request was missing. We want a continuation. Perhaps authentication. Perhaps a required header. The model must assign scores to many possible next tokens, using what it has learned and what appears in this particular context.

There are two separate things at work. The weights contain patterns acquired during training. The current context supplies information for this particular prediction. Neither is a searchable folder of sentences, even though a model can sometimes memorize and reproduce training text.

Keep that separation in mind: learned weights, temporary context, next-token prediction. Almost everything we discuss fits into one of those three ideas.

2. From text to a working representation

Before GPT-2 can do any neural computation, a tokenizer converts text into integer IDs. GPT-2 uses byte-level byte-pair encoding. Common sequences can become single tokens; less common text can be divided into smaller pieces. A token is not reliably a word. Spaces and punctuation matter too.

For the released tokenizer, Hello world becomes two IDs: 15496 and 995. Those numbers are labels. They do not mean that world is mathematically smaller than Hello. Adding a space before the first word can change the tokenization, which is one reason exact formatting matters.

Now imagine a very large table with 50,257 rows and 768 columns. Each row belongs to a token ID. Looking up an ID retrieves its row: 768 learned numbers. That row is its token embedding.

Do not imagine that the first number always means noun, the second means computer, and the third means happy. A useful feature can be spread across many coordinates. Individual coordinates do not arrive with human-readable labels.

GPT-2 also looks up a position embedding. It has learned vectors for 1,024 positions. The model adds the token vector and the position vector. A token at the beginning and the same token near the end therefore start with different combined representations.

To make the mechanics concrete, use a pretend model with only three features. Suppose a token vector is 2, minus 1, and 0.5. Suppose its position vector is 0.1, 0.2, and minus 0.3. Adding them gives 2.1, minus 0.8, and 0.2. That is all an embedding addition does. Real GPT-2 performs the same kind of operation with 768 features, using learned values rather than our invented example.

If our prompt has 20 tokens, we now have 20 vectors. Think of them as 20 rows of a working notebook. Every row initially says something about token identity and position. The transformer layers repeatedly update those rows with information derived from the context.

We call this evolving representation the residual stream. That phrase sounds more exotic than the operation. It is the running state to which successive parts of the network add their contributions.

The important boundary is that the tokenizer has not understood the sentence. It has supplied IDs. Embeddings have turned those IDs into trainable numerical representations. Understanding, to the extent the model develops it, depends on what happens through the entire trained network.

3. The repeating transformer block

GPT-2 small contains 12 transformer blocks. Each block has two main branches: attention and a feed-forward network, often called an MLP. Both contribute updates to the residual stream.

Picture one row in our working notebook. First, a normalized version of the current representation enters attention. Attention computes an update, and the block adds that update to the original representation. Next, another normalized version enters the MLP. The block adds that update too.

So the rhythm is: normalize, compute, add. Normalize, compute, add. Repeat through the stack. After the last block, GPT-2 applies a final normalization before producing vocabulary scores.

Why add updates instead of replacing everything? Consider editing a technical note. One pass adds context about the server. Another adds a clue about the request. You do not have to throw away the whole note after each pass. This is only an analogy: the model is adding vectors, not writing annotations. But it captures why a running state is useful.

These residual paths also help gradients travel through deep networks during training. They do not guarantee that training will be stable, and they do not freeze the original information forever. Later operations can amplify, suppress, or transform what came before.

Layer normalization works across the features of each token position. It centers and rescales that position's feature values, then applies learned scale and shift. It does not normalize all the words into one average sentence. Nor must its final output have exactly zero mean, because the learned transformation comes afterward.

The word decoder can be confusing here. GPT-2 does not require a separate encoder that first reads the prompt. Decoder-only describes this causal transformer arrangement. The prompt itself passes through the same stack used to produce continuations.

At this point, you know the skeleton. But we still owe explanations of the two branches. Attention lets one position use information from other permitted positions. The MLP transforms the features at an individual position. These are different jobs, and the distinction will matter again when we reach mixture-of-experts models.

4. Attention, without the magic

How does the final word in a prompt use something mentioned near the beginning? Attention provides a learned mechanism for combining information across positions.

Each position's representation is transformed into three kinds of vector: a query, a key, and a value. The query determines which keys receive high compatibility scores. The values supply the information that will be combined.

Try a library analogy. A query is a request for relevant material. Keys help decide which entries match. Values are the material you actually retrieve. Unlike a conventional library search, all three are learned numerical projections, and the retrieval usually blends multiple entries rather than selecting one exact document.

For each permitted position, including the current one, the model compares a query with that position's key using a dot product. It scales the scores, applies the causal mask, and uses softmax to turn the remaining scores into nonnegative weights that sum to 1. Those weights determine a weighted average of the value vectors.

Here is an invented example. Suppose a query produces weights of 0.6, 0.3, and 0.1 over three available positions. Suppose the value vectors are, respectively, 2 and 0; 0 and 2; and 2 and 2. The result is 1.4 and 0.8. We multiplied each vector by its weight and added the results. That little calculation is the central mixing operation.

In a real head, queries, keys, and values each have 64 features. Scores are divided by the square root of 64, which is 8. That scaling helps control the magnitude of the dot products before softmax. The learned projection matrices decide which combinations of features matter.

Now the causal rule. At position 5, the model may use positions 1 through 5, but not position 6. When we train it to predict the next token, allowing it to inspect that next token would leak the answer. The mask prevents that shortcut. The current position is allowed because its representation is used to predict what follows it.

GPT-2 small uses 12 attention heads. Each head can compute a different mixture. Their outputs are joined together and transformed back into an update of width 768. The heads are not 12 complete language models, and we do not preassign one to grammar and another to cybersecurity. Whatever specialization develops is learned and can be messy or overlapping.

Also, a bright square on an attention diagram is not a full explanation of why the model answered something. Attention weights are one part of a larger computation that includes value vectors, output projections, residual paths, and many later layers.

Finally, notice the cost. With full attention, every position can compare against many other positions. If we double sequence length, the number of position-pair scores grows by roughly four times. Efficient implementations can avoid storing the entire score matrix at once, but that does not erase the underlying pairwise work. This is one reason very long context needs more than a larger number in a configuration file.

5. The MLP and the vocabulary readout

After attention has mixed context into each position, the MLP processes the updated features. GPT-2 expands a 768-feature vector into 3,072 features, applies a nonlinear function called GELU, then projects back to 768 features.

The same MLP weights are used at every position within that layer. Each layer has its own weights. Unlike attention, this branch does not directly mix separate positions. It works on each position's current representation, which already includes information gathered by attention.

Why the nonlinearity? If you stack only linear transformations, their combined effect can be represented by another linear transformation. The nonlinear step gives the network a richer class of functions. The expanded space gives it room to form useful intermediate features before combining them back into the residual stream.

Imagine that attention has brought together clues about an error message, a missing request header, and an authentication requirement. The MLP can transform that contextual representation into features useful for prediction. This is a teaching analogy, not a claim that there is one identifiable authentication neuron doing the whole job.

After all 12 blocks and the final normalization, each position still has 768 features. To score the vocabulary, GPT-2 compares that final representation with the token embedding table. The output reuses the input embedding weights. This is called weight tying.

The result is a score for each of the 50,257 token identities. These raw scores are logits. Softmax can convert them into a probability distribution. During ordinary generation, we use the distribution at the last position to choose the next token.

Suppose three candidate continuations receive probabilities of 0.5, 0.3, and 0.2 in a toy vocabulary. Greedy decoding always chooses the first. Sampling can choose another. Temperature changes how concentrated the distribution is; other sampling rules can restrict the candidate set. Those choices affect the output, but they do not teach the model new knowledge.

6. How those numbers learn

An architecture is a recipe for computation. At random initialization, its matrices have not yet acquired the patterns that make useful language predictions. Pretraining changes that by repeatedly comparing predictions with actual continuations and updating the weights.

Use four abstract token IDs: A, B, C, and D. We provide A, B, and C as the input and ask for B, C, and D as the corresponding next-token targets. The causal mask makes each prediction depend only on the appropriate prefix. We can compute these training predictions in parallel, because the complete training sequence is already available.

Cross-entropy measures how much probability the model assigned to the correct targets. For one target, a probability of 0.5 produces a smaller penalty than a probability of 0.05. Backpropagation computes how the loss changes with the parameters. An optimizer then adjusts them. Repeat across batches, and the model can gradually become better at predicting the training distribution.

That description hides difficult engineering choices: data quality, batch size, learning rate, numerical precision, checkpointing, and evaluation. But it identifies the learning signal. We do not manually write English grammar into individual matrix entries.

There is a subtle difference between training and generation. During training, the prefix generally contains actual tokens from the dataset. During generation, the next step sees tokens the model itself just produced. An early mistake can therefore affect many later decisions.

At inference time, the normal process keeps weights fixed. Read the prompt, predict a token, append it, and repeat. Reading a correction in the current conversation can change the next answer through context. It does not ordinarily mean the model's weights have been retrained.

A key-value cache saves the keys and values already computed for earlier positions, so generation does not need to recalculate all of them on every step. The cache is temporary state for this sequence. It is not another pretrained model, and it is not permanent learning.

Think of three different tests. Training loss asks whether predictions improved on the examples used for optimization. Held-out loss asks whether that improvement extends to separate text. A practical task evaluation asks whether the model can do something useful, such as explain a technical paragraph or repair a small program. Passing the first test does not guarantee passing the other two.

This is why an early checkpoint is interesting without being a finished assistant. It may reveal emerging spelling or local syntax before it can sustain coherent explanations. Evaluate what it actually produces, and keep comparison prompts fixed so that changes across checkpoints mean something.

7. Where the parameters and memory go

Let us do a small audit. The token embedding table alone contains about 38.6 million numbers. The learned position table adds about 0.79 million. The 12 transformer blocks contribute about 85.1 million. Add the final normalization, and the exact tied-weight total is 124,439,808 parameters.

Within a block, the two MLP projections contain roughly twice as many matrix weights as the attention projections. Attention gets much of the publicity, but the feed-forward network occupies a substantial part of the model. The companion guide shows the arithmetic so that you can reproduce it.

Weight tying matters to that count. If we accidentally count a second independent vocabulary output matrix, we add another 38.6 million parameters that this GPT-2 model does not actually have.

Now distinguish parameter count from memory use. If each stored parameter used 2 bytes, 124.4 million parameters would occupy roughly 249 million bytes before other overhead. That multiplication estimates weight storage, not total training memory.

Training also needs activations, gradients, optimizer state, and temporary workspaces. Long sequences increase activation costs. Depending on the training method, extra copies or representations of weights may also be present. This is why a model that fits comfortably for inference can require much more memory to train.

Quantization reduces the precision used to represent some numbers. It can reduce storage and memory traffic, but it does not reduce the number of learned parameters. Pruning removes or disables parts of the model. Distillation trains a student from a teacher's outputs or other signals. These operations are related by practical goals, but they are not interchangeable.

Nor can we promise a particular speed just from the parameter count. Hardware, kernels, sequence length, batch size, precision, and whether we are training or generating all matter. A credible time estimate begins with a measured throughput for the actual workload, then divides the planned token count by that throughput and allows for evaluation, checkpoints, and interruptions.

8. What GPT-OSS changes

We can now approach GPT-OSS as a set of changes to a familiar machine. It remains an autoregressive transformer, but several internal components differ from GPT-2.

Its mixture-of-experts layers replace the single dense feed-forward path with a collection of expert networks and a router. For each token, the router selects a small number of experts whose outputs are combined. Attention remains another part of the block.

The 20B release has about 21 billion total parameters and about 3.6 billion active per token. It uses 32 experts per layer and selects 4. The 120B release has about 117 billion total parameters and about 5.1 billion active per token, with 128 experts per layer and again 4 selected. Total and active counts answer different questions about storage and computation.

An expert is not a complete little chatbot. It is a feed-forward component inside a larger network. It depends on the surrounding attention, routing, residual representations, and other layers. Taking one expert does not extract a self-contained model with all the original capabilities. Keeping one expert per layer creates a substantial architecture change that would need recovery training and evaluation.

GPT-OSS also uses grouped-query attention. Multiple query heads share key and value heads. This can reduce the key-value cache compared with giving every query head separate keys and values. It alternates sliding-window attention with full attention, allowing different layers to have different access patterns.

For positions, it uses rotary embeddings with a long-context scaling scheme, rather than GPT-2's learned absolute position table. Its supported context is 131,072 tokens. That does not mean that changing GPT-2's context setting to the same value gives the same capability. The position mechanism, training distribution, computation, and evaluation all matter.

Other changes include RMS normalization and gated feed-forward computations. These modify how features are scaled and transformed. They are useful architectural choices, but no single substitution explains the difference between a modern trained reasoning model and GPT-2.

Finally, MXFP4 is a low-precision format used for GPT-OSS expert weights. It combines very low-precision floating-point values with shared scaling over small groups. It does not mean every computation or every stored tensor is 4-bit, and it does not turn 21 billion parameters into a 4-billion-parameter model.

Think of architecture, training, and deployment as three connected decisions. Architecture determines the operations. Training determines the learned weights and behavior. Deployment determines how those operations and weights run on particular hardware. A good explanation keeps all three in view.

9. Building a smaller model with a teacher

This brings us back to a practical question: can we take an architecture, make it smaller, and fill it using a larger teacher?

Yes, a teacher can supply training signals for a smaller student. But there is no general operation that pours knowledge directly from one set of differently shaped matrices into another. If the student starts from random weights, it still has to learn its representations through optimization.

One approach is to generate high-quality text with the teacher and train the student on that text. Another asks the student to match the teacher's probability distribution, when those probabilities are available and compatible. A softer distribution can carry information about alternatives that a single sampled answer does not show.

For a tiny illustration, suppose a teacher assigns 0.7 probability to one technical term, 0.25 to a plausible synonym, and 0.05 to an unrelated word. A sampled target might reveal only the first term. Matching the distribution can also teach which alternative was plausible. In real systems, vocabulary alignment, storage, and teacher computation complicate that process.

Distillation from random initialization is therefore a form of training from scratch guided by a teacher. It can be part of pretraining, and it does not eliminate the need for enough varied data or a sound training setup. A powerful teacher also cannot guarantee that a much smaller student will reproduce every capability.

If you want a 1-to-2-billion-parameter experiment, a dense decoder is easier to understand and audit than a newly designed mixture of experts. That is an engineering judgment about complexity, not a law that small expert models cannot work. You would compare alternatives with measured quality and throughput under the same budget.

Long context should have its own tests. A model that accepts 262,144 tokens has passed an input-handling test. Finding the right detail near the beginning, using multiple distant facts, and resisting irrelevant distractions are separate capability tests. Memory capacity alone does not establish useful long-context reasoning.

Here is the model you can now picture. Token IDs become vectors. Positions are represented. Attention gathers permitted context. Feed-forward layers transform features. Residual additions carry the working state through the stack. A vocabulary readout produces scores. Training improves those scores by updating weights; generation uses the learned weights to continue a sequence.

Once those steps are clear, the bigger models stop being an entirely different mystery. They become variations whose costs, benefits, and limits you can ask precise questions about.