From words to a prediction.
GPT-2 repeats one recipe twelve times, then turns its last vector into a probability for every token it knows.
Tokens and embeddings
Text is split into tokens: whole words, pieces of words, or punctuation. Each token looks up a learned vector of 768 numbers, and a second vector records its position.
Attention
Each head compares a token's query with every earlier token's key. The softmax of those scores decides how much of each value flows forward. Future tokens are masked, so the model can't peek ahead.
Switch a head off to test what it does. In the Mary and John example, block 12, head 11 looks at “Mary”, yet without it “Mary” rises from 45% to 73%: the head was arguing against the answer.
MLP and the next word
An MLP then reshapes every token's vector on its own. After twelve blocks, the last vector is compared with all 50,257 token embeddings to score what comes next.
Model and credits
GPT-2 small and its tokenizer are by OpenAI, released under the MIT license. This page runs the original weights, converted to 8-bit blocks with exact values kept where rounding would change predictions; next-token probabilities stay within a fraction of a percentage point of the original. Inference is Neural Network Visualizer's own implementation, running in a Web Worker: WGSL kernels on your GPU through WebGPU where the browser allows it, and TypeScript on the CPU otherwise. The two agree to within rounding.
The idea of exploring GPT-2 this way comes from Transformer Explainer by Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Jay Wang, Seongmin Lee, Benjamin Hoover and Polo Chau (Georgia Tech). This page is an independent design; no code or graphics are shared.