Neural Network Visualizer

Watch GPT-2 choose its next word.

This is GPT-2 small, the real 124-million-parameter model, running in your browser. Type anything and follow it through every layer.

Load GPT-2 small

The model is about 130 MB. It downloads once and stays in this browser, so your next visit opens instantly.

Sampling

Embedding

0 tokens

    Your text becomes tokens. Hover one to see what it attends to.

    Token embedding 768 values each
    + Position where it sits
    = Input to block 1

    Transformer blocks

    12 blocks, each with 12 attention heads and an MLP

    Attention heads in block 1

    Each head looks for something different. Choose one to open it.

    Head 1

    Switch off a head to see what it was contributing. Its output is replaced by zeros; everything after it is recomputed.

    Query Q = X · WQ
    Key K = X · WK
    Value V = X · WV
    This head’s weights 768 inputs across, 64 outputs down
    WQ WK WV
    Scores Q·Kᵀ / √64, future masked
    Attention softmax of each row
    Output attention · V

    MLP 768 → 3,072 neurons with GELU → 768, for every token

    Next token

      Lower the temperature for safer choices; raise it for surprises.

      Final hidden state scored against all 50,257 tokens

      How the guess forms

      What could follow the last token, block by block, before temperature and sampling

        Early blocks guess loosely. Watch the answer settle by the last block.

        Which heads matter

        Switch off each head on its own and see how the top guess changes.

        Amber heads were helping the top guess; violet ones were holding it back. Choose one to open it.

        From words to a prediction.

        GPT-2 repeats one recipe twelve times, then turns its last vector into a probability for every token it knows.

        Tokens and embeddings

        Text is split into tokens: whole words, pieces of words, or punctuation. Each token looks up a learned vector of 768 numbers, and a second vector records its position.

        Attention

        Each head compares a token's query with every earlier token's key. The softmax of those scores decides how much of each value flows forward. Future tokens are masked, so the model can't peek ahead.

        Switch a head off to test what it does. In the Mary and John example, block 12, head 11 looks at “Mary”, yet without it “Mary” rises from 45% to 73%: the head was arguing against the answer.

        MLP and the next word

        An MLP then reshapes every token's vector on its own. After twelve blocks, the last vector is compared with all 50,257 token embeddings to score what comes next.

        Model and credits

        GPT-2 small and its tokenizer are by OpenAI, released under the MIT license. This page runs the original weights, converted to 8-bit blocks with exact values kept where rounding would change predictions; next-token probabilities stay within a fraction of a percentage point of the original. Inference is Neural Network Visualizer's own implementation, running in a Web Worker: WGSL kernels on your GPU through WebGPU where the browser allows it, and TypeScript on the CPU otherwise. The two agree to within rounding.

        The idea of exploring GPT-2 this way comes from Transformer Explainer by Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Jay Wang, Seongmin Lee, Benjamin Hoover and Polo Chau (Georgia Tech). This page is an independent design; no code or graphics are shared.