Two numbers per digit.
An autoencoder learns to squeeze its input into a few numbers and to rebuild the input from them. With only two numbers, every digit gets a place on a map.
Squeezing a digit into two numbers
The encoder reads all 784 pixels and boils them down to two numbers: a place on the map. Two numbers can’t hold every detail, so it keeps what tells digits apart most, mainly which digit it is and roughly how it’s written. Digits that look alike end up close together.
Drawing a digit from two numbers
The decoder does the reverse. From any two numbers it draws a 28 × 28 image, giving every pixel a chance of ink. It learned by redrawing 60,000 training digits from their two numbers and being scored on how far each pixel was off.
Why the map is smooth
This is the variational part. In training, the encoder gives a small cloud instead of an exact point, and the decoder has to redraw the digit from a random spot in that cloud, so nearby points must decode to similar digits. A second term, the KL divergence, pulls every cloud toward the middle, so the map has no empty gaps: wherever you put the point, there’s a digit.
Model and credits
The network is a variational autoencoder with 469,268 parameters, trained for this site on the 60,000 training digits of the MNIST database by Yann LeCun, Corinna Cortes and Christopher J. C. Burges (CC BY-SA 3.0). The dots are 5,000 digits from the MNIST test set, which it never trained on; for most of them, most of their five nearest neighbors on the map are the same digit. The examples come from the test set too, and the “looks like” reading comes from the network on the Digits page. Everything runs in your browser; nothing you draw leaves your device.