Memory, one character at a time.
A recurrent network keeps no copy of the text. Everything it knows about what came before has to fit in the memory it carries forward.
It reads one character at a time
Each character updates the memory, and the memory alone predicts the next character. Once read, a character is gone; only what the memory kept of it can still help.
Gates decide what to keep, write and reveal
For every cell, the forget gate decides how much of the old memory to keep, the input gate how much new information to write, and the output gate how much of the memory to reveal to the prediction. Each is a number between 0 and 1.
Why memory fades
Every step multiplies the old memory by a forget gate, so a character’s influence shrinks as more text arrives unless the gates hold it open. Without gates nothing can hold it open: every step shrinks the pull of older text by about the same factor, so it fades away steadily. Even the LSTM remembers only the last dozen or so characters well, enough to finish a word but not to keep a thought going, which is why its own sentences drift. The Transformer avoids this: attention looks back at every earlier token directly instead of carrying a memory forward.
Model and credits
The network is a long short-term memory (LSTM) with one layer of 256 cells and about 346,000 parameters, trained for this site on nineteen public-domain children’s books from Project Gutenberg, among them Alice’s Adventures in Wonderland, The Wonderful Wizard of Oz, Peter Pan, The Jungle Book, Treasure Island, The Secret Garden, The Wind in the Willows, Anne of Green Gables, and the fairy tales of the Brothers Grimm and Hans Christian Andersen. Paragraphs with the period slurs some of these books use were left out. The last 5% of each book was held out from training; on that text it averages about 2 bits per character. The network it is compared with has no gates: one layer of 512 units, h = tanh(Wxx + Uh + b), with about 329,000 parameters, trained the same way on the same books. On the same held-out text it averages about 2 bits per character. Both run entirely in your browser; nothing you type leaves your device.