JCR Labz builds teaching material for natural language processing: the book, the slides, the code that produces every table in them, and animations that build a representation in front of you one word at a time.
This is the first idea in the course, running on your sentence. A bag of words keeps which words appeared and how often, and forgets the sequence entirely. Type something and watch how much a model gives up for a fixed-length vector.
Seven units in order, so nothing arrives out of nowhere. Start at "what even is a word" and finish knowing why attention works. Read it in the browser or take the PDF.
Small facts about language and machines that are worth knowing even if you never train a model. Each one links to the page that explains it properly.
To a bag-of-words model these are the same sentence. Both become the same four counts: the 2, dog 1, bites 1, man 1. Word order is the first thing you throw away, and buying it back is most of what the next fifty years of NLP is about.
2 sentences, 1 vectorRead it →Rank words by frequency and the counts fall off a cliff: rank 2 gets about half of rank 1, rank 3 about a third. It holds in every language anyone has checked, in Tamil as in English. Nobody decided this. It falls out of how people use words.
f x rank is about constantRead it →They see tokens. Byte-pair encoding starts from single characters and repeatedly glues the most frequent pair together, so 'unhappiness' can arrive as three pieces while 'the' is one. Vocabulary size is a budget: too small and words shatter, too large and the embedding table eats your parameters.
50k to 100k tokens, typicallyRead it →In the HAL model, 'chuck' scores 36 against 'woodchuck' when you count what came before it, and 10 when you count what came after. Word order leaves a fingerprint in the counts, which is why a HAL vector keeps both directions rather than averaging them.
36 before, 10 afterRead it →Firth wrote that in 1957 and it is still the whole idea behind word vectors. Nothing in word2vec, GloVe or a transformer's embedding layer knows what a word means. They know which words turn up nearby, and that turns out to be enough to do arithmetic on meaning.
Firth, 1957Read it →A model with perplexity 100 is as unsure as if it were picking uniformly from 100 words at every step. That is the whole intuition. It also means perplexity is only comparable between models that share a tokenizer, which is why leaderboard numbers so often are not.
same tokenizer, or no comparisonRead it →Multiply probabilities together and a single zero takes the whole product to zero. The fix is old and blunt: pretend you saw everything a fraction of a time more than you did. Every modern model still needs some version of it.
one zero kills the productRead it →A query asks a question, every key offers a match score, the softmax turns those scores into weights, and the answer is a weighted blend of the values. That is it. The magic is not the mechanism, it is that you can run it on every word at once.
query, key, valueRead it →Self-attention looks at all positions at once, which means it has no idea which came first. Position has to be added to the input explicitly. Without it, the most sophisticated architecture in the field falls back to the least sophisticated representation there is.
no position, no orderRead it →It flattens the probability distribution before sampling. Near zero the model always takes its top guess and repeats itself. Turn it up and unlikely words get a real chance, which reads as invention right up until it reads as nonsense.
0 repeats, 2 ramblesRead it →It counts n-gram overlap with a reference. A translation that says the opposite thing in familiar words can score above one that says the right thing in unusual words. Everyone knows this, and the field still reports it, because the alternatives cost money or people.
overlap is not meaningRead it →The arithmetic that gives you king minus man plus woman equals queen also gives you doctor minus man plus woman equals nurse. Nothing went wrong in the training. The model faithfully learned what the corpus contains, which is the problem.
same arithmetic, both resultsRead it →All of it runs in your browser, on your text, with no server and no model download. Drag a slider until something breaks. That is the point.
Two sentences that mean opposite things, dropped word by word into a bag. Both bags come out identical.
A ramped ten-word window fills two matrices, and they disagree. That asymmetry is the whole idea.
A symmetric four-word window fills one matrix, step one of the three that make a COALS vector.
Train BPE on your own text, fit Zipf and Heaps to a real corpus, do arithmetic on word vectors, and generate from an n-gram model.
Everything here is written for a 30-hour course in natural language processing, and every piece is built from the same source, so the book, the slides and the code cannot drift apart.
307 pages that go where slides cannot: the derivations, the worked tables and eight projects you build. Reads in the browser, or download the PDF.
26 decks, in the order they are taught, with the diagrams the lectures use. Free to take and re-teach.
21 scripts that produce every worked table in the book. If a number is printed, a script computes it, and you can open that script in Colab.
Seven demos that run in your browser. Nothing to install, no model to download, and nothing hidden behind a signup.