<Technical Deep Dive> How do language models memorize

How much do language models memorize?

We propose a new method for estimating how much a model knows about a datapoint and use it to measure the capacity of modern language models. Prior studies of language model memorization have struggled to disentangle memorization from generalization. We formally separate memorization into two components: unintended memorization, the information a model contains about a specific dataset, and generalization, the information a model contains about the true data-generation process. When we completely eliminate generalization, we can compute the total memorization, which provides an estimate of model capacity: our measurements estimate that GPT-style models have a capacity of approximately 3.6 bits per parameter. We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point “grokking” begins, and unintended memorization decreases as models begin to generalize. We train hundreds of transformer language models ranging from $500K$ to $1.5B$ parameters and produce a series of scaling laws relating model capacity and data size to membership inference.

arXiv.orgJohn X. Morris

Imagine buying a brand new notebook. It has a fixed number of pages, and once you fill them, that is it. There is no more room, no matter how much you still want to write. It turns out that large language models (LLMs), the technology behind tools like ChatGPT, work in a surprisingly similar way. They have a fixed amount of "room" for facts, and researchers have now measured almost exactly how big that notebook is.

A team of researchers from Meta, Google DeepMind, Cornell University, and NVIDIA, led by John X. Morris, set out to answer a deceptively simple question: how much does a language model actually memorize? Their paper, "How much do language models memorize?", gives a concrete number, and along the way it clears up a lot of confusion about what these models are really doing when they seem to "know" things.

Two kinds of remembering

Before this paper, people struggled to define what "memorization" even means for an AI. Here is the tricky part. If a model correctly answers "What is 2 to the power of 100?", did it memorize that exact answer, or did it simply learn how to do math? Those are two very different things, and telling them apart had been a long standing headache in the field.

The researchers untangle this by splitting memory into two clean buckets. The first is generalization, which is the useful, reusable knowledge about how the world works. This is like a student who understands the rules of multiplication and can solve any problem you throw at them. The second is unintended memorization, which is the model storing specific details about one particular example it saw during training. This is like a student who simply crammed and memorized the answer to question 7 without understanding why.

That second bucket is the one that matters for privacy. If a model has genuinely memorized a specific sentence from its training data, word for word, that raises real questions about whether it could accidentally leak private information.

Measuring memory in bits, like a zip file

So how do you actually measure how much a model has memorized? The researchers borrow a clever idea from the world of file compression. Think about how a zip file works. If a piece of text has a lot of predictable patterns, you can squeeze it into a much smaller file. Random gibberish, on the other hand, barely compresses at all because there are no patterns to exploit.

The insight is this: if a model has memorized something, then that model can help you compress it into fewer bits. The number of bits you save is a direct measurement of how much the model "knows" about that specific piece of data. This idea has deep roots. It connects back to Claude Shannon's information theory from 1948, which established that prediction and compression are two sides of the same coin, and to Andrey Kolmogorov's work on measuring the information content of a string. In plain terms, a bit is the smallest unit of information, a single yes or no, and the researchers count exactly how many of them a model has stashed away.

The clever trick: feeding models pure randomness

Here is where the experiment gets really smart. To measure pure memorization without any generalization getting in the way, the team trained models on completely random data, strings of bits with no patterns at all. Since random data has nothing to learn or generalize about, anything the model retains has to be raw memorization. There is nowhere to hide.

They trained hundreds of models for this, ranging from tiny ones with 500,000 parameters up to fairly large ones with 1.5 billion parameters. (Parameters are the adjustable dials inside a model, the things it tweaks during learning. More parameters generally means a bigger, more capable model.)

The headline number: 3.6 bits per parameter

This is the big result. Across all their experiments, GPT style models consistently stored about 3.6 bits of information per parameter. That number was remarkably stable no matter how they sized the models.

Think of it like discovering that every filing cabinet drawer, regardless of the office it sits in, holds roughly the same number of folders. Once you know that fixed rate, you can estimate the total capacity of any model just by counting its parameters. A model with a billion parameters can memorize roughly 3.6 billion bits, and no more.

One neat detail: they tested whether giving each parameter more numerical precision (more decimal places, so to speak) would boost capacity. Doubling the precision from 16 bits to 32 bits per parameter barely helped, nudging capacity from about 3.5 to 3.8 bits per parameter. In other words, the extra storage space mostly went unused. The model does not need all those decimal places to hold facts.

When the notebook fills up: enter "grokking"

This is my favorite part of the paper. What happens when you keep feeding a model more and more data than it can possibly memorize? Something wonderful. Once the notebook is full and there is no room left to memorize individual examples, the model is forced to get smart. Instead of storing every example separately, it starts finding shared patterns and general rules, because that is the only way to keep improving. In other words, running out of memory is exactly what pushes a model to start truly understanding.

This moment, when a model shifts from memorizing to genuinely generalizing, is known in the field as "grokking." The researchers found that grokking begins precisely when the amount of data exceeds the model's memorization capacity. This also gives a satisfying explanation for a mysterious phenomenon called "double descent," where a model's performance gets worse, then surprisingly gets better again as you add more data. It turns out that dip happens right at the point where the data outgrows the notebook.

Good news for privacy

This capacity limit has a reassuring consequence. A common worry is whether someone can perform a "membership inference" attack, which means figuring out whether your specific data was part of a model's training set. The researchers built a scaling law, essentially a predictive formula, connecting model size, dataset size, and how easy this kind of attack is.

The pattern is intuitive once you see it. The more data you cram into a fixed size model, the more that model is forced to blend everything together into general patterns rather than storing your specific example. So your individual data point gets diluted in the crowd. Their formula predicts that for today's biggest models, which are trained on truly enormous amounts of data, reliably detecting whether any single average example was in the training set becomes essentially impossible. To validate this, they trained models up to 1.5 billion parameters and found their predictions landed within a couple of percentage points of reality.

There is a catch worth mentioning. When they looked at which specific examples models did memorize, the winners were the unusual ones, texts full of rare words or snippets from other languages like Japanese, Chinese, or Hebrew sitting inside an English dataset. The oddball data sticks out and gets memorized, much like you are more likely to remember the one bizarre stranger at a party than the dozens of forgettable ones.

Why this matters

For years, the debate about whether language models are "just memorizing" or "actually learning" has been fuzzy and hard to pin down. This work gives us a ruler. It shows that models memorize until their fixed capacity fills up, at roughly 3.6 bits per parameter, and then they pivot toward genuine understanding. That single finding ties together memorization, learning, privacy, and the strange behavior of grokking into one clean story.

The next time you chat with an AI and it recalls a fact, you can picture that little notebook inside it, its pages filled at a precise and now measurable rate. And when it says something genuinely insightful rather than parroted, that may well be a sign its notebook filled up and forced it to actually learn.

You can read the full paper, "How much do language models memorize?" by Morris and colleagues, on arXiv (arXiv:2505.24832) using the link at the top of this post.