The Court of Public Prose
From the clerk's desk
Strong Model Collapse: What the Paper Claims, and What It Cannot Tell Us About the Web
Strong model collapse is a result from an ICLR 2025 paper: in its setting, even a small share of synthetic training data can stop larger training sets improving a model. It is a claim about training. It does not show the web's writing converging.
By Jack Stovell · Published
Short answer: the result holds inside a stated setting, a supervised regression model with synthetic data mixed into its training set. To apply it to the real web you need several assumptions the paper does not test. This post sets out what the paper says, what it tested and what it leaves open.
What ‘strong model collapse’ means in one paragraph
The paper is ‘Strong Model Collapse’ by Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian and Julia Kempe. The arXiv version lists their affiliations as FAIR at Meta, the NYU Center for Data Science, the NYU Courant Institute and UCLA. It was posted to arXiv (2410.04840) in October 2024 and published at ICLR 2025. The core claim, from the arXiv abstract: “even the smallest fraction of synthetic data (e.g., as little as 1% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance.” Normally, more data buys a better model. The strong version says that once some synthetic data is in the mix, more data can still help at first, but the gains slow and performance “eventually plateaus above the baseline”. Under the paper’s assumptions, scale cannot remove the error the synthetic share adds. It is a statement about a training process.
Check which version you are reading. The abstract in the ICLR proceedings gives a different example: “as little as 1 per 1000”. Same paper, same claim, different illustrative fraction. Quote the version you actually read.
How it differs from ordinary model collapse
Ordinary model collapse is covered in what is model collapse. Read that first if the term is new. The short version: it describes models trained in a loop on earlier models’ output, losing rare content first.
The strong result asks a narrower question: can scale rescue you at all? The loop is not the point. The point is a fixed mix of real and synthetic data, and whether a bigger pile of that mix keeps paying off. The paper’s first main result holds “as long as the fraction of training data which is synthetic does not vanish”, and its conclusion says the effect “persists even with small fractions of synthetic data, in the asymptotic regime.” Note both conditions: the synthetic share stays above zero as the dataset grows, and the result describes where performance ends up as the data gets very large.
What the authors tested, and under what conditions
This section decides how far the headline travels. A theory-plus-experiment paper has two halves that do different jobs.
The theory half proves a result under stated assumptions. Here the abstract says the authors “consider a supervised regression setting”, and the model-size results come from “a simplified regime where neural networks are approximated via random projections of tunable size.” Regression is predicting a number from inputs. It is a long way from a chat model writing an essay, and the authors chose it because it can be analysed exactly.
The experiment half checks whether the picture survives in less tidy conditions. The paper lists toy settings, “random projections model on Gaussian data, and shallow networks fully trained on the MNIST dataset” (MNIST is a standard set of handwritten digits). It also lists what it calls a “realistic setting of GPT-2 models trained on BabiStories”, a set of short synthetic stories. GPT-2 is a small, older language model. The abstract reports that the theory was “empirically verified through experiments on language models and feed-forward neural networks for images.” In the GPT-2 runs the authors report that “even moderate amounts of synthetic data delay the progression of the scaling laws”, and they “expect this to eventually lead to plateaus”. That is an expectation, not a plateau they observed.
So when someone says “the paper shows a little synthetic data ruins models”, ask: which models, trained how, on what data, mixed in what proportion? The answers are in the setup section. Read the conditions first and the headline second.
Does a bigger model help or hurt? What the paper says
The answer depends on the regime. In the simplified random-projections regime, the authors “both theoretically and empirically show that larger models can amplify model collapse.” The abstract then adds: “beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it.”
Roughly, the interpolation threshold is the point where a model has enough capacity to fit its training data exactly. Below it, in their setting, bigger can be worse. Beyond it, bigger may help, but it does not cure the problem. Anyone who quotes only one half of that sentence is giving you half the finding.
A finding about model size still belongs to the paper’s setting. It tells you how the result moves as you turn one dial inside the experiment. It does not tell you what any particular commercial model will do.
What the result cannot tell us about the web
A training result lives inside a pipeline. Someone collects data, someone filters it, someone trains a model. Several assumptions have to hold before strong model collapse describes what is happening with real models trained on the real web:
- Synthetic text reaches training sets in a share that stays fixed as the data grows, as the paper models it.
- That synthetic text behaves the way the paper’s synthetic data does. In the theory, that means data drawn from a distribution that differs from the real one.
- Builders do not filter, label or reweight it in ways that change the result. Commercial pipelines are mostly not public, so this is hard to check.
- The models in question behave like the ones studied, either a regression model or a small model such as GPT-2.
If any of these fails, the result stops transferring cleanly. None of that makes the paper wrong. It makes it narrow, which is what a careful theory paper is meant to be. The stretch happens downstream, when “in a regression setting, a constant fraction of synthetic data stops scaling helping” turns into “AI is eating the internet”. The authors did not write the second sentence.
Training-set collapse versus the web writing alike: two different questions
There are two questions here, and they keep getting run together.
The first: what happens to a model when synthetic data enters its training set? That is where strong model collapse lives.
The second: is the writing actually published on the web becoming more alike? That is a question about the web itself. You answer it by measuring published text, not by training anything.
This court measures the second question: whether published web writing converges. The theory page sets out the claim and how it is tested. You can answer either question and learn little about the other. A model degrading in a lab does not show the web reads alike. A web that reads alike would not show that any model has collapsed.
For the second question measured directly, read the shrinking range, which tests convergence in writing style, and two cases on prevalence, meaning how much of the web is machine-made: the websites nobody wrote and half the articles, and a plateau. Prevalence is not convergence, and each case says which one it measures. Their numbers are on those pages, with their caveats. If the vocabulary itself is the problem, what does AI slop mean sorts out the terms.
How to read a paper like this when you see it cited
You will see this paper in threads and slide decks for a while. Work through it in this order.
- Find the claim in the authors’ words. Use the abstract, and note which version you are reading. Here, arXiv and the ICLR proceedings give different example fractions.
- Find the conditions. What model, what data, what mix, held fixed or growing?
- Separate theory from experiment. What was proved, in what simplified setting, and what did the experiments actually check?
- Read the whole sentence on any qualified finding. “Can amplify” and “may mitigate” are both in this abstract.
- Check the venue. An ICLR acceptance means the paper passed the conference’s peer review. An arXiv posting on its own does not tell you that.
- Ask what the citation is being used for. Is it still about training, or has it quietly become a claim about the web, the future of writing or whether to panic?
If the claim has grown since the abstract, go back and read the abstract again, slowly.
Sources
- Dohmatob, Feng, Subramonian and Kempe: Strong Model Collapse (arXiv 2410.04840, 2024)
- Strong Model Collapse, ICLR 2025 proceedings
- Strong Model Collapse, full text (arXiv HTML, v2)
Questions people ask
What is strong model collapse?
It is a result from 'Strong Model Collapse' by Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian and Julia Kempe (ICLR 2025). In a supervised regression setting, a small fraction of synthetic data in the training set means "larger and larger training sets do not enhance performance."
How is strong model collapse different from model collapse?
Ordinary model collapse, set out in our post on model collapse, is about models trained generation after generation on earlier models' output. The strong version asks something narrower: once some synthetic data is in the mix, can more data still rescue the model? In the paper's setting, more data can help for a while, but the gains slow and performance "eventually plateaus above the baseline". Scale does not remove the error the synthetic share adds.
Does a small amount of AI-generated data really ruin a model?
Within the paper's setting, the authors show that a small fraction can stop performance improving as training sets grow. The example fraction differs by version: the arXiv abstract says "as little as 1% of the total training dataset", and the ICLR proceedings version says "as little as 1 per 1000". Neither is a measurement of any commercial model.
Does making a model bigger fix or worsen collapse?
Both, depending on the regime. In a simplified model, the authors report that "larger models can amplify model collapse", and that beyond the interpolation threshold "larger models may mitigate the collapse, although they do not entirely prevent it."
Does strong model collapse mean the internet's writing is converging?
No. The paper studies what synthetic data does to training. Whether published web writing is growing alike is a separate question, and it can only be answered by measuring published text. That is the question this site's cases test.