138 papers audited
2,035 problems found
147 serious enough to change a headline number
69% of papers have at least one
97% of major and moderate errors with a clear direction overstate the paper’s claim
And on the method:
Two independent audits of each paper. One by Claude Opus 5.5, one by OpenAI’s GPT-6 Astra, neither seeing the other’s work. Each obtained the paper, its appendices and its replication data, recomputed the headline numbers from the authors’ own files and, where possible, from the original sources (tax statistics, national accounts, Federal Reserve and survey data), checked that cited sources say what is claimed, and probed every assumption the conclusions rest on.
Then an adjudicator. A third agent re-verified every claimed problem, tried to refute each one, discarded those that did not survive, searched for problems both auditors missed, and graded what remained by how far it undermines the paper’s own headline claim: major changes a core number materially, moderate affects secondary results or a core result modestly, minor is a presentation or arithmetic slip. Confidence is graded separately.
There is much, much more at the link, which goes through the research paper by paper, and with further sustained analyses behind the links. The survey is both extensive and thorough. (And I thank an anonymous MR reader for this work.)
I am not even sure what to say at this point. We need a radical revaluation of this work, and of the associated policy arguments. More generally, this is only a single step in the story of how AI is going to revise economic science, and indeed past research more generally. As I stress in many of my recent talks, there will be significant gains and losses in status, with the losses often outweighing the gains. I see this as one of those cases.
