Why the intelligence explosion can't happen inside a data centre | Tom Reed

<p><a href="https://jack-clark.net/2026/05/04/import-ai-455-automating-ai-research/">AI systems are starting to build themselves</a>. Because each generation of model will be better at building its successor than the last, it seems plausible that the full automation of AI R&amp;D could rapidly lead to an exponential growth in overall AI capabilities. A natural inference is that domain-general superintelligence arrives shortly after AI research is automated.</p><p>Host Tom Reed does not think this will happen.</p><p>He believes the automation of AI R&amp;D will not rapidly lead to domain-general superintelligence because:</p><ol><li>It’s impossible to get good at most things without practice.</li><li>AI companies lack the data their models would need to practice most things.</li><li>This can’t be fixed with “sample efficiency.” In most cases, the relevant data <em>doesn’t exist at all</em>.</li><li>This also can’t be fixed with simulations or synthetic data.</li><li>This means that the relevant data for superintelligence in most non-coding domains will only become available through deployment of AI models throughout the economy.</li></ol><p>The singularity, therefore, will be bottlenecked on signal. The output of the R&amp;D produced by an isolated data centre of geniuses would be a mere “Goodhart Singularity”:</p><em>Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.</em><p><br>An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimising for eval performance that fails to generalise beyond the lab.</p><p>This suggests that the automation of AI research will not rapidly produce superintelligent capabilities in other domains — their arrival will largely be a function of deployment and data collection in the real world. AI models need real-world deployment for the same reason the body needs pain and corporations need profit: signal is sovereign.</p><p>This essay takes each of the above points in turn.</p><p><a href="https://80k.info/goodhart"><strong>Learn more, video, and full transcript:</strong> https://80k.info/goodhart</a></p><p><em>“The Goodhart Singularity” </em><a href="https://meagreprotestanthistory.substack.com/p/the-goodhart-singularity"><em>originally appeared</em></a><em> on Tom’s Substack in May 2026, and this narration was recorded on August 26, 2026.</em></p><p>Chapters:</p><ul><li>Introduction (00:00:00)</li><li>Practice makes perfect (00:05:05)</li><li>Good data is hard to find (00:08:22)</li><li>Simulation is shallow (00:13:43)</li><li>What a Goodhart Singularity looks like (00:19:04)</li></ul><p><em>Our production team includes:</em></p><ul><li><em>Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour</em></li><li><em>Producers: Elizabeth Cox and Nick Stockton</em></li><li><em>Coordination and support: Katy Moore and Lou Moran</em></li><li><em>Camera operator: Dominic Armstrong</em></li></ul>