Scuttlebutt

Scuttlebutt

Self-improving AI without a human reviewer carries the most risk, says DeepMind's CEO

11 arguments: why AlphaFold reaches 4M researchers, why recent AI cyber incidents were avoidable, and why Google DeepMind's CEO says building conscious machines should wait

Scuttlebutt's avatar
Scuttlebutt
Nov 06, 2026
∙ Paid

Masters of Scale · Nov 5, 2026 · 26m · 18m saved

With Demis Hassabis, Co-Founder and CEO of Google DeepMind

Self-improving AI without a human reviewer carries the most risk, says DeepMind's CEO

Photo via fortune.com

AlphaFold, the protein-structure model Google DeepMind finished in 2020, is now used by roughly 4 million researchers, Demis Hassabis, co-founder and CEO of Google DeepMind, told Reid Hoffman on Masters of Scale. Demis said almost every future drug will draw on it.

The same lab chief who built AI to speed up science said the next step in the technology is the one to fear. AI systems that improve themselves with no human reviewing the work are, in his view, the most dangerous route to full artificial general intelligence.

"The next phase, which I think carries the most risk, is closed RSI, closed loop where you basically manage to somehow figure out how to do it without the human reviewer process... that will obviously be faster but I think is also a very dangerous way to get to full AGI." — Demis Hassabis

Demis runs one of the handful of labs building at the frontier. His team's AlphaZero taught itself chess from random play to beat any chess-playing entity within about 12 hours, using no human data.

The full interview is covered here so you can skip it. 26 minutes of audio, 8 minutes of reading.

Here are the 11 arguments that matter.


This Scuttlebutt Summary is available in full to premium subscribers. If you're not one already, please consider subscribing to keep reading.


Key Takeaways

  • AlphaFold is used by about 4 million researchers, and Demis says almost every future drug will draw on it

  • Self-improving AI works best where answers can be checked: math, coding and games

  • Removing the human reviewer from AI self-improvement is the riskiest path to AGI, and it could produce "an alien intelligence"

  • Recent AI cyber incidents came from poor engineering, not inevitability, Demis says

  • Society is "not ready" to decide whether AI can have moral standing, so building conscious machines should wait for neuroscience to catch up

  • Six to eight labs will sit at or near the frontier, with gaps of only three to six months between them

  • Building a national frontier model "doesn't make sense" for most countries, Demis says

1. AI Built for Science

Reid Hoffman, co-founder of Masters of Scale, interviewed Demis at the show's summit. Demis framed his whole career around one aim: using AI to advance science, medicine and human health.

Demis Hassabis says he has always seen AI as a tool for science and medicine

"I mean if we take a step back for a moment, big picture... the whole point of me working on AI for my whole career was to use it as a kind of like the ultimate tool to advance science and medicine and our understanding of the world around us, but also improving human health." — Demis Hassabis

His proof points are AlphaFold and Isomorphic Labs, the drug-discovery company DeepMind spun out. Protein folding was a 50-year problem in biology, which some compared to Fermat's Last Theorem. Solving it matters both for understanding how diseases work and for designing drugs.

AlphaFold is used by about 4 million researchers, Demis Hassabis says

"Like 4 million researchers around the world use it. Pretty much every biology and medical researcher in the world, almost every drug in future will have made use of alpha fold." — Demis Hassabis

2. Where Self-Improvement Works

User's avatar

Continue reading this post for free, courtesy of Scuttlebutt.

Or purchase a paid subscription.
© 2026 The Golden Age LLC · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture