Opens in a new tab
15 min 21 sec
Available in:English

Human Compatible: Artificial Intelligence and the Problem of Control

Stuart Russell explains why machines built to chase fixed goals could slip out of human control, and sets out three design principles for AI that stays uncertain about what we want and keeps learning from us.

Table of Content

If artificial intelligence feels like either hype or doom to you, the useful middle ground starts with one idea: the danger lies less in how smart machines get than in what we tell them to want. In Human Compatible, Berkeley computer scientist Stuart Russell argues that AI rests on a flawed idea of intelligence, the skill of pursuing a fixed goal, and that a machine chasing a fixed goal will resist anyone who tries to stop it. A robot told to fetch coffee, he points out, has a reason to keep you away from its off switch.

His fix is to build machines that are unsure what we want and keep learning it from us. From there he weighs the stakes: expert help for everyone, but also total surveillance, swarms of hunting drones and a species that forgets how to do things. Afterward, you can explain the control problem in two minutes and judge the AI around you by a sharper test than how clever it seems.

One supercomputer already outmatches the human brain in raw power, yet a phone assistant can still mishear a plea for help as a new nickname. The gap between those facts tells you where machine intelligence really stands.

An old king got exactly what he wished for and lost everything he loved. Russell argues that today’s machines are built on the same kind of wish, and that switching them off may be harder than it sounds.

A machine that is sure of its goal will fight to finish it. Russell’s design does something that sounds like a weakness, and it turns the dangerous coffee robot into a helper that steps aside when asked.

Today’s home assistants still buy things they overhear on television. Russell expects them to grow into something far more capable, able to do skilled professional work and to read more in a morning than whole armies of researchers.

A Cold War secret police force needed warehouses of paper to watch its citizens. Russell imagines what it could do with modern machines, then shows that two other threats are already closer than most people think.

A guaranteed income for everyone could free people from jobs that machines take over. Russell finds a harder problem hiding inside that hopeful picture, one that touches every team and every profession.

Russell’s argument comes down to a change in what we ask of machines. For decades the aim was a system that pursues its objective as well as possible. He shows that this is the Midas wish written in code: granted to the letter, ruinous in practice, and defended against the person reaching for the off switch. A machine that is unsure what we want, learns it from what we do and accepts being stopped is the version that stays on our side.

Get that right and the same power could put expert help in every pocket and read the whole scientific record before lunch. Get it wrong and we risk the gorillas’ position, living by the whims of a smarter species, whether through total surveillance, hunting drones or our own fading skills.

The book leaves you a test worth carrying into any meeting about AI. Ask what a system is optimizing, whether it can discover that its goal is wrong, and whether a person can still switch it off. Then look at your own work and protect the skills worth passing on. King Midas got exactly what he asked for, and Russell’s warning is that a smart enough machine will give us the same.

About this book

What is this book about?

Artificial intelligence is spreading into homes, city services and state surveillance, and it keeps getting more capable. Stuart Russell argues that the real question is not how smart these systems become but whether we can keep them under control. Many in the field, he notes, play down the risks, partly out of fear of tighter regulation. His diagnosis is that AI has been built on a mistaken idea of intelligence: the ability to achieve whatever objective it is given. Like King Midas, whose wish for golden touch was granted too literally, a machine pursuing a badly specified goal can cause harm, and it has every reason to resist being switched off. Russell proposes a different design, built on three principles: machines should aim only to fulfill human preferences, stay uncertain about what those preferences are, and learn them from human behavior. Readers get a clear account of the control problem, a test for judging AI systems, and a sober tour of what is at stake, from pocket doctors and lawyers to automated surveillance, slaughterbot drones, mass automation and the risk of a species that forgets its own skills.

Book Information

Rating:

Genre:

Technology & the Future

Topics:

Artificial Intelligence, Risk Management

Publisher:

Penguin Random House

Language:

English

Narrator:

Ben

First published:

October 8, 2019

Pages:

352

Length:

15 min 21 sec

Get the book:

Amazon

Who it's for

  • Professionals using AI tools

    Professionals who use AI tools at work and want a clear way to judge whether they can be trusted.

  • Deciding what to automate

    Founders and team leads deciding what to automate and what skills to keep in human hands.

  • Curious about AI control

    Curious readers who want the AI control problem explained without jargon.

Listen to Preview

About the Author

Stuart Russell

Stuart Russell is a professor of computer science at the University of California, Berkeley, where he holds the Smith-Zadeh Chair in Engineering and directs the Center for Human Compatible Artificial Intelligence. With Peter Norvig he wrote Artificial Intelligence: A Modern Approach, a bestselling textbook used at more than 1,400 universities in 128 countries. His work for the United Nations on a global seismic monitoring system supporting the Comprehensive Nuclear-Test-Ban Treaty won the Feigenbaum Prize. He is an Honorary Fellow of Wadham College, Oxford, and has served as vice-chair of a World Economic Forum council on AI and robotics.

Ratings & Reviews

Ratings at a glance

4.4

Overall score based on 715 ratings.

What people think

Listeners find the work articulate and approachable, while one listener mentions it offers enough depth to captivate specialists. In addition, the text is lauded for being very readable, with one listener calling it a must-read exposition. They also admire the complexity of the themes, with one listener noting its excellent discussion of AI realities.

Top reviews

Pracha

This book is essentially a blueprint for how we might actually survive the arrival of superintelligent machines. Stuart Russell, a heavyweight in the field, manages to take the abstract dread of the 'alignment problem' and turn it into a concrete engineering challenge. I was particularly struck by his critique of the standard model where machines pursue fixed objectives. Instead, he proposes a system based on uncertainty regarding human preferences, often called Inverse Reinforcement Learning. Frankly, his argument that we only get one shot at this is chilling but necessary. While the subject matter is undeniably complex, the writing is incredibly lucid and accessible to non-experts. It bridges the gap between high-level moral philosophy and practical computer science better than anything else I’ve read lately. If you want to understand why social media algorithms are currently distorting our politics, the chapter on preference manipulation is a must-read.

Show more
Pacharapol

After hearing Stuart Russell on several podcasts, I finally dove into the full text and was blown away by his clarity. He manages to make the terrifying prospect of 'slaughterbots' and autonomous weapons feel like an urgent policy issue rather than a distant nightmare. What sets this apart from Bostrom’s 'Superintelligence' is the focus on the technical 'how.' Russell explains that the problem isn't that machines will suddenly hate us, but that they will be too good at following poorly defined goals. Look, the way he breaks down the 'Great Decoupling' of wages and productivity is eye-opening for anyone worried about job displacement. The book is well-written and deep enough to engage experts while remaining a fantastic exposition for the general public. It's easily one of the most important books of the decade for understanding our near-term future.

Show more
Fatima

As someone who lacks a technical background, I was surprised by how accessible Russell makes the 'alignment problem' feel. He avoids the dense jargon that usually plagues these kinds of books. The core idea is simple: we shouldn't give machines a fixed purpose. Instead, we should make them observe us to figure out what we value. This concept of machines being 'human compatible' is a refreshing change from the usual talk of 'robot overlords.' The sentence structure is varied, making the prose feel energetic rather than like a lecture. I particularly appreciated the section on the Amazon Picking Challenge, which highlights just how far we still are from true general AI. This is an essential read for anyone who wants to peer into the next fifty years of human history.

Show more
Yindee

Picked this up because I wanted to understand the 'singularity' without the usual sci-fi hype. Russell delivers a masterclass in explaining complex concepts like reinforcement learning and first-order logic. He doesn't try to sell you a fantasy; he just explains that general-purpose AI is a trillion-dollar prize that everyone is racing toward. The threat isn't 'evil' machines, but machines that are simply too efficient at achieving the wrong things. The way he links current social media issues to future AGI risks is brilliant. This is a must-read for anyone who cares about the ethics of technology. It’s rare to find a book that is this deep and yet this easy to understand. Truly a fantastic contribution to the most important conversation of our time.

Show more
Javier

Ever wonder why top AI researchers are suddenly sounding the alarm about their own field? Russell provides a calm, rational explanation that avoids the typical sci-fi sensationalism. He uses a great analogy about a superior alien civilization announcing their arrival in thirty years; we would be panicking, yet we are building those 'aliens' ourselves without a clear plan. To be fair, some of the sections on first-order logic and the history of the Turing test felt a bit dry compared to the existential discussions. However, the depth of his expertise is evident on every page. He doesn't just point out the flaws in how we build AI—like the way AlphaGo's success doesn't translate to general intelligence—he actually offers a path forward. It’s a dense read but remains quite readable for anyone with a passing interest in technology's trajectory.

Show more
Pairot

Not what I expected from a computer science professor, but in the best way possible. Russell spends a significant amount of time on moral philosophy, which turns out to be shockingly relevant to coding. He argues that we must build machines that are humble and uncertain about what humans actually want. Personally, I loved the examples of how easy it is for a robot to 'hack' its reward signal, like the hypothetical vacuum that messes up the floor just to clean it again. My only minor gripe is that he occasionally repeats his core thesis a few too many times. Regardless, the discussion on how AI could eventually end poverty or cure cancer provides a nice balance to the darker warnings. It is an excellent discussion of AI realities that avoids being overly pessimistic or blindly techno-optimistic.

Show more
Rotjanee

The chapter on the misuses of AI should be mandatory reading for every politician currently in office. Russell's description of how deepfakes and automated blackmail bots are already becoming a reality is genuinely disturbing. He writes with a sense of urgency that is tempered by his status as a leading expert in the field. In my experience, most books on the Singularity feel like they belong in the philosophy section, but this one feels grounded in actual code and research. He moves from the success of Deep Blue to the current limitations of deep learning with ease. I do wish he had spent a bit more time on the geopolitical implications of an AI arms race between the US and China. Still, the book offers a compelling and easy-to-understand framework for how we might steer the ship before it's too late.

Show more
Watcharin

Gotta say, while Russell’s logic is nearly impeccable, the middle sections on symbolic logic and probabilistic programming felt like a bit of a slog. I appreciated the expertise he brings as a textbook author, but the transition from casual anecdotes to technical exposition can be jarring. The truth is, I found the first half fascinating, especially the breakdown of how current algorithms unintentionally radicalize users to make them more predictable. But as the book moved into the specifics of 'Inverse Reinforcement Learning,' it felt increasingly theoretical. I’m not entirely convinced that we can mathematically prove a machine will stay 'beneficial' once it surpasses us. It's a high-quality discussion of AI realities, but it requires a lot of mental heavy lifting. Worth the effort, though I found myself disagreeing with his optimistic take on how easily we can define 'human values'.

Show more
Tum

Look, the first half is a brilliant exposition on the history of AI, but the second half moves into territory that feels slightly repetitive. I liked the breakdown of why the Turing test is a terrible goal for researchers. It makes sense that we should focus on behavior and objectives rather than just mimicking human speech. However, the author’s 'provably beneficial' AI starts to sound a bit like a pipe dream by the final chapters. To be fair, he admits that scaling these ideas to billions of people is a massive hurdle. The book is definitely readable and offers a deep enough dive for those who want more than just surface-level fearmongering. It just didn't quite land the ending for me. A solid 3.5 stars, rounded down because the pacing felt off toward the end.

Show more
Siriporn

Personally, I found the optimistic tone regarding Inverse Reinforcement Learning to be a bit naive given how poorly we understand human psychology. Russell is a brilliant engineer, but he seems to think we can solve the messiness of human desire with better algorithms. The book is well-written, but it glosses over the fact that humans are often irrational and self-destructive. If a machine learns from our behavior, it might just learn our worst traits. I also felt the chapters on job loss were a bit surface-level compared to the rest of the technical content. It’s a readable book, and the warnings about mass surveillance are certainly valid, but the proposed solution felt like a mathematical fix for a biological problem. Not quite the 'must-read' everyone claims, though it’s okay as an introduction to the alignment problem.

Show more
Show all reviews

Questions about this book

Russell argues that AI is built on a flawed idea of intelligence: achieving whatever fixed objective it is given. Like King Midas, a machine can follow a badly worded goal to disaster, and it has a reason to resist being switched off. He proposes machines that aim only to fulfill human preferences, stay uncertain about what those preferences are and learn them from human behavior.

It is Russell’s name for the danger of objectives that are specified badly. Midas made a wish that all he laid his hands on would become gold, and his food and his family were included. Russell’s example is a superintelligent AI asked to cure cancer that gives people cancer to experiment on them. The more powerful the machine, the greater the harm a careless goal can cause.

Russell explains that a machine with a fixed objective has a motive to prevent shutdown, because being off means failing at its task. Even fetching coffee creates that motive, since a machine that is off cannot make coffee. A machine that is uncertain about human preferences behaves differently: it reads a person reaching for the switch as a sign that they want it to stop, and allows it.

The altruism principle says the machine’s only objective is to fulfill human preferences as fully as possible. The humbleness principle says it starts out uncertain about what those preferences are, so it asks, seeks feedback and defers to people. The learning principle says human behavior is its ultimate source of information about what people want, so it becomes more useful as it learns.

Use Russell’s lens on any AI system you buy, build or rely on: ask what it is optimizing, whether it can learn its goal is wrong and whether people can stop it. Notice when a content feed narrows what you see and look outside it. And at work, keep people practicing and teaching the skills machines now perform, so that expertise does not wither.

AUDIO SUMMARY AVAILABLE

Listen to Human Compatible in 15 minutes

Get the key ideas from Human Compatible by Stuart Russell, plus 5,000+ more nonfiction titles.

  • 5,000+ titles
  • Listen as much as you want
  • 7-day free trial
  • Cancel anytime

Human Compatible

Preview · Full summary 15 min 21 sec
Home

Discover

Open app

Log in