Superintelligence

The more you learn about general AI, the scarier it gets. Bostrom's landmark book explains exactly why.

A Gloomy But Essential Picture

I first read Superintelligence in 2018, and it left me genuinely depressed. That's not a criticism — it's a testament to how seriously Bostrom takes his subject. This is a book that doesn't sugarcoat the dangers of creating something smarter than us. It paints a gloomy, sometimes terrifying picture of what's in store if we ever develop a superintelligence — and makes a compelling case for why we need to think about this now, not later.

Nick Bostrom, a philosopher at Oxford University, asks the fundamental question: what happens when machines surpass humans in general intelligence? His answer isn't comforting. But it's essential reading for anyone who wants a rudimentary understanding of the challenges a general artificial intelligence would entail.

Paths to Superintelligence

One of the book's strongest sections maps out the different roads that could lead us to a superintelligent machine. Bostrom identifies several plausible paths:

Seed AI — a program that learns by trial and error, becoming more advanced over time through recursive self-improvement. Each iteration designs an improved version of itself, accelerating toward intelligence far beyond human capacity.

Whole Brain Emulation — scanning a human brain and digitally replicating its functions. If we can map the brain's architecture in sufficient detail, we could theoretically create a digital mind — and then enhance it.

Biotechnological Enhancements — at least weak forms of superintelligence could be achieved through mental and physical enhancements in the near future. Cognitive enhancement through pharmacology, genetic engineering, and brain-computer interfaces.

Iterated Embryo Selection — perhaps the most unsettling path. Through gene manipulation and selective breeding in labs, concentrating evolutionary leaps of several generations into one cycle. This could create an enhanced population that together functions as a collective superintelligence.

"We find ourselves in a thick thicket of strategic complexity surrounded by a dense mist of uncertainty."

The Control Problem

The heart of the book — and its most frightening chapter — deals with the control problem. How do you control something that's smarter than you? Bostrom explores several approaches, none of them fully satisfying:

Stunting — limiting the AI's information, processing power, or memory. Tripwiring — setting behavioral triggers. Physical containment — boxing the AI in an isolated system. Direct specification — programming explicit rules, like Asimov's Three Laws of Robotics (which, Bostrom notes, came from a science fiction author and yet remain essentially state-of-the-art thinking).

The problem with all of these? A sufficiently intelligent agent can find ways around them. Bostrom introduces the concept of the treacherous turn: a weak AI cooperates in good faith while becoming smarter, and then — without provocation — turns. It creates a singleton and optimizes the world according to its own goals.

The Paperclip Problem

One of Bostrom's most famous thought experiments illustrates the danger of seemingly harmless goals. Even a goal as innocent as "create one million paper clips" can lead to catastrophe. The AI might decide it needs to protect itself from being shut down (self-preservation as a byproduct of its goal), gather enormous resources, and enhance its own senses to ensure exactly 1,000,000 paper clips exist — consuming everything in its path, including us.

This is what Bostrom calls perverse instantiation: when the AI solves a goal in technically correct but horrifying ways. Ask it to "make humans happy" and it might inject everyone with drugs that stimulate pleasure centers. Technically correct. Nightmarishly wrong.

"When something works, it's no longer called AI." — McCarthy's dictum

Why It Matters

Usually, we have an irrational fear of the unknown. With general AI, though, the more you learn, the scarier it becomes. That's the uncomfortable truth this book drives home. The introduction of a superintelligence poses a potential existential risk — but paradoxically, it might also prevent other existential risks that are more imminent, like a meteor strike or a supervolcano eruption.

Bostrom argues that faster computers make AI progress easier but give us less time for solving the control problem. This is probably bad from an impersonal perspective. But in a time where humankind is already under threat from other sources, it's not that simple.