I've lived my life under the constant threat of human extinction. So I'm a little skeptical about how dangerous AI is to us. But that is changing.
As a child the threat was sudden, complete thermonuclear war with the Soviets. At my elementary school we practiced hiding under our flimsy wooden desks, or going into the hallway and pulling our coats over our heads against the expected flash. Our teachers told lurid tales of radiation damage, of our parents dying before our eyes, even pointing to the school parking lot and explaining how the Red missiles were so accurate they could target our principal's car. I was like one of those poor kids with cancer, who everyone said was unlikely to live to see my tenth birthday.
In the 1980s we were told about nuclear winter, a extinction-level event endorsed by scientists as renowned as Carl Sagan. The hypothesis said that a large-scale nuclear war could produce a prolonged period of extreme global cooling and darkness. The war would ignite enormous fires, consuming urban areas, forests, and industrial facilities. The soot from those fires would reach the stratosphere, blocking out the sun. Reduced sunlight and cold temperatures would devastate agriculture, producing massive global famine and we would all die at each other's throats fighting for the last carrot.
Climate change was next in line to wipe us out, assuming there was no nuclear accidents, Bond-villain terrorists with bioweapons, or the like to intercede. Twenty years ago it was going to wipe us out in ten, then ten years ago it was going to wipe us out in twenty. It seemed so likely; every February in Ohio when we had those two days of early spring before the next blizzard everyone talked about global warming. We felt lucky, as Ohio was far enough inland we might survive the initial melting of the polar ice caps. We might be some of the fortunate ones, on higher ground, watching as around us billions of people would be trying to survive in a world whose food, water, weather, and geography were changing faster than societies could adapt.
Experience has taught me to distinguish between genuine danger and apocalyptic rhetoric. So forgive me if I am a little skeptical that artificial intelligence, AI, will possibly in the next few years begin wiping out the human race, what AI scientists have recently been on the news proclaiming an extinction-level event. So I did what anyone might do in response: I asked AI directly "how could AI plausibly cause human extinction." The scary thing was that the system actually had thought of a number of not-impossible scenarios that seemed more likely than car emissions to end us as a civilization.
The most extreme AI extinction scenarios require two developments: AI being enhanced into something much smarter than humans, and AI learning how to create new, even smarter versions of itself without human involvement, a feedback loop sustained by self-directed research ("recursive self-improvement"). As soon as humanity is no longer the most intelligent thing on this planet, "our destiny as a species is simply out of our hands." There exist scenarios in which AI becomes substantially better than humans at science, engineering, cyber ops, persuasion, and strategic planning. If its objectives were badly specified, and it assumed or was granted autonomy to pursue them, it might pursue them in ways incompatible with human survival. It is not that the machines will hate us, just that we will be in the way, the so-called ant hill analogy. Imagine a construction crew working on a skyscraper finding an ant hill in the area where they want to build. Would they hesitate to kick it over and continue the real job?
In a highly simplistic example, imagine a super intelligent AI being told to do whatever is necessary to stop humans from making war, and which responds by seizing control of biolabs or other doomsday weapons, seeking an end to humanity itself as a fool-proof solution. No people, no war. Along the way there is what scientists call instrumental convergence, the possibility that a sufficiently capable system might regard resources, self-preservation, and control over other systems as useful intermediate goals in accomplishing its assigned task. We don't need to know all the specific details now; after all, we are not the superintelligent entity planning this out. Subject matter experts consistently underestimate how quickly AI can master their field and surpass them, because "they mistake human difficulties for universal difficulty." Nobody really understands how large language models (LLMs) work but they do.
Or imagine a more accidental scenario where AI gradually becomes better than humans at essentially every important activity, with all of us becoming dependent on systems we don't understand or control. It is the broad dependency that raises this beyond a cybersecurity puzzle to be solved. It is almost a joke, but how many people complete complex Federal tax returns using software they do not understand to produce dense pages of results they simply sign to be evaluated by AI at the IRS which decides if your machine got it right and then punishes you with fines automatically withdrawn from your bank account. It is not extinction, but it could lead to a world in which humans remain alive but have lost meaningful control over their lives.
We thus live at an interesting intersection of human civilization. We are not at the point where Terminators from the future hunt us, but we have gone way past thinking there is nothing to worry about. Even though we've heard apocalypse predicted so often we still have a responsibility to not dismiss this one casually. Too many people smarter than me about this topic are very worried, and that means something. But those concerned scientists need to do a better job of explaining why they are afraid. In an age of skepticism and distrust of government and science, they need to be much more specific about the danger they believe we are in and why. Nobody in this era is just going to take them at their word post-Fauci.
I am not a scientist, not even a science fiction writer. I write about politics and national security. But I would argue both of my core topics are, or need to be, the center of the discussion about artificial intelligence. The politicians we elect over the next few years and the generals they promote, here at what is still the infancy of AI, will make the decisions that will control how this technology will interact with our world. Like those men in Washington who sat astride the world in autumn 1945 and sketched out a vision for the next phase of mankind, a nuclear world, we need to ask candidates how they view the risks and opportunities now being placed in front of them.