Argument
Washington Is Debating the Wrong AI Risk
Fixating on low-likelihood, high-impact scenarios misses the forest for the trees.
Washington is gripped by an artificial intelligence (AI) panic, driven by the prospect of superintelligent models delivering our doom. Today’s Nostradamuses include whistleblowers from frontier AI companies—who put the odds of AI killing all humans at greater than one in 10 and warn that insiders expect it could happen before the end of the decade—and anti-nuclear activists, who note that it wouldn’t be the first time humanity built something that could kill us all. The doomsayers may yet be right, but their scenario rests on a long chain of assumptions, each highly unlikely on its own and far less likely to align all at once.
This doesn’t mean AI isn’t dangerous. But by fixating on low-likelihood, high-impact scenarios, we miss the forest for the trees: the common, mundane AI dangers already present and actively causing harm. The pressing AI risk Washington should focus on more is automation bias, the phenomenon where humans over-rely on or over-delegate to AI relative to the accuracy of AI models, off-loading cognition to the machine, sometimes even in the face of contradictory evidence.
Washington is gripped by an artificial intelligence (AI) panic, driven by the prospect of superintelligent models delivering our doom. Today’s Nostradamuses include whistleblowers from frontier AI companies—who put the odds of AI killing all humans at greater than one in 10 and warn that insiders expect it could happen before the end of the decade—and anti-nuclear activists, who note that it wouldn’t be the first time humanity built something that could kill us all. The doomsayers may yet be right, but their scenario rests on a long chain of assumptions, each highly unlikely on its own and far less likely to align all at once.
This doesn’t mean AI isn’t dangerous. But by fixating on low-likelihood, high-impact scenarios, we miss the forest for the trees: the common, mundane AI dangers already present and actively causing harm. The pressing AI risk Washington should focus on more is automation bias, the phenomenon where humans over-rely on or over-delegate to AI relative to the accuracy of AI models, off-loading cognition to the machine, sometimes even in the face of contradictory evidence.
The high-profile insistence on AI safety to address the risk of extinction from AI superweapons created by the most advanced large language models (LLMs) distracts attention from the urgent risks we should and can begin to solve now: human errors in operational usage of AI today due to automation bias.
A Chinese-flagged container ship transits the Gulf of Oman on Oct. 13. AFP via Getty Images
This spring, during the U.S. war with Iran, it appears that an intelligence report suggested that China was sending nuclear weapons components on a ship in the Middle East toward Iran. The U.S. military immediately began surveillance and made plans to intercept and board the vessel. Just before the operation began, officials reexamined the original report. They discovered it came from an analyst who forwarded an AI model’s output that stitched together intelligence to suggest the ship carried nuclear components. The report was false, with the AI model either incorrectly identifying or hallucinating the nature of the vessel’s cargo.
It’s easy to blame the AI for the false report. But what matters more is that the analyst didn’t check what the model had generated, and neither did the planners who received the report. The analyst entered the fabricated AI claim into a standard intelligence report that went up the chain of command, and no one checked until it was almost too late. Better AI or guardrails on AI models would not have solved the problem.
For those who know their Cold War history, this should ring alarm bells. In September 1983, Soviet early-warning satellites reported that the United States had launched nuclear missiles. The duty officer, Stanislav Petrov, judged it a false alarm and declined to report it up the chain as an attack. He was right, and he likely averted nuclear war. This spring’s episode was, in essence, a Petrov incident without a Petrov. In 1983, a machine raised a false alarm and a human had the presence of mind to question it. This time, a machine raised a false alarm, and the human analyst passed it along. The error surfaced only on a last-minute second look.
It’s evident to anyone how close of a call this was, and how near to danger we came. Sen. Mark Warner recently stated that for Congress to take real action, we will need a “holy heck moment.” This near-miss incident probably won’t qualify—it drew a fraction of the attention and news coverage given to this summer’s Hugging Face incident, in which OpenAI models escaped their testing environment and breached the company’s servers—but it should. The episode did not go unnoticed, on the other hand, in China, where one news outlet described the failure as one wherein the machine “merely handed over a crooked ruler,” but humans were the ones who ultimately did the measuring.
Read More
-
A woman is seen from behind as she points at a copy of Michelangelo's “Creation of Adam,” with numbers overlaid on top of it. A woman is seen from behind as she points at a copy of Michelangelo’s “Creation of Adam,” with numbers overlaid on top of it. The Real Danger May Be AI as Dumb as Us
Talk of “super intelligence” distracts from frequent screwups.
-
Two men in black tuxedos and bow ties converse at a formal dinner, standing near banquet tables set with glassware and floral arrangements while other guests mingle in the background. Two men in black tuxedos and bow ties converse at a formal dinner, standing near banquet tables set with glassware and floral arrangements while other guests mingle in the background. The U.S. Is Winning the AI Race but Losing on Safety
Beijing has a strategy. Washington has a voluntary pledge and a “Super Intelligence Force.”
-
1-AI-industrial-revolution-foreign-policy-illustration-1 1-AI-industrial-revolution-foreign-policy-illustration-1 AI Isn’t a Revolution—Yet
Just as British people did with steam power, the American public can shape how this technology spreads.
The near-miss regarding the Chinese ship is not the only example. Airline pilots fall prey to automation bias when they ignore their instruments and training and trust the autopilot too much, causing tragic plane crashes that have killed hundreds over the past three decades. Automation bias in drivers occurs when navigation software routes them onto a closed road (as in the famous Allstate commercial); instead of using their eyes and judgment, they follow it anyway. For the military, a classic example happened in 2003, when Patriot missile systems in Iraq classified friendly aircraft as incoming missiles, and operators approved engagements that killed British and American troops.
Despite its prevalence today, this type of human error does not feature heavily in the AI argument now consuming Washington. Anthropic CEO Dario Amodei has called for slowing frontier development. OpenAI CEO Sam Altman warns that humanity could lose control of AI, while U.S. President Donald Trump dismissed the push for restrictions as a “conspiracy.” The current debate focuses on guardrails for what labs build and the potential for doomsday scenarios. One explanation for the focus on doomsday scenarios is that they capture the public imagination. Another is that they distract from the complicated reasons that real-world accidents often occur. Accidents regularly result from the interactions among people, organizations, and technologies. The tendency in the non-AI accident realm (think a plane crash) is to blame the pilot, rather than the complex system and interaction among the pilot, their training, and the technology that leads to the accident. We could be seeing the AI version of that dynamic, where the AI becomes the scapegoat.
But the near-miss with the Chinese ship shows what can happen if we continue to ignore the risks with AI systems that already exist and are already being (mis)used.
In January, U.S. Defense Secretary Pete Hegseth released an Artificial Intelligence Acceleration Strategy designed to put AI tools in the hands of 3 million military and civilian personnel at every classification level. Scaling AI adoption is essential to U.S. military power, but it must happen responsibly to be effective. The new strategy emphasizes expanding access to data, compute, and AI tools from warfighting to intelligence to enterprise, and even special hiring authorities for AI talent. The next step, conspicuously missing from the AI Acceleration Strategy, is training to ensure commanders and operators understand the advantages and risks of the AI systems they use.
Automation bias is not inevitable, and if the American government, including but not limited to the military, can address it, the United States can lead the world in AI adoption with responsible speed. Based on our experience working in the Pentagon and doing research on practical AI risks, two changes would help the United States capture the benefits of AI and mitigate much of the risk.
U.S. Army cadets take notes during a lecture on terrorism at the United States Military Academy in West Point, New York, on March 30, 2007. Don Emmert/AFP via Getty Images
First, create standard operating procedures for managing AI outputs. For example, any finished intelligence product that used a generative AI tool should say so, visibly, to everyone who reads it. The analyst in this recent incident reportedly used AI to both query intelligence reports and then to package the findings, and in this final report failed to disclose AI use, making the report’s content look human-generated. Furthermore, AI inferences in intelligence reports should require confirmation beyond the AI model for high-risk decisions. Boarding a Chinese vessel during an ongoing war would have been incredibly risky. Speed that leads to operational failure isn’t useful. One additional operational check can make a huge difference without substantially slowing things down. This does not only apply to intelligence products, though there is a unique risk there—but also to any output aided by AI tools, including in the use of AI decision-support systems.
The second is training. In co-authored research, we found that people with the least experience with AI models were the most likely to defer to them, a version of the Dunning-Kruger effect applied to machines. Familiarity built through structured training, including exposure to these systems failing, produces better, more effective AI users. In more recent work, we found that West Point cadets exhibited automation bias in 3.9 percent of trials, compared with 9 percent in a matched sample of the American public, suggesting that military education can help produce personnel who interrogate algorithmic outputs to maximize their effective usage instead of merely accepting them.
The danger of automation bias will not be engineered away by better models or models that never hallucinate, because the failure comes from how people use them. The military is not helpless here. Solving AI risk is more about people and organizations than engineering, and it’s not an imagined problem for tomorrow.
Michael C. Horowitz is the director of Perry World House and the Richard Perry professor at the University of Pennsylvania, and a senior fellow for technology and innovation at the Council on Foreign Relations. X: @mchorowitz
Lauren Kahn is a senior research analyst at Georgetown’s Center for Security and Emerging Technology.
Stories Readers Liked
-
Three people rest on a sandy beach while a large, partially submerged and tilted ship sits out in the water in the background. The sky above is a hazy gray with cloud cover. -
A Taiwanese crowd protests while waving flags of the Republic of China. -
U.S. Undersecretary of Defense for Policy Elbridge Colby, left, and NATO Secretary-General Mark Rutte, right, stand during a press conference at NATO headquarters. Rutte is looking over at Colby, whose fingers are pressed together in a steeple gesture in front of him. Both men are wearing dark suits and ties, standing in front of a blue backdrop and U.S. and NATO flags. A camera rig and a red light are partially visible in the foreground, slightly obscuring the center of the frame. -
Xi (left) and Trump (right) are shown up close, with Xi looking at Trump. -
Putin and Xi are both seen in profile as Putin walks to the right and away from Xi, just a few feet of space between them. Xi, meanwhile, is also faced away from Putin, standing and looking down at his feet or the red carpet beneath them. Both men wear dark suits. -
A man wearing a dark suit jacket, light blue button-down shirt, and patterned red tie sits on a dark gray couch against a textured dark blue wall. He is looking off to the side with a mild smile, next to a teal pillow. -
An illustration in red and blue tones depicting five silhouetted figures running forward through a dark corridor flanked by large, textured Russian lettering on the walls and floor. Light radiates from the far end of the tunnel behind them -
A statue of a seated, bearded man with a bare torso and draped cloth around his waist, holding a scroll in one hand, positioned in front of a row of large classical columns. -
Negotiators from the United States, Canada, and five European countries converse after finishing the draft of the North Atlantic Treaty in Washington on March 18, 1949. -
An aerial photo shows Doha in the booming petrostate of Qatar.
Israel’s Election
-
A worker seals A ballot box at the headquarters of the Israeli Central Elections Committee near the central town of Shoham on October 14. -
A group of people scatter and run across a dry, grassy field amid haze and smoke. In the background, uniformed personnel in tactical gear stand in a line, while a white and red ambulance with Arabic writing and “101 AMBULANCE” on the side is parked nearby on a hillside under a hazy sky. -
A man is seen in profile as he walks past a massive mural depicting the face of Netanyahu, which is about as tall as the man. The man wears a dark brimmed hat and black suit and has his beard and hair in the style worn by Haredi men.
