In 2023 the heads of OpenAI and Google DeepMind signed a warning that ranked AI's extinction risk alongside pandemics and nuclear war, yet in a 2025 Pew survey half of Americans said AI will worsen people's ability to form meaningful relationships, and only 5% expected it to improve

On May 30, 2023, the Center for AI Safety released a statement only one sentence long: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Among those who signed it were Sam Altman, chief executive of OpenAI, Demis Hassabis of Google DeepMind and Dario Amodei, as well as Geoffrey Hinton and Yoshua Bengio, two of the field's best-known researchers. The statement gave no estimate of how likely such an outcome is. It simply put the question of human survival on the table.

Two years later, the Pew Research Center asked ordinary Americans what they thought AI would do to their lives, and their answers sounded much less like science fiction. In a survey of 5,023 US adults taken in June 2025, 50 percent said AI will worsen people's ability to form meaningful relationships. Only 5 percent expected it to improve.

Those two documents point to two quite different ideas of what makes a technology morally risky. One concerns a possible catastrophe. The other concerns the slow wearing away of the ordinary things a good life is made of. Philosophers have been arguing about both for a long time, and their arguments are a useful guide to the question of whether developing AI puts human society at risk.

A worry about Tuesday afternoon, not the end of the world

The Pew figures go beyond relationships. Half of those surveyed said they were more concerned than excited about the growing use of AI in daily life, up from 37 percent in 2021. Only 10 percent were more excited than concerned. Fifty-three percent thought AI would make people worse at thinking creatively, against 16 percent who thought it would make them better. And about six in ten said they would like more control over how AI is used in their own lives.

A survey cannot tell anyone whether those fears are correct. What it records is where people feel the risk: in friendship, in creative thinking and in control over their own lives. These are close to the goods that ancient ethics put at the center of a life that goes well. Aristotle gave two of the ten books of his Nicomachean Ethics to friendship. Seen that way, the Pew respondents are raising a moral question as much as a practical one: what kind of lives a technology helps people lead.

Wiener's warning about the purpose put into the machine

The worry that machines might serve the wrong ends is older than the computer industry. In 1960 the mathematician Norbert Wiener, who founded the field he called cybernetics, published an essay in the journal Science titled "Some Moral and Technical Consequences of Automation." If we hand our purposes to a machine whose action is so fast and irrevocable that we cannot step in before it is complete, he wrote, "we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it."

Wiener illustrated the point with old stories: the sorcerer's apprentice, whose enchanted broom keeps carrying water until it nearly drowns him, and W. W. Jacobs' tale "The Monkey's Paw," in which every wish is granted in a way that brings disaster. The moral of both is the same. A wish taken literally can be worse than no wish at all. Wiener's question was less about evil machines than about careless people, who might give a powerful system a goal that sounds right but leaves out what they actually care about.

Weizenbaum's secretary and the limits of calculation

A few years later, Joseph Weizenbaum discovered how easily people confide in a machine. Between 1964 and 1966 he wrote ELIZA, one of the first chatbots, at MIT. It worked by turning a user's statements back into simple questions. His own secretary, who had watched him build the program and knew how it worked, asked him to leave the room so she could talk to it in private.

The experience helped turn him into one of computing's sharpest critics. In his 1976 book Computer Power and Human Reason he drew a line between deciding, which is a kind of calculation a computer can be programmed to carry out, and choosing, which comes from human judgment and values. Since no one knows how to make computers wise, he argued, "we ought not now to give computers tasks that demand wisdom." His concern was not that machines would rise up. It was that people would quietly hand over the parts of life that need care and judgment, simply because a machine could produce an answer.

The everyday costs of progress

Some costs of AI can be measured. The International Energy Agency's 2025 report Energy and AI estimates that data centers used about 415 terawatt-hours of electricity in 2024, around 1.5 percent of the world's total. It projects that figure will more than double to around 945 terawatt-hours by 2030, slightly more than Japan's entire electricity consumption today, with AI as the most important driver of the growth. A typical AI-focused data center, the agency notes, uses as much electricity as 100,000 households. These are projections, and they depend on how quickly the technology spreads, but they show that the machines draw on the same power grids that supply homes, hospitals and factories.

In January 2025 the Vatican published a note on AI, Antiqua et Nova, that made similar points in moral language. It warned that current models "consume vast amounts of energy and water," that they can "deskill workers" and subject them to automated surveillance, and that they risk leading to "harmful isolation." It described AI as a product of human intelligence rather than an artificial form of it, a tool to complement people rather than replace them.

The hope on the other side of the ledger is old as well. In Book I of the Politics, Aristotle imagined a world where "the shuttle would weave and the plectrum touch the lyre without a hand to guide them," and concluded that then "chief workmen would not want servants, nor masters slaves" (in Benjamin Jowett's translation). In 1930 the economist John Maynard Keynes, in his essay "Economic Possibilities for our Grandchildren," guessed that technical progress might solve "the economic problem" within a hundred years, so that "three-hour shifts or a fifteen-hour week" could share out the work that remained. The real problem, he thought, would then be how to use that freedom "to live wisely and agreeably and well."

For my own part, this is where my concern lies. I am concerned about the balance between the needs of people on a daily basis and the advancement of technology.

Jonas and a duty to the future

The philosopher Hans Jonas gave that kind of concern a formal shape. In The Imperative of Responsibility (published in German in 1979 and in English in 1984), he argued that older ethics dealt mostly with the near consequences of what people did to one another, while modern technology can change the world for generations. "The gap between the ability to foretell and the power to act creates a novel moral problem," he wrote. His answer was a new imperative: "Act so that the effects of your action are compatible with the permanence of genuine human life." Where the outcome is uncertain, he went further, "the prophecy of doom is to be given greater heed than the prophecy of bliss."

Jonas wrote with nuclear power and genetic engineering in mind, but his phrase "genuine human life" fits the AI debate closely. It asks not only whether people survive a technology, but whether the life they are left with is still recognizably human: one with work that means something, friends who know them and decisions they make for themselves.

So is AI a moral risk?

Read together, these thinkers suggest the answer depends on which risk is meant. The signatories of the 2023 statement put the most extreme possibility first. Wiener, Weizenbaum and Jonas point to something nearer and more ordinary: people handing their goals and their judgment to systems they do not fully control, and paying for them in energy, skills and companionship. The Pew survey suggests that is where many people already feel the danger.

Keynes, for one, expected machines to make room for a better life. What these thinkers asked is whether the purpose put into the machine is the one people really want, and whether anyone is checking. That question is no longer only for engineers. It belongs to everyone who has to live with the answer.