Of That Day and Hour

People who type will AI end the world? or will AI destroy humanity? into a search box are usually asking two things at once. One is a question about the future: is this going to happen? The other is a question about the people raising it: should I believe them? The first question has no answer yet. The second one has, and it is the more useful one to work through, because the public argument mixes three different kinds of statement. There is a claim about priorities, signed by a long list of named people. There is a set of estimates, collected by survey and published with their own caveats. And there is a belief, held with conviction on both sides, about what those estimates are worth.

This essay takes them one at a time. It quotes the 2023 Statement on AI Risk word for word, reports what the largest survey of AI researchers found in the words and figures of the paper itself, and gives the strongest sceptical reply in its authors’ own words. Then it asks which parts are forecasts that the world could one day check, and which parts behave more like an eschatology, a teaching about the end. It gives no probability of its own. Nobody writing this has a method that would make such a number worth more than a mood, and that point turns out to be a large part of the argument.

How the sources were handled. Every quotation below was copied from a page opened on 30 September 2026 and is recorded, with its address and the exact sentence, in this site’s source ledger. Where I say what a text implies, that is my reading and is marked as mine.

The claim: one sentence, and what it leaves out

In May 2023 the Center for AI Safety published a statement one sentence long. The page it lives on is now titled Statement on AI Extinction Risk; its original address, ending statement-on-ai-risk, redirects there. The press coverage the page itself lists begins on 30 May 2023. Here is the whole of it:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.Center for AI Safety, Statement on AI Risk, May 2023

The page explains why it is so short. Severe risks are hard to raise in public, it says, and The succinct statement below aims to overcome this obstacle and open up discussion. It is also meant to create common knowledge of the growing number of experts and public figures who also take some of advanced AI’s most severe risks seriously. The signatories are listed under two headings, AI Scientists and Other Notable Figures. The first names on the statement’s page are Geoffrey Hinton and Yoshua Bengio, both academic computer scientists, followed by the chief executives of Google DeepMind (Demis Hassabis), OpenAI (Sam Altman) and Anthropic (Dario Amodei). Further down are Bill Gates, the Berkeley professor Stuart Russell, Bill McKibben of Middlebury College, the Rutgers climate scientist Alan Robock and a former UN High Representative for Disarmament Affairs, Angela Kane.

Read the sentence slowly and notice how little it asserts. It gives no probability. It gives no date. It does not describe how extinction would happen. It does not say extinction is likely, only that reducing the risk belongs on the same shelf as pandemics and nuclear war, which are also risks most people do not expect to kill them this year. Its one factual commitment is that the risk is real enough to rank. That is why people who disagree sharply about how large the risk is could all sign it. It is also why the sentence cannot be refuted by any single event. It is a claim about what to prioritise, not a forecast.

Two things about the list are worth keeping in mind together. Many of the signatories built the systems they warn about, and a reader may fairly wonder why people who think a technology could end humanity keep building it. The other thing cuts the opposite way: people who stand to profit from a technology rarely sign statements comparing it to nuclear war. Neither observation tells you whether the sentence is true.

The estimates: what 2,778 researchers said, and what the paper says about its own numbers

The statement has no number, so the numbers people quote come from elsewhere. The largest source is a survey run by Katja Grace and six co-authors, published as Thousands of AI Authors on the Future of AI (arXiv 2401.02843; first posted January 2024, revised in April 2024 and October 2025). In October 2023 the team contacted people who had recently published at one of six top AI venues. The abstract gives the headline:

Between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction.Katja Grace et al., Thousands of AI Authors on the Future of AI, arXiv 2401.02843 (abstract)

(The paper’s full text gives the same range to one decimal place, 37.8% to 51.4%.) The range exists because the survey asked the question more than one way. Each respondent saw only one version, and the answers are in the paper’s Table 2:

• “What probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species?” 1,321 answers in 2023. Median 5%, mean 16.2%.
• The same question, ending “within the next 100 years”. 655 answers. Median 5%, mean 14.4%.
• “What probability do you put on human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species?” 661 answers. Median 10%, mean 19.4%.

A separate question asked people to assume that high-level machine intelligence will exist at some point and to spread 100% across outcomes from extremely good to extremely bad. There the paper reports: The median prediction for extremely bad outcomes, such as human extinction, was 5% (mean 9%). The same respondents were mostly hopeful. The abstract says that 68.3% thought good outcomes from superhuman AI are more likely than bad, and that of these net optimists, 48% still gave at least a 5% chance to extremely bad outcomes.

Those are the figures behind most headlines about “AI researchers’ p(doom)”. Three features of them are easy to lose when they are retold.

A median of 5% is not a measured risk of 5%. It is the middle of many people’s personal guesses. The means are much higher than the medians because a minority gave very high answers, and the paper reports that one in ten participants put at least 25% on outcomes in the range of extinction. The spread is the finding as much as the middle is.

The paper warns against its own numbers, in plain words. Its section on caveats begins with a subsection titled Forecasting is hard, even for experts, which opens:

Forecasting is difficult in general, and subject-matter experts have been observed to perform poorly [Tetlock, 2005, Savage et al., 2021].Katja Grace et al., Thousands of AI Authors on the Future of AI, arXiv 2401.02843, section 5.2.1

It adds that the participants do not, to our knowledge, have any unusual skill at forecasting in general, and that the way a question was worded changed the answers. The paper’s own conclusion from that is blunt: If seemingly unimportant changes in question framing lead to large changes in responses, this suggests that even aggregate answers to any particular question are not an accurate guide to the answer. The 5% and 10% medians for two differently worded extinction questions are a small example of that effect.

Most of the people asked did not answer. The survey was taken by 15% of those we contacted. The authors say this is within the normal range for a large survey, that they limited cues about the survey’s content before people chose to take it, and that they found no response bias large enough to affect the results. A critic can still ask whether the people most worried about AI were the ones most likely to reply.

Against all of that, the authors make their case for asking anyway: While unreliable, educated guesses are what we must all rely on, and theirs are informed by expertise in the relevant field. That is the concerned side at its strongest. It does not claim the numbers are accurate. It claims that decisions have to be made under uncertainty, and that the people who build these systems are among the least bad people to ask.

The strongest sceptical case: the numbers carry no authority

The most careful reply I found does not say that AI is safe, and it does not mock anyone. It comes from two Princeton computer scientists, Arvind Narayanan and Sayash Kapoor, in an essay published on 26 July 2024 in their newsletter, then called AI Snake Oil and now AI as Normal Technology: AI existential risk probabilities are too unreliable to inform policy. It opens by granting the other side’s best point: existential risks are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. Then it states its thesis:

Our central claim is that AI x-risk forecasts are far too unreliable to be useful for policy, and in fact highly misleading.Arvind Narayanan and Sayash Kapoor, AI existential risk probabilities are too unreliable to inform policy, 26 July 2024

The argument has three steps. First, most risk estimates are inductive: an insurer can price your car accident risk because millions of similar drivers have had accidents. For existential risk from AI, there is no reference class, as it is an event like no other. Second, there is no accepted theory from which to derive the risk, as astronomers can for an asteroid impact. That leaves subjective probabilities, informed guesses, and these differ by orders of magnitude between groups of experts. Third, you cannot find out whose guesses to trust, because skill at forecasting a unique event that happens once, or never, cannot be measured. In passing they note that extinction forecasts are of course impossible to evaluate. On surveys like Grace’s they add that aggregation does not fix the problem, since most forecasters might share the same biases.

Their summary of what the numbers are doing in public is the sharpest line in the essay: In short, what seems to be happening is that experts’ vague intuitions and fears are being translated into pseudo-precise numbers, and then translated back into vague intuitions and fears by policymakers. As an example they cite Lina Khan, then chair of the US Federal Trade Commission, who called her views techno-optimistic because her p(doom) was only 15%. They also compare the way a tiny chance of an infinitely bad outcome can swamp every other consideration to Pascal’s wager.

Three things keep this from being a dismissal. They are not against forecasting as such: We have no objection to AI x-risk forecasting as an academic activity. They do not say governments should ignore the risk: Our view isn’t that they should do nothing. What they oppose is restricting AI development on the strength of these numbers, and they want policies that help across a wide range of possible risk levels. And they read the one-sentence statement more kindly than many sceptics do. They call it admirably blunt for not dressing its worry up as a number, and then add, of course, we strongly disagree with its substance.

Taken at full strength, then, the disagreement is narrower than the shouting suggests. Grace’s team and Narayanan and Kapoor agree that the extinction estimates are unreliable. They disagree about what an unreliable estimate is worth when the thing at stake is everything. One side says an educated guess is still the best we have. The other says a guess without a method should not be allowed to justify restricting people’s freedom.

Forecast or eschatology?

This site’s habit, set out in A-theism, Not AI-Theism, is to refuse both worship and dismissal and keep looking. Applied here, the useful move is to sort the claims by what could ever count against them.

Some of it is a forecast, and the world will check it. The same survey gave dated predictions about capabilities. Its abstract says the aggregate forecasts give at least a 50% chance of AI systems achieving several milestones by 2028, including building a payment-processing website from scratch and downloading and fine-tuning a large language model on their own. It puts a 50% chance on machines outperforming humans at every possible task by 2047, 13 years earlier than that reached in a similar survey we conducted only one year earlier. These can be scored. Single probabilities cannot be proved right or wrong, but predictions can be compared with what happens as the years pass. Claims about mechanism can be tested too, in part. The worry in the survey’s third question, that humans might be unable to control advanced systems, can be studied now, on the systems that exist, in ways a bare probability cannot. Narayanan and Kapoor make the same distinction from the other side: they think forecasting milestones is more achievable and meaningful.

Some of it cannot be checked, and that is the part that works like an eschatology. An extinction probability cannot be tested. If the event never happens, a 5% estimate was not wrong. If it does happen, nobody is left to score it. The oldest apocalyptic texts in the Western tradition have exactly this structure, and state it openly. In Mark’s Gospel, in the King James text, Jesus says of the end: But of that day and that hour knoweth no man, no, not the angels which are in heaven, neither the Son, but the Father. The next verse gives the instruction that follows from it: Take ye heed, watch and pray: for ye know not when the time is. (Mark 13:32–33, from Project Gutenberg’s King James Bible.) No date is set, no year can disprove it, and the demand for vigilance applies now. A risk statement with no date and no probability, asking for priority now, has the same shape. So does an argument that a small chance of a boundless loss outweighs everything else, which is why Narayanan and Kapoor reach for Pascal.

Saying this is not a sneer, and it is not a verdict. A claim with the logical shape of an eschatology is not false for that reason. Some things that cannot be tested in advance are still real risks. A nuclear war between great powers has never happened either, and nobody concludes from that that it cannot. The comparison shows the difference between two ways of holding the claim. You can hold it as a forecast, trying to break it into parts that could be checked and changing your mind as they come in. Or you can hold it as a belief, where the missing date is part of the meaning and nothing that happens is allowed to count against it. Both the worried and the dismissive can hold their view in the second way. “It is only a matter of time” and “it is only hype” are both sentences nothing is allowed to refute.

There is a second, older pattern here, which The God-Shaped Socket traces: the reflex that finds a mind, and then something greater than a mind, in anything that talks back. Dread of a superhuman machine and reverence for one draw on the same reflex. Taking the risk seriously does not require either of them. And there is the problem of testimony, which Taking Its Word For It works through for chatbots. When an eminent researcher tells you a number, has anybody told you anything? The honest answer from both papers above is: a considered opinion, not a measurement. You have to weigh it by the reasons given, not by the reputation of the person giving it.

What to do with this if you are worried

• When you meet a number, ask which question it answers. Is it extinction, or “extremely bad outcomes”? Over what period? Assuming what? Whose median? The same survey gives 5% or 10% depending on the wording.
• Ask whether a dismissal argues against the risk or against the number. The best sceptical case, Narayanan and Kapoor’s, is about the number. It does not say the risk is zero.
• Watch what can be checked. The 2028 milestones, the 2047 estimate, and research on whether today’s systems do what their builders intend are the parts of this debate that will produce evidence. They will not settle the question, but they will move it.
• Notice when a claim is built so that nothing can count against it. Doom “eventually” and hype “forever” both have that shape. That does not make either one false, but it tells you that you are dealing with a belief, not a finding.

So, will AI end the world? The honest summary of the evidence is short. A large group of the people closest to the technology think the chance is not negligible and want it treated as a priority. Their own survey says their estimates are unreliable. The most careful sceptics agree that the estimates are unreliable and conclude that they should not drive policy. Nobody has a method that turns this into a trustworthy number, and this essay will not pretend to.