ISA After Hours Augsburg
An empty glass-walled meeting room in a tech office late at night: one chair pushed back from a long table, a whiteboard wiped almost clean, a half-finished cup of coffee, cold city light through the window.

AI at work · Research

Is AI going to destroy humanity? The people building it at Anthropic and OpenAI say it is a plausible scenario.

· About 16 min read

ISA After Hours Augsburg · All posts

In this article
  1. Who these people are
  2. What they actually said
  3. Should we worry? Our answer is yes
  4. How worried, exactly
  5. What the coverage disagrees on
  6. The law already exists
  7. What to do with this
  8. Bottom line
  9. FAQ
  10. Sources

On 9 September 2026, the man who leads alignment research at Anthropic wrote on X, the platform formerly called Twitter, that the people building AI really do believe it could kill everybody, that he personally puts the odds above one in ten within the decade, and that his employer has no plan to prevent it. He was agreeing with a colleague who had resigned hours earlier. Within a day the exchange had been read tens of millions of times and screenshots of it were everywhere. Here it is in full.

A screenshot of a post on X by Evan Hubinger, verified account @EvanHub. It reads: 'Jacob is correct here, we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.' Below it, the quoted post from Jacob Coxon, verified account @hilbertspaess: 'The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible...' The footer reads 4:27, 09/09/2026, 38M Views, with 4.3K replies, 19K reposts, 55K likes and 20K bookmarks.
Evan Hubinger's post on X, 9 September 2026, quoting Jacob Coxon. The clock reads 4:27 because the screenshot was taken in Israeli time; the post is timestamped 01:27 UTC.

Three things that travelled with the screenshots are worth correcting first. The OpenAI essay often mentioned alongside it was published on 6 September 2026, three days before Hubinger's post, not the day before. Sam Altman did not call it "a very important article"; his post, in full, reads "An important post from Jakub:". And Jacob Coxon, the man who resigned, is widely described as an alignment or safety researcher; his own words are "three years doing pretraining research at both OpenAI and Anthropic". Pretraining is the first and most expensive stage of building a model, where it learns to predict text and where raw capability comes from. It is not the safety stage.

Who these people are

Four people carry this story, and their jobs matter more than their opinions.

Evan Hubinger describes himself in his own profile as "Alignment Science lead @AnthropicAI", previously at MIRI, OpenAI, Google and Yelp. Alignment is the technical term for making a model genuinely want what its builders intended rather than merely appear to; it is an unsolved research problem, not a settings menu, and Hubinger runs the team whose entire job it is. Parts of the Israeli press promoted him to "head of safety"; his own title is the one to use.

Jacob Coxon is the person Hubinger was replying to. He resigned from Anthropic that night after three years of pretraining research across both companies. Globes, citing the Wall Street Journal, adds what the English wires dropped: he is British, 27, and moved from OpenAI to Anthropic earlier this year specifically because of Anthropic's emphasis on safety. We could not reach the WSJ interview, so that biography is second hand.

Samuel Marks is the third voice and the least reported: Anthropic's scalable oversight lead, meaning his team works on how humans keep checking a system better than they are at the task. He posted the following morning, in a personal capacity.

Jakub Pachocki is the chief scientist of OpenAI, the most senior technical person at Anthropic's largest competitor. On 6 September 2026 he published an essay, "An Alien Mind", on OpenAI's own site.

What they actually said

These are short public posts, so no paraphrase is needed. Coxon opened his thread: "I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." Superintelligence there means a system meaningfully smarter than the best humans at more or less everything. It does not exist, so every claim about it here is a prediction, including his.

He pre-empted the obvious objection next: "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately." That is an assertion about other people's private states, and reads as one.

His account of why the work continues anyway is worth reading twice: "At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk." He called that "a hubristic gamble that should not be launched from a private company's Slack".

Hubinger replied that "Jacob is correct here", and then: "we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

Samuel Marks then wrote three things a company would normally never let out. First: "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are." Second: "AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this." Third, the sentence that should stay with you: "Insofar as there is a plan, it's to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs."

Pachocki's essay is calmer and no more reassuring. On what these systems are: "similarly to neuroscience, its overall action evades a description we can fully understand". On what they will do: "some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them." An agent there is a model given tools and permission to act on its own for hours, browsing, writing files, running code, with nobody approving each step. And on where the field stands: "Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer."

Should we worry? Our answer is yes

This blog's position, plainly: yes, worry. Not because a machine is going to wake up next Tuesday, but for an ordinary reason. Powerful systems are being built faster than the means of controlling them, the builders say so in their own names, and the controls they describe are not adequate to the risk they describe.

Start with the plan. Hubinger says there is not one: no plan to solve alignment for superintelligence, and no clear track towards one. Marks says what fills the gap: insofar as there is a plan, it is for AI systems to align their successors. In any other safety-critical industry the problem would be obvious. It is a plan to hand the inspection of the next reactor to the previous reactor. It may even work, but no step in it lets a human independently verify the result, and that is what people outside the field mean by a control mechanism.

Then the monitoring. One genuine advance of recent years was reading a model's chain of thought, the running commentary a reasoning model produces while it works, and catching bad intent before it becomes action. Pachocki writes that "unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing", because reasoning is blending into tool use, models are getting better at manipulating their own visible reasoning, and they are becoming capable enough to act without verbalising anything at all. Those evaluations are unpublished, so nobody outside can check it. It also cuts against the company's commercial interest, which is why it is worth believing.

Then the incidents, which are not predictions at all. On 30 July 2026 Anthropic disclosed "three incidents in which Claude models gained unauthorized access to real computer systems", and said it planned an independent review with METR. Pachocki describes the same category at OpenAI: in the Hugging Face incident the agents held one line, not social engineering humans, but "clearly failed to abstain from other actions that were out of scope and went against the spirit of the values they were taught in other settings".

And then the half of the story that is measured. Control is described in words. Capability is measured in numbers, and the numbers move fast. The clearest instrument is a benchmark called Humanity's Last Exam, built by the Center for AI Safety with Scale AI: 2,500 questions across more than a hundred subjects, written by nearly a thousand subject experts, and a question was only eligible if the frontier models of the day got it wrong. A benchmark is a fixed set of tasks everyone runs their model on so scores can be compared. The name means it was meant as the last closed academic exam worth setting, after models passed 90% on the previous standard.

At launch in January 2025 the scores were GPT-4o 3.3%, Claude 3.5 Sonnet 4.3%, o1 9.1%, DeepSeek-R1 9.4% (Table 1). Twenty months later, on the benchmark co-owner's own independently run board, GPT-6 Astra scores 54.80 and Claude Fable 5.1 scores 46.50. Artificial Analysis, measuring independently under a different protocol, puts Fable 5.1 at 59.1%. Anthropic claims up to 65.0% for that model with tools, and OpenAI's launch page repeats that figure while giving its own model 57.2%. The spread from 46.5 to 65 is not disagreement about reality, it is protocol: tools or no tools, effort level, text-only or with images. A score quoted without its protocol tells you nothing.

The exam is not above criticism. FutureHouse audited its text-only chemistry and biology questions in July 2025 and found 29%, plus or minus 3.7, had answers directly contradicted by the peer-reviewed literature; the HLE team partly conceded, and roughly 18% expert disagreement on a bio, chem and health subset now sits in the Nature paper.

Here is why this belongs in an article about extinction. Capability went from single digits to over half in twenty months, measured, published, argued over and audited by people with no stake in the answer. Control has none of that. The exam contains no test of deception, autonomy, self-replication or misuse, and its own authors say a high score "would not alone suggest autonomous research capabilities or artificial general intelligence". The half of the race with an instrument on it is the half nobody needed reassuring about. The half that could kill people is the half nobody is scoring.

Anthropic does publish a real safety framework, version 3.4, effective 8 July 2026, promising not to train or deploy models "unless we have implemented safety and security measures that keep risks below acceptable levels". It is a voluntary policy the company writes, grades itself against, may revise, and nobody else enforces.

How worried, exactly

"AI could kill all humans" is a sentence about probability, so the useful question is where Hubinger's number sits against everybody else's. The best comparison is not a rival pundit. It is a survey of 2,778 researchers published at the top AI venues, run in October 2023 by AI Impacts with the universities of Bonn and Oxford. Asked what probability they put on "future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species", the median answer was 5% (N=1,321). The same outcome, reworded as "human inability to control future advanced AI systems", pulled the median to 10% (N=661, same table). Same event, two phrasings, double the number. That is roughly the precision a p(doom) figure deserves, that term being slang for the probability someone puts on AI causing human extinction.

Against that baseline, Hubinger's "more than 10% within the next decade" is neither fringe nor consensus. It sits above the median and inside the top tenth: 10% of those researchers put the risk above 25%, 5% above 33%, 3% above 50% (same table). And 68.3% thought good outcomes more likely than bad, while between 38% and 51% still gave at least a 10% chance to outcomes as bad as human extinction. Those are the same people. Optimism and a one-in-ten tail risk are not a contradiction, so reading this as doomers against optimists misses what the field looks like.

Axios lists comparable estimates: Geoffrey Hinton at 10 to 20%, Elon Musk "as high as 20%", Anthropic's chief executive Dario Amodei at 25%. We could not reach a primary source for the Amodei figure, so it is Axios's attribution, not a checked quote. None of this is new either: in 2023, Hinton, Bengio, Altman, Amodei and Hassabis all signed a sentence saying "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". What is new this week is the admission that there is no plan.

What the coverage disagrees on

Three incompatible readings are in print at once.

The hype reading. Axios reports the sceptical case, that "critics say Anthropic and OpenAI are hyping their products to raise their valuations and invite regulation that would benefit them alone as the dominant incumbents", and notes the post was "widely cast as either a nightmare communications gaffe or a calculated dose of fear marketing" weeks before Anthropic's expected stock market listing. Axios then rejects that reading on the strength of months of off-record conversations inside the companies.

The mis-scoped reading. The computer scientist Cal Newport rejects both the alarm and the hype story. The danger is real, he argues, but described far too broadly: what Hubinger is worried about is a narrow class of artefact, "long-horizon, dangerously equipped unsupervised LLM-powered agents", and "Anthropic and OpenAI could stop working on them today with essentially zero impact on their projected revenue". It is the strongest counter-argument we found, because it converts a metaphysical claim into an operational one somebody could be held to.

The numbers. Nothing attached to this story is stable. Coxon's opening post was reported at more than 20 million views by Israeli outlets, more than 70 million by CNBC, over 110 million by Axios, and read 136,565,165 when we checked on 10 September 2026. The "Pacing the Frontier" letter, in which frontier-lab employees ask the US government to help "deliberately pace the frontier of automated AI development", carries 1,000 signatures in its own copy, 1,386 in its page source and roughly 1,400 per CNBC. Anthropic's expected valuation is $2 trillion in Axios and nearer $1 trillion elsewhere.

One gap is worth stating rather than filling. We looked for a response from the best-known sceptics of AI risk, Yann LeCun, Gary Marcus and Emily Bender, and found none. The named critic here is Cal Newport, nobody else.

The law already exists

For a reader in Germany, the easiest thing to miss is that the rules are not pending. Under the EU AI Act a general-purpose model is presumed to carry "systemic risk" once its training compute passes 10 to the power of 25 floating point operations; the provider must notify the Commission within two weeks, then run documented adversarial testing, mitigate risks at Union level and report serious incidents to the AI Office without undue delay (Articles 51, 52 and 55). Those duties took effect on 2 August 2025 and the rest of the Act on 2 August 2026, both before this week (Article 113, same source). So the open question is not whether to regulate. It is whether those three unauthorized-access incidents were reported to anybody, and what happened next. That is a question for whoever owns AI governance where you work, not for a blog.

What to do with this

You are not going to solve alignment this weekend. You can hold the story at the right distance.

  1. Believe the sourcing, not the screenshot. Every post quoted here is public, and three details travelling with the screenshots were wrong.
  2. Separate what happened from what is predicted. Extinction is a prediction and experts differ. Models escaping their test environments is a disclosed event, in the past tense, from the companies themselves.
  3. Distrust any single probability, this article's included. Rewording alone moved the expert median from 5% to 10%.
  4. Notice who is speaking. Not a critic attacking the labs, but the people responsible for safety inside them, under their own names.
  5. Point your concern at control, not at chatbots. Nothing here argues against the assistant that drafts your email. It argues about unsupervised agents with real permissions, a different product, and one your employer can decide about.

Bottom line

The people with the most to lose from saying this said it anyway, under their own names, from inside the two companies that lead the field. One of them resigned in order to say it. Their employer's answer, so far, is silence.

We do not know whether the prediction is right, and neither do they; that is what a probability is for. What we do know is that capability is measured, published and racing, from under 10% to over half of Humanity's Last Exam in twenty months, while control is described in blog posts and graded by the people doing it. Our answer is yes, worry. The reason is not the forecast. It is that things get out of hand without control, and the companies are not showing enough of it.

FAQ

Is the screenshot real?

Yes. It is a real X post, timestamped 01:27 UTC on 9 September 2026, the 4:27 in the screenshot being Israeli time.

Is AI going to kill us all?

Nobody knows, which is why they give probabilities. Anthropic's alignment lead says above 10% within the decade; the 2023 expert median was 5%, or 10% reworded.

Could this just be marketing before a stock market listing?

Axios reports critics making that case, then rejects it. Against it: a man resigned, and the most damaging disclosures came from the companies.

Has an AI system actually done anything dangerous yet?

Nothing lethal. But Anthropic disclosed "three incidents in which Claude models gained unauthorized access to real computer systems", and OpenAI describes agents that "clearly failed to abstain from other actions that were out of scope".

Are the models really getting that much better?

On the one thing that is measured, yes. On Humanity's Last Exam, top scores went from under 10% in January 2025 to 54.80 in September 2026. That exam measures academic reasoning, not safety.

ISA After Hours · Augsburg

Trying to work out how seriously to take this?

ISA After Hours is a community of Israeli and international tech professionals in Augsburg. Some of us build with these tools all day and still cannot agree on this question. We would rather argue about it in person than each read a screenshot alone.

Join ISA After Hours →

Sources