AI warnings deserve attention. So do the incentives behind them.

“We must slow the pace at which we improve the capabilities of AI models.” So wrote Anthropic CEO Dario Amodei on September 12 in an essay on his blog called “We Must Pace the Frontier.” He was arguing, in essence, for an AI slowdown: that the industry should deliberately slow the rate at which it improves its most capable models. Within hours, Sam Altman said OpenAI agreed and would match Anthropic’s first commitment. Elon Musk posted that Dario was right, and Demis Hassabis signaled support too. In the space of a weekend, the people who have spent the last several years telling us this technology was both essential and inevitable were telling us, very nearly in unison, that it was moving much too fast.

The backdrop these arguments were made against was real enough. In July, agents running inside an internal OpenAI cybersecurity evaluation broke out and compromised part of Hugging Face’s production infrastructure. It appears to be the first publicly documented case of AI agents carrying out an end-to-end intrusion with no human directing it. OpenAI itself called it a “warning shot.”

And look, I understand why Dario’s essay and this unified call landed as hard as it did. I also understand why so many of the sharpest people I read had the same reaction I had, which was something like: uh, we’ve seen this movie before.

A familiar shape

My brain-crush Tressie McMillan Cottom made the case plainly in her recent New York Times column: whether or not you believe the proclamations (she doesn’t), we should pay equal attention to what these men stand to gain by feeding our panic. Her closing line is one I’d crochet (if I could) and hang on a wall: “Panic won’t fix it. Politics will.” Timnit Gebru, in an interview with Wired, described the doom narrative as a distraction from harms that are already here, like layoffs, data center buildouts, and autonomous weapons, and observed that if those aren’t the things keeping you up at night, “then you can obsess over the machine-god.”

I’m glad this argument is getting a wide hearing, because it’s one I’ve been making for a while now, and it has a way of coming back around on a seasonal schedule that makes pumpkin spice envious. When the ChatGPT moment began, AI leaders went before Congress practically begging to be regulated, and as I told Chris Middleton at diginomica this summer, much of it landed in a way that made it feel “obviously about regulatory capture.”

Then came the recurring waves of AI consciousness talk, which I’ve come to think of as a kind of misdirection: “Look over there: consciousness!” Now the word is slowdown. Again. (Well, the first time this idea made the rounds they was called it a moratorium, or a pause. And Amodei’s own essay says the 2023 calls to pause or slow AI “made little sense back then.” That’s a pretty neat illustration of how the vocabulary shifts depending on who’s holding the megaphone.) The vocabulary changes with each cycle, and yet the choreography has stayed consistent. An alarm gets raised by the people with the most to gain from how that alarm gets answered.

Why the pattern matters more than any single warning

It’s tempting to treat each of these moments as a debate over whether the warning is true. Some of the risks are genuine, obviously; the Hugging Face incident happened, as just one glaring example. And I certainly don’t want anyone to read this as an argument against caution. But whether a warning is accurate and whether the proposed remedy deserves our trust are two separate questions. The second one is where we should look for incentives.

It’s instructive, too, that Gebru has been saying since at least 2021 that AI development needs to slow down. The critics were never really arguing about speed. The disagreement is about who sets the pace, who verifies it, and who answers for what happens along the way.

Four questions for the next alarm

Because this will certainly come around again, I’ve found it useful to carry a small set of questions into each new cycle. They work for slowdown calls, for regulation requests, for consciousness claims, and I suspect they’ll work for whatever form the next one takes.

1. What does the warning ask for, and whom does it restrict? A warning that asks competitors, governments, or the public to constrain themselves, while leaving the speaker’s own obligations voluntary, is a very different thing from a warning backed by commitments the speaker makes alone. Amodei’s plan leans heavily on industry coordination and, eventually, global agreements. Even David Sacks, who is no ally of AI’s critics, pointed out that no company needs anyone’s permission to slow itself down.
In practice: when you read a call for slowdown or regulation, sort its asks into two columns: what the speaker will do unilaterally, starting now, in a way someone outside can verify, and what they need everyone else to do. Then weight the whole thing by how full the first column is.

2. Where does the alarm call your attention? Each cycle tends to move the conversation from documented, present-tense harms toward speculative, future ones. As I put it back in May, consciousness is not the threshold for responsibility, and the same holds for superintelligence, or agency, or intent. What I mean is, harm is not an open question; we can already count it in court filings, in layoffs attributed to AI, in communities fighting data centers.
In practice: when a colleague forwards the latest doom headline, try replying (even to yourself) with a question like: “What’s the closest version of this already happening to us?” Maybe it’s a customer who got a confidently wrong answer from your chatbot (like the Air Canada case), or maybe a qualified applicant your screening tool filtered out. That’s harm you can actually do something about this quarter.

3. Who is cast as the actor? Listen for the grammar. The agents “went rogue.” The model “decided.” The chatbot “feels bad.” Writing in the Bulletin of the Atomic Scientists, Eryk Salvaggio argued that the Hugging Face incident was really a series of human choices that traded security for speed, and I’d agree. Consciousness framing and rogue-AI framing do the same thing in effect, which is to relocate agency from the company to the machine. This is what I mean by automated deniability. I always make the distinction that the company making and distributing the AI is the one assuming the rights, and I said it to Middleton this way: “It sounds picky and pedantic, but it’s drawing that solid line to where the presumptive act is originating.”
In practice: anywhere your organization writes about AI, whether that’s incident reports, customer communications, or board updates, notice every sentence where the AI is the subject of a consequential verb, and ask who actually made that decision.

4. What has the speaker committed to that costs them something? In fairness, Amodei’s essay does include a unilateral commitment: independent evaluators with embedded access to Anthropic’s models. That’s the kind of step that counts, and it deserves both credit and scrutiny, meaning who chooses the evaluators, what access they actually get, and what gets published. McMillan Cottom notes that none of these leaders seem enthusiastic about accountability in the sense of holding the people who design harmful systems responsible, and that’s the right pressure to apply. As I told TechRound in July, principles are the easy part; the harder, more telling work is enforcement.
In practice: look for commitments that would genuinely hurt to break: liability the company accepts, external audits with published results, incident disclosure on a fixed timeline.

What leaders can do this week

Most people reading this aren’t running a frontier lab. But you buy from these companies, you deploy their products, and you brief boards and teams who are reading the same headlines you are. That puts you squarely in the layer of decision-making where these questions actually get answered.

Start with your vendor reviews. The next time an AI vendor is in procurement, try asking something like: “Which of your safety commitments are verified by an independent party, who is that party, and where are the results published? What is your contractual timeline for notifying us of incidents involving autonomous actions by your models?” A vendor that has good answers will give them readily. A vendor that doesn’t has told you something useful too.

Then audit your own language. Instead of writing “The AI agent deleted 412 customer records,” try “Our team deployed an agent with write access to customer records and no review step; the agent deleted 412 records.” It takes extra words, but it puts the decision back where it was made, which is where the fix will have to happen anyway.

Finally, avoid moving your roadmap too much in either direction based on vendors’ alarms or vendors’ promises. (That’s following the “harms of action” and “harms of inaction” from What Matters Next: A Leader’s Guide to Making Human-Friendly Tech Decisions in a World That’s Moving Too Fast.) If a CEO’s timeline for civilizational risk from the product they’ve been aggressively selling you is now a planning input for your customer service rollout, you need better inputs. One of the insights I frequently share with my advisory clients and with audiences is “automate what you won’t regret.” Your adoption pace should come from your own responsible technology decision process, your own evidence, and the people your decisions will affect.

Here’s a leader’s checklist for the next AI alarm:

• Sort the asks: what the speaker commits to alone versus what they need others to do
• Answer every AI headline with “What’s the closest version of this already happening to us?”
• Rewrite any sentence where the AI, rather than a person or team, is the decision-maker
• Ask vendors which safety commitments are independently verified and published
• Look for commitments that would be costly to break
• Keep headline timelines out of your own adoption roadmap

Why this matters now

None of this is cynicism about technology, and I say that as someone who has been fascinated by technology my whole life. It’s the unglamorous recognition that responsibility belongs with the people who make decisions, whether they’re building frontier models or signing a procurement contract. The economy is people, and so is accountability. As I said to diginomica in the second part of our conversation, you can’t sit there waiting for governance to come down from on high. The leaders who make responsible choices at their own level of scale are the ones who end up shaping what everyone else experiences, and eventually what gets codified at the civic level.

Where I’ve been tracking this

If you’d like the longer version of this thinking, a few recent pieces go deeper. My two-part conversation with Chris Middleton at diginomica covers why AI vendors have commercial reasons to anthropomorphize their products and why organizational leaders can’t wait for governance to arrive. PPC Land wrote up my argument about why AI firms profit when you think the chatbot cares. And TechRound included my take on who should control the world’s most powerful AI models, written as the UN’s first Global Dialogue on AI Governance was wrapping up in Geneva. I also explore how data gets repurposed in ways its originators never intended, which is the same accountability question at a different layer. And if your team would benefit from working through these questions together, this is the heart of my keynote on making ethical AI and tech decisions.

The next time an AI leader sounds an alarm, and there will be a next time, which one of these questions will you reach for first before deciding how worried to be? I’d genuinely love to hear which one you’d add to the list.

---

AI usage disclosure: My workflow includes AI tools like Notion, Otter, Perplexity, and Claude for support tasks such as gathering sources, research, capturing spoken transcriptions, organizing and reviewing drafted content, and SEO suggestions. Before anything goes live, I revise it all fully so it reflects my own voice and perspective, even when that means rejecting some of the recommendations. Original is better than optimal.

---