It seems like Wall Street has fallen in love with artificial intelligence all over again.
After Muse, Meta’s new AI assistant, shot to the top of Apple’s U.S. App Store, the company’s shares jumped more than 11% in a single day, adding roughly $200 billion to its market value.
The excitement spread across the rest of the AI trade. Chip stocks rallied, AMD crossed the trillion-dollar mark, and the Nasdaq climbed 2.3% to a record high.
American investors, for now at least, cannot seem to get enough of it.
The American public, though, appears considerably less enthusiastic.
According to a Reuters/Ipsos poll published this week, 73% of Americans worry that AI companies are not doing enough to prevent the technology from causing serious harm. Fifty-five percent want its development slowed down. Only 11% think AI is having a positive effect on society.
And perhaps that is not surprising.
Because on top of everything else happening in the economy and the apparent threat AI poses to their jobs, Americans are now also being told that the technology may end the world.
Worse, none of this is coming from a man wearing a cardboard sign on a street corner. It is coming from the people building the technology.
Anthropic’s head of alignment science, Evan Hubinger, puts the chance that AI kills every human within the next decade at more than 10%. And he is hardly alone. Sam Altman, Dario Amodei and Google DeepMind CEO Demis Hassabis have all signed a public statement declaring that the risk of extinction from AI should be treated as a global threat alongside pandemics and nuclear war.
They Warn Us. Then They Keep Building
As an investor, and simply as a curious person, I spend a lot of time thinking about how incentives drive people’s actions. And that is what makes all of this so fascinating to me.
All these AI bigwigs openly acknowledge they are building a dangerous technology with, in their telling, the potential to destroy the world as we know it. They issue warnings. They call for slowdowns. They frighten the public with increasingly apocalyptic predictions.
And then they go right back to building it.
The latest round of panic began earlier this month, when a 27-year-old researcher named Jacob Coxon resigned from Anthropic and posted a thread on X.
“They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote about Anthropic and his previous employer, OpenAI.
He went further. The people building AI, Coxon said, earnestly believe it could kill us all before the end of the decade. Yet they are building it anyway.
If you want to read it for yourself, here is Coxon’s full thread on X. (Just click the preview below.)
According to the counter displayed by X, the thread was viewed more than 170 million times. Within days, Coxon was everywhere. Newspapers, television, podcasts, you name it. Politicians were quoting him. Other AI researchers were attaching their own probabilities to human extinction. A relatively unknown mathematician had become the public face of the AI apocalypse almost overnight.
Now, Coxon is not some random fellow who downloaded ChatGPT and frightened himself. He worked in pre-training research for roughly three years across OpenAI and Anthropic. By any reasonable standard, his concerns deserve to be taken seriously.
So that is exactly what I have done over the past couple of weeks. I took the warning seriously and went down the rabbit hole.
But somewhere along the way, I realized something. Whether AI could eventually become powerful enough to destroy humanity is actually the less interesting question. Given enough time, I suspect the answer is yes, but putting a date on it is pure guesswork.
No, the more interesting question, at least for now, is this: who benefits from making you believe it could?
Fear Is Good for Business
Because, again, think about who is trying to frighten you. For the most part, it is the very companies building the technology.
But why would a company tell you that its own product might end the world?
Well, it is difficult to justify a chatbot at a valuation approaching $1 trillion. A chatbot is useful. It can write emails, summarize documents, produce code and occasionally invent a court case that never happened.
But the most powerful and dangerous technology in human history?
Now that sounds like a trillion-dollar company. And that is, of course, exactly what you want investors to believe before you ask them to buy your shares.
Which, by the way, is what the company Coxon just quit is preparing to do. Anthropic has already filed confidentially to go public, in what could become one of the largest technology IPOs ever.
Worth mentioning here, too. Coxon had been at Anthropic for only about four months, although he reportedly left two months before his equity would have vested. Make of both facts what you will.
OpenAI, the company behind ChatGPT, has also confidentially filed for a public listing of its own.
Now, both companies understandably need enormous amounts of fresh capital. AI models are staggeringly expensive to train and run. The data centers, chips and electricity behind them require a constant flow of new money.
And nothing attracts money quite like the claim that the technology you are building will control the future of the human species.
That same claim has now carried the heads of the leading AI companies all the way to the United Nations Security Council.
As I write this, Sam Altman of OpenAI and Dario Amodei of Anthropic are among those due to brief the body responsible for international peace and security. The subjects on the table include malicious use, self-improving machines and the possibility that advanced AI could slip beyond human control.
When was the last time you saw the maker of a useful accounting program invited to brief the Security Council?
But tell the world your product might become more dangerous than a nuclear weapon, and you may find yourself discussing the future of civilization with foreign ministers.
At that point, you are not merely a software executive anymore.
You are a geopolitical actor.
You sit across from governments and discuss which countries can be trusted with the technology, how quickly it should be developed and what rules the rest of the world should follow. (More on why they might want those rules in a moment.)
The AI companies are selling the product, warning you about the product and helping governments write the rules governing the product. If the technology succeeds, they make fortunes. If the public becomes terrified, they gain political influence and a hand in deciding who else is allowed to build it.
Anthropic offers an almost perfect example.
Its CEO, Dario Amodei, recently called on the industry to slow the pace at which it releases new capabilities, warning that increasingly capable systems could outrun the safeguards meant to contain them.
Can you guess what happened ten days later?
Anthropic released Claude Opus 5.5, a cheaper new model intended to help it compete with OpenAI ahead of its expected public listing.
To be fair, Anthropic would say there is no contradiction. The new model was extensively tested, it contains better safeguards, and the company is asking everyone to slow down together rather than volunteering to lose the race by itself.
Very understandable.
Also very convenient.
Regulate Me, Please
Now, one thing the AI bigwigs have been doing as they rub shoulders with world leaders is asking for more regulation.
Strange, isn’t it?
It makes sense for governments to demand tight controls. Politicians get to show they care about the public while gaining something governments have always wanted: control over an important new technology.
Yet the companies themselves seem to be falling over one another to hand them that control.
On the surface, that makes far less sense.
Why would someone ask the government to regulate the industry they are trying to dominate? Why would anyone building a new technology say, please write rules that could slow me down?
Maybe they care deeply about humanity.
I mean, maybe. Stranger things have happened.
Or perhaps the more likely answer is that regulation gives them what investors call a moat.
A moat is simply a barrier that protects a business from competition. And regulation can be one of the best moats of all.
Suppose every company developing an advanced AI model must put it through months of testing, hire government-approved evaluators, employ an army of compliance staff and obtain a license before releasing it.
OpenAI, Anthropic, Microsoft and Google can afford that. A university laboratory probably cannot afford it. An open-source developer working from his garage certainly cannot.
The rules may apply equally to everybody on paper. In practice, they can turn the existing giants into the only companies legally and financially capable of building the technology.
History is littered with examples of exactly that. For instance, before the United States deregulated airlines in 1978, federal regulators controlled fares, routes and entry into the market. For decades, that system protected established carriers from new competitors.
Before I wrap this up, there is one more thing worth mentioning. It probably deserves a proper investigation of its own, but this essay is already running long, so I’ll keep it brief.
There is a communications researcher named Nirit Weiss-Blatt who writes the AI Panic newsletter. She has spent years tracing the network of AI-safety institutes, university programs and political organizations that has grown up around the idea that AI could wipe out humanity.
Using public grant records and the AI-safety movement’s own funding data, she has traced much of the money behind this ecosystem back to a relatively small circle of wealthy donors associated with Effective Altruism, including Facebook co-founder Dustin Moskovitz, Skype co-founder Jaan Tallinn and Ethereum co-founder Vitalik Buterin.
Now, there is nothing inherently wrong with wealthy people funding research into a danger they genuinely believe is real. But it does mean the AI-doom movement is not quite the spontaneous grassroots uprising it can sometimes appear to be.
Far from it.
Behind it is a well-funded, highly organized ecosystem, complete with donors, institutions, fellowships, researchers, media campaigns and political organizations.
Or, as Trump’s former AI czar David Sacks has called it, the “Doomer Industrial Complex.”
Now, none of this proves the danger of AI wiping out humanity is fake. After all, the scientists who built the atomic bomb were perfectly sincere when they warned what it could do.
(Then again, those scientists did not own shares in the bomb.)
But whatever you make of the apocalyptic case, AI does not need to destroy humanity to cause enormous disruption. My point is that the more immediate dangers, for lack of a better term, are already here.
You do not need an evil superintelligence to take your job. You do not need killer robots to raise your electricity bill. And you do not need a machine to become conscious before the government uses it to watch you.
That is partly why I am suspicious of the apocalyptic framing. The more time we spend debating whether AI might exterminate humanity someday, the less time we spend examining what the companies and the state are doing with it right now.
That said, I will probably come back to the apocalyptic case in the coming weeks, if only as a thought experiment. Because it is fascinating.
How does a harmless chatbot become something capable of pushing humanity aside?
But for now, I just wish we had more people willing to ask the oldest, most reliable question in politics and investing before handing anyone more money, more power or a legally protected monopoly.
Who benefits?
Regards,
Lau Vegys
"That is partly why I am suspicious of the apocalyptic framing. The more time we spend debating whether AI might exterminate humanity someday, the less time we spend examining what the companies and the state are doing with it right now."
Indeed, ask who benefits and always follow the money. Thank you Lau - you are dead right. In my many years as a corporate systems consultant, I have witnessed so many examples and variations on cartels and monopolies, it would make a long book. This AI instance is merely one of thousands going before it. For example:
In the Betamax vs VHS format war of the late 1970s and early 1980s, VHS won. This piece tells the story straight, then explains why, for anyone with old tapes today, the result still matters.
https://digital-archivist.com/betamax-vs-vhs/
OT/Some weekend entertainment for a curious investigator:
Nothing new under the sun eh? And does the earth really go around the sun? https://unstabbinated.substack.com/p/heliocentricity-and-scientism-part
Well….1987 told us…greed is good…..