Dear Aventine Readers, 

By now you've heard about the calls to slow AI down from the people most responsible for speeding it up over the last three years. On its face, a slowdown seems like a reasonable idea to consider, but there's plenty of skepticism out there, all reflected on Substack.  Whether it's doubting the stated motives of the AI companies calling for a slowdown or questioning whether the US is even capable of imposing the necessary oversight, writers have been weighing in. We round up the arguments. 

Also in the issue:

Thanks for reading,

Danielle Mattoon
Executive Director, Aventine

Subscribe

Subscribe to our newsletter and be kept up to date on upcoming Aventine projects

Earlier this month, it seemed that we had come to a collective agreement about the state of AI: It was time to slow it down. 

In the wake of hacks by poorly managed AI agents, the release of more capable yet less monitorable models and the resignation from Anthropic by an engineer convinced AI is an existential threat to humanity, Dario Amodei published an essay calling for an AI slowdown with a suggested a framework for how to make it happen. Soon after, OpenAI CEO Sam Altman, Google DeepMind chairman Demis Hassabis and SpaceXAI CEO Elon Musk all expressed agreement with Amodei’s general view. 

The moment was fleeting. In the hours and days that followed, the proposal has been picked apart and the CEOs’ motives questioned. How can effective regulation be based on such a vague plan? How much self-interest lies behind the calls to pause? Is the US government interested in — or even capable of — overseeing such a slowdown? And if the people building this technology are so worried about its capabilities, why don't they just slow it down on their own? 

All of this and more has been expressed on Substack over the past few weeks, where writers reflect growing concern over AI safety but little consensus over how to achieve it. Everything and nothing, it seems, is on the table. 

Can we trust the messenger? 

Skeptics have no shortage of reasons to question the doomer narrative and the motives of AI lab CEOs in spreading it.

Matt Stoller, who writes about market power and antitrust on BIG, argues that frontier labs are burning cash and an industry-wide slowdown inspired by fears of end times would allow them to collectively cut spending and improve their bottom lines. Seth Dobrin, who writes about AI investment on Silicon Sands, argues that the weight of regulation described by Amodei — with its embedded evaluators, regular communication with the government and extensive use of AI to monitor AI — is a significant operating cost that will be a larger burden on small insurgents than on the established companies pushing for a pause. “Fixed costs favor scale,” he writes. “The proposal is a compliance regime that a $965 billion company funds from petty cash and a $5 billion company funds from its Series B.”

Ben Tarnoff, who writes with a leftwing bent about tech at Metal Machine Music, points out that the firms also benefit when a focus on AI doomerism eclipses the technology’s existing harms — like  deepfakes, misinformation, AI psychosis and surveillance — by drawing public debate toward bigger, scarier and yet more distant threats. Regulation on current-day issues would likely dent the revenues of these companies sooner than rules about how to handle a rogue superintelligence.

But others, including Derek Thompsom, Noah Smith and Zvi Mowshowitz of Don’t Worry About the Vase believe that the CEOs are sincere in their expressed alarm. (Though, as Brian Merchant, writes in Blood In The Machine, the two positions aren’t mutually exclusive: The fear can be sincere and good marketing at the same time.) And Thompson lays out a rebuttal to those who ask: If the leaders of AI companies are so worried, why don’t they just slow down the pace themselves? There are a few potential reasons, Thompson writes. One is that the companies don't trust each other enough, and fear that others will race on if they slow down. Another, as Pranav Bhasin points out on his Substack Future Proof, is that a coordinated slowdown could be considered an antitrust violation in that it would be an agreement among competitors to restrict output — something the Sherman Act was written to prevent. A third is geopolitical concern: China won't slow down on building AI even if American labs do, so pausing without an international accord wouldn't be in the interest of the US.

A lack of governance

Some of these issues could be addressed through the government intervention Amodei is asking for, though an obvious obstacle there is President Trump, who does not see rogue AI agents as a serious threat and described efforts to paint them as such as a hoax. Yet there are other arguments for why it might be difficult to regulate AI in the US. Sayash Kapoor and Arvind Narayanan, who write a Substack called AI as Normal Technology point out that some of the government regulatory machinery is being defunded: The U.S. Cybersecurity and Infrastructure Security Agency, for example, is set to lose about $700 million, or roughly 30 percent, from its 2027 budget and has already seen its staff cut by a third. Alex Chalmers, writing on the Substack of the Cosmos Institute, which works on building AI for human flourishing, argues that calls to pace frontier AI development are dangerously vague. In asking the government to regulate an AI slowdown without explaining how, he argues, advocates are leaving such questions to whomever ends up wielding the power — potentially agencies, such as the Pentagon, whose priorities of military advantage and strategic advantage may not align with AI safety.

So maybe energies would be better spent encouraging international action. Scott Galloway, in his No Mercy/No Malice newsletter argues for “an international artificial intelligence agency (IAIA) modeled after the International Atomic Energy Agency,” writing that we could start first with “a U.S. regulator similar to the Department of Energy that records incidents, approves new models, promulgates safety rules, and has subpoena powers.” AI researcher Gary Marcus, meanwhile, has also called for international organized oversight, which he said should “evaluate the risks versus benefits of new systems before they are introduced” and “audit new systems after they are released, on an ongoing basis” He also proposed an “international agreement to not deploy architectures that are demonstrably harmful.”

What about laws we already have?

One idea around which some consensus does seem to be forming on Substack is that the US should be making better use of the tools it already has. Sean Speer, a center-right commentator and public policy analyst, argues on the City Journal Substack that tort law could be employed “when an autonomous system causes financial losses, hacks another system, or otherwise harms third parties.” Computer scientist Melanie Mitchell, meanwhile, wants companies held accountable when products fail through negligence or badly designed training. Merchant thinks that making AI companies responsible for automated lawbreaking would go a long way toward preventing AI doom. 

Liability is obviously of little comfort in the face of possible human extinction: It can deter risky behavior but it can’t compensate for irreversible catastrophe. And not everyone thinks it will be an effective solution. Dean Taylor, who usually writes about AI’s impact on the legal industry, argues that it could be hard to impose liability on AI firms for the same reason it’s hard to impose liability on knife makers for stabbings, because the tools they make are general purpose. And Marcus, in a subsequent post, argues that while liability could be useful, it shouldn’t detract from wider efforts to regulate the AI industry — we must do both. He also contends that doomerism is going too far: There are lots of problems with what these companies are doing, but we don't need to worry about human extinction.

There could be a bright side to the current state of panic and confusion. For Kapoor and Narayanan, recent events are evidence for what they call the “continuity hypothesis”: that dangerous capabilities reveal themselves through smaller warning shots before they become catastrophic. Poorly controlled AI agents may have done things we didn't want them to, but so far they have caused no serious harm, and were clumsy at covering their tracks. That the reaction across the tech industry, politics and the media has been so intense is useful, they argue, because it creates an opportunity to act while the harms are still small.

Perhaps this is where a pragmatic middle ground lies. You don't have to believe that superintelligence is about to wipe out humanity to think AI companies should be liable for what their products do. Underscoring the need to do something (albeit vague), on the sidelines of the UN General Assembly this week, a coalition of 22 countries signed a letter calling for control of frontier AI models, so that they “remain under human direction, oversight and control.” 

Why can’t technologists and economists agree on AI?, from Asterisk

Technologists' views on the economic impact AI will have on the economy tend to be wildly bullish, projecting that AI will boost economic growth by between 3 and 30 percent in the next 10 years or so. Economists’ estimations, on the other hand, are much lower — somewhere between 0.1 and 1.5 percent. It would be enormously helpful to know which side is more likely to be right, or at least for the two sides to find a compatible middle ground, but for years both have refused to budge. Matt Clancy, from the philanthropic funder Coefficient Giving, explores the roots of the disagreement in an attempt to find a way around it, ultimately coming to the conclusion that arguments from both sides are simply too weak to convincingly shift the other.  And what drives each camp's intransigence, he argues, are collective lifetimes of siloed experience shaping different views of how the world works. Economists learn that growth is famously hard to shift, that markets already act as a kind of collective intelligence no individual can outsmart, and that change tends to happen at the margin. Technologists watch companies grow explosively on the back of new ideas, and learn to think in terms of paradigm shifts rather than increments. Is it possible to find a middle ground that reflects the experience and insights of both sides? Perhaps. To do so, he writes, we should pay more attention to people who straddle the two worlds — PhD economists who go to work at tech companies, say. And it might also make sense for each side to spend more time examining why they think as they do and communicating that to the other group, rather than falling back on the evidence that has so far failed to convince the other side.

Escape Velocity, from Threading the Needle 

Earlier this year, we explored the idea of data centers in space and concluded that it wasn't as far-fetched as it first might seem. Here, Anton Leicht, a fellow at the Carnegie Endowment for International Peace, goes a step further, asking: What are the policy implications of putting AI data centers into space? One seems to be a loss of control for government oversight of AI. While laws do extend to space, it’s harder to intervene with physical infrastructure in orbit than, say, with a data center in Virginia. That, argues Leicht, means governments should want a robust, remote "kill switch" in place before tech companies launch data centers in space at any scale. Another is that the nexus of control over AI within governments would shift so that, in the US for instance, the Federal Aviation Administration and Federal Communications Commission might need to play a significant role in policing the technology. Geopolitically, space data centers would make the ability to launch rockets into space an incredibly valuable asset, so nations with that capacity, or with geography particularly suited to launching satellites into certain orbits, may find themselves with increased power. Finally, and most provocatively, he argues that orbital data centers might mean that more countries build anti-satellite weapons in order to shut down a rival nation’s offensive AI capabilities — which might, in the long run, make space data centers a new consideration in geopolitical risk.  

Industrial Overcapacity Enables Scientific Discovery, from The Republic of Science

This is a fascinating story about how found or borrowed industrial equipment has been put to great scientific use by people for whom it was never intended. In the 1940s, a Rockefeller Institute scientist used New York City's only electron microscope, owned by a paint and ink manufacturer, to produce the first image of an intact cell, an advance later regarded as the birth of cell biology. In the 1950s, the UK's Met Office, which had no computer and no money for one, ran its first computational weather models during the weekends, using a machine belonging to a catering company. And in 2020, Facebook's AI lab generated vast chemistry datasets during idle time at its data centers, giving researchers at Carnegie Mellon a faster way to search for new catalysts. All of which is to say: Scientists seem adept at making use of any scraps industry can offer.

Distributing AGI's wealth worldwide is a very tricky problem, from Transformer

Jacob Schaal, who researches the economic impacts of AI at King’s College London, lays out why it will be difficult to redistribute wealth from AI-generated fortunes in the future. There's certainly no shortage of ideas for how it could be done, but all of them, he argues, have drawbacks. Sovereign wealth funds would let governments take equity stakes in AI firms, but that’s likely an offer that would extend only to nations like the US and China. Universal basic income, the most commonly cited solution, looks like it could work well in the poorest countries, where people are more cash-constrained, but a global UBI would require a level of multinational governance that doesn't exist. Universal basic capital, by which everyone gets a slice of the economy at birth rather than a stream of payments, suffers similar problems, and also has a mixed real-world record. Then there are ways to share the wealth between governments: benefit-sharing, say, with the US offering computing power or cash in return for commitments not to smuggle AI chips to China; or taxation of goods (like luxury clothes) and services (like travel) that AI-rich countries will want regardless of price. But in both of those cases, wealthier countries benefit more. Ultimately, Schaal argues, it may fall to philanthropy to take the first steps in building out wealth redistribution.

Where Has Construction Automation Been Successful? from Construction Physics

The construction industry is incredibly labor-intensive. This post dives into how automation efforts have tried, with varying degrees of success, to change that through automation. The main finding is that automation has only really had an impact on two areas of the industry. The first is off-site manufacturing: things like steel fabrication, precast concrete, window manufacturing and so on, for which it's possible to create assembly-line-style processes. The second is construction-site tasks, but only when the processes involved are incredibly uniform: consistent paths to work along, for example, or repetitive motions that don’t change. A good example is extruding concrete to form pavements or curbs. On the other end of the spectrum, a prime example of why construction is so hard to automate is bricklaying. One would think the process — repetitive and seemingly predictable — would lend itself to automation, and there have been many attempts to automate it. But the reality of bricklaying is far more complex. The mortar between bricks is highly variable, and most buildings have many corners, doors, windows and other details that interrupt what would otherwise be a continuous wall. In other words, what seems like a repeatable process is fraught with variations, all of which make automation difficult. But innovation might be near: Modern AI is giving machines like welding robots the ability to adapt to unpredictable conditions in a way they never could before.