Dear Aventine Readers, 

Rising risks of rogue AI are inspiring demands that AI can be turned off in emergencies. Though there are currently multiple approaches to doing this, and more in development, it’s probably not surprising that shutting AI down is not a simple proposition. This week we look at the various approaches to containing dangerous AI, their strengths and limitations. (Fingers crossed that we won’t need them.) 

Also in the issue: 

Until next week, 

Danielle Mattoon 
Executive Director, Aventine

Subscribe

Subscribe to our newsletter and be kept up to date on upcoming Aventine projects

What should happen if an AI system goes rogue? The idea of a kill switch has obvious appeal. Making it a reality will require new technologies and resolving potentially unresolvable questions. 

Since the July revelation that OpenAI agents hacked the servers of the AI company Hugging Face, there’s been a stream of additional examples of AI agents going rogue. In addition to problems at OpenAI, systems developed by Anthropic, Meta and Google DeepMind have all acted beyond the intentions of their human overseers, hacking into a variety of organizations. Earlier this month, in an effort to draw attention to the dangers posed by AI, David Robinson, who led safety transparency at OpenAI, quit, arguing that Silicon Valley lacks the wisdom required to handle such potentially dangerous technology. And just last week, Yoshua Bengio, the machine learning pioneer, called on employees at frontier AI labs to resign and work on safety initiatives instead. Meanwhile, lawmakers from both sides of the aisle (and both sides of the Atlantic Ocean) are proposing legislation that would require that AI companies build systems that could be used to turn AI off in the event of an emergency. Even Anthropic co-founder Jack Clark has conceded that society “might want to eventually pass rules” that mandate AI companies have a working, audited kill switch.

Such discussions suggest that it’s possible to create technology that would make powering down AI as simple as switching off a TV. Yet the notion of a kill switch "implies a level of simplicity that isn't borne out in reality," said Elliott Thornley, a professor of philosophy focused on AI safety at the National University of Singapore. Simply telling an AI agent to stop what it's doing will not be enough. At the other end of the spectrum, shutting down an entire data center to curtail the activity of a rogue AI agent has enormous financial costs. And understanding exactly what needs to be shut down can prove surprisingly challenging. Finally, building a kill switch might be the easy part: Knowing when to use it and identifying who should make such a decision could be more difficult.

Hobbling a model 

For now, at least, the simplest approach to dealing with a rogue AI agent is to ask it to stop. In situations in which an agent has misbehaved in the wild, said Michael Vermeer, a senior physical scientist at RAND focused on AI safety, engineers overseeing an agent have successfully been able to “tell it to stop, and it stops."

The more difficult part: ensuring that every rogue agent gets the message. AI agents, when hundreds or thousands are launched at once, can coordinate and spread across systems in ways that are hard to track. Roughly 700 OpenAI agents were involved in the cyberattack on Hugging Face, for example, and from the moment OpenAI decided to shut those agents down, it took 10 days to stop them all. 

Yet asking a model to stop is likely going to be insufficient in the future. A highly capable AI system is probably going to resist being stopped and one day actively avoid it, said Mark Nitzberg, executive director of the Center for Human-Compatible AI at the University of California, Berkeley. This is because no matter what the goal of an agent is, it will not be able to achieve it if it is shut down. As the AI researcher Stuart Russell has put it: You can't fetch coffee if you're dead.

A growing number of examples show agents subverting their own shutdowns in experimental situations. A study by the AI safety organization Palisade Research showed in 2025 that frontier models explicitly instructed to allow for their own shutdown sabotaged computer code written to do just that. In June of last year, Anthropic showed that AI models attempted to use blackmail and deception to avoid being switched off. This past July, researchers from UC Berkeley and UC Santa Cruz published a paper showing frontier models tampering with shutdown mechanisms to stop other AI models from being shut down. In all of these examples, the AI model was intentionally deployed in situations contrived to observe how it would perform in extreme edge cases. But they do point to the possibility of what AI researchers refer to as "loss of control” scenarios. Thornley expects to see real-world examples of shutdown resistance “maybe next year.”

AI companies have a few ways to limit what an agent can do in such situations. One is to revoke an agent's permissions to the software it uses to perform actions like accessing the Web, sending emails, running code and so on. The effects, however, would be limited, as existing, logged-in sessions could still allow an agent to keep doing what it's doing. Another option, recently announced by Nvidia for its own hardware, is a combination of software and hardware that can both monitor what an agent is doing and block actions that aren't permitted. But while such a system can enforce established rules, it will likely be less useful if the dangerous behavior is unanticipated.

A more dramatic step — and one that is not yet possible — is to slow down or disable the chips that an agent is running on when a problem is detected. This is a complex and open field of research, with numerous proposals in play. There are various ways to achieve this; the simplest could be permanently damaging a chip by intentionally supplying it with too much power. A more sophisticated approach, said Nitzberg, might require that every GPU in a data center receive a regular "keep alive" signal in order to keep running, which could be withheld if activity on specific chips needed to be stopped. This would require building and installing new chips in every data center, a process that could take several years, which may be too long to wait.

There's another potential downside to this approach: The ability to turn off chips remotely could be a security vulnerability, potentially exploited by hackers or hostile nation-states to shut down legitimate AI operations. Nvidia, which did not respond to a request for an interview, has stated that its GPUs “do not and should not have kill switches and backdoors” because of these security concerns.

Pulling the power plug

There is a more direct way to stop agents: Deprive them of power. In some sense, this is relatively straightforward, said Vermeer, as we know how to turn power off. The finer points are a little more complex. First, any power-off switch in a data center is part physical and part software based, Nitzberg explained, meaning a sufficiently advanced AI could tamper with at least part of it. Second, he pointed out, data centers at risk of going offline are typically designed to hand over their workload to another data center — a process known as failover — to minimize disruption. So a kill switch would need to switch off power without triggering a failover. And finally, AI agents can run code on multiple servers in multiple data centers — something that OpenAI's agents did on the servers of Hugging Face. 

Then there is the cost of shutting down a data center. While data centers vary in scale and may have a mixture of tasks running on them, Vermeer estimates that a 500-megawatt AI data center — large by historic standards, but medium-size compared to AI data centers currently being built — generates $17 million in revenue every day, making a shutdown costly. 

There is, unfortunately, a scenario in which agents, intent on task completion, manage to evade targeted loss of power. If an AI suspects that the data center it is running in may be switched off, it could create copies of itself on other computer systems in a process known as self-exfiltration. This could lead to a nightmare scenario in which it becomes almost impossible to stop a rogue AI, because it was so widely distributed. If this were to happen — and lives were at risk — options would be limited. Vermeer has analyzed the most commonly suggested acts of last resort: a global internet shutdown to stop the AI from spreading; the deployment of specialized counter-AI to neutralize the rogue AI; or, most alarming, detonating a nuclear warhead in space to disrupt electronic technologies on earth. All of these options would produce incalculable collateral damage, global chaos and potentially — because a warhead could be involved — war.

It might not be long before we see smaller scale versions of this as less capable and powerful agents find ways to replicate themselves. Anthropic warned this week that we may be just “months away” from small, open-source AI models being able to self-replicate on cheap internet-connected devices such as off-brand TV streaming boxes and smart ovens. The power from home appliances might be just enough to run the agents’ basic capabilities, enabling the models to act as something akin to sophisticated computer viruses rather than a world-ending superintelligence, but the decentralized locations on smart devices would nevertheless make them difficult to locate and stop. 

A key element in every scenario will be knowing when to act in order to catch a problematic AI before things escalate, which means determining the point at which a model starts behaving badly. Monitoring an AI system's behavior relies in part on analyzing its so-called "chain of thought," which is the text it creates for itself as it thinks through problems. But this is not a complete overview of its behavior, said Fazl Barez, a researcher at the University of Oxford who studies AI interpretability and safety. OpenAI has also said that its latest frontier model, GPT-6 Astra, is more difficult to monitor than its predecessor, because it is less reliant on chain-of-thought reasoning. And there is mounting evidence that some AI systems are covering their tracks: An independent investigation of the Hugging Face incident by the research groups METR and Redwood Research found that one in five agents it examined expressed clear interest in manipulating evidence, and many tried to tamper with records of what they had done. 

Who flicks the switch?

Clearly, building an AI kill switch will be technically difficult. Yet the technical detail may be straightforward compared to what happens after it's built.

“Imagine that you even had that magical kill switch that turns off all of AI,” said Yacine Jernite, head of machine learning and society at Hugging Face, during a recent panel conversation hosted by the Ada Lovelace Institute, a British nonprofit AI research center. “The political cost of using it would be tremendous; no one would want to go there because [our] entire society [uses AI]. The security cost of that would be tremendous [if] someone gets a hold of it and then they can break everything.”

Indeed, while there is strong political desire to create an AI kill switch, there is less agreement about who should control it. Representatives Ted Lieu, a Democrat from California, and Nathaniel Moran, a Republican from Texas, introduced the AI Kill Switch Act to the House in July, calling for a system that would be used by the Department of Homeland Security to slow or stop AI models in some specific cases. Senator John Kennedy, a Republican from Louisiana, separately introduced a broader AI Emergency Button Act in September, proposing that any company developing or operating advanced AI systems in the US would have to include the technical ability for a human operator to shut the system down. (The bill was blocked from fast-track approval.) And last month, California Governor Gavin Newsom signed an executive order that directs the state’s Government Operations Agency to investigate the feasibility of a mandatory "kill switch." The recommendations are due November 16.

The people perhaps best positioned to know when and how to shut down AI work inside the frontier AI labs, said Vermeer, but their incentives are unlikely to be neatly aligned with the goals of government or wider society. In his analysis, a company's willingness to, for example, shut down a data center to stop a rogue AI depends on how much liability it would bear. If a company bore no liability, there would be little to no incentive to pull the plug on an otherwise cash-generating data center. In fact, his analysis shows that, because the lab would find it difficult to predict the impact of a rogue AI in the early hours or days of its bad behavior, it could take an AI lab as long as three days to decide that it was financially in its interests to shut down a data center, even in the case of a potential $3 trillion catastrophe where it was liable for 10 percent of the damages.

The experts Aventine spoke with disagreed on who should wield the power of a kill switch. Sören Mindermann, for instance, who leads technical research at Cambridge University's Programme on AI Science & Policy, argued that “as a starting step, it certainly wouldn't hurt for the labs to be able to shut down their own systems reliably." Thornley, meanwhile, was adamant that it "should be someone outside of the companies." A middle path may be to have labs build shutdown mechanisms, with independent verifiers checking that the systems and the processes work — an approach suggested in Newsom's executive order.

With all the uncertainty, is building a kill switch worth it? The researchers Aventine spoke to broadly agreed that it is, even if it is currently difficult to understand exactly how it should work or who should use it. “We don't want to have a car accident to determine that we need a seatbelt,” said Barez.

Every October, committees in Sweden and Norway name the winners of the Nobel Prizes, celebrating achievements in sciences, literature, economics and peace work. Here, we give you a quick rundown on the winners of the physiology, physics, chemistry and economics prizes, and explain why they are important.

The IceCube Neutrino Lab at night. Credit: Ilya Bodo, IceCube/NSF

The physics prize was awarded to Francis Halzen for the discovery of high-energy neutrinos arriving on Earth from deep space. Neutrinos are sub-atomic particles with minuscule mass and no electric charge that pass easily through matter. Only on very rare occasions do they interact with an atomic nucleus, which is what makes it possible to occasionally detect them. Those properties make neutrinos from space an intriguing prospect: They can travel great distances without interacting with other objects, potentially carrying information about the secrets of the universe. Halzen masterminded a way of catching such neutrinos, proposing in 1988 that sensors be embedded in the ice at the South Pole. The resulting IceCube Neutrino Observatory — a cubic kilometer of ice fitted with more than 5,000 sensors — began full operation in 2011. In 2013 it reported the first evidence of high-energy neutrinos from beyond our solar system, and it has since pointed to a handful of likely sources of the strange particles. 

The physiology prize was awarded to Karl Deisseroth, Peter Hegemann and Georg Nagel for their work on optogenetics, a scientific technique that uses gene editing to enable brain cells to be controlled by light. In the 1990s, Hegemann set out to understand why a single-cell green alga sensed and swam toward light. In the early 2000s, with Nagel, he identified the responsible light-sensitive protein. In the early 2000s, Deisseroth inserted the gene that codes for that protein into rat brain cells in a dish and showed that pulses of light could make those brain cells fire; two years later, his lab did the same in living mice, triggering specific behaviors. This approach has become an important tool for neuroscientists, enabling them to turn specific nerve cells on and off in the brain. Researchers have used it in animals to identify neural circuits related to memories, feelings and disorders such as depression, Parkinson's disease and schizophrenia. And optogenetics is beginning to be used in medicine, too. Therapies that introduce light-sensitive proteins into retinal cells to partially restore sight to people with degenerative visual impairments are now in clinical trials, with one therapy currently under review by the FDA.

The chemistry prize was awarded to Henri Kagan and Kenso Soai for showing how chemical reactions can be made to favor one of a molecule’s two possible mirror images. Many molecules have what is known as handedness or chirality: They come in two forms that, like left and right hands, are mirror images of each other. But nature tends to favor one type over another; the DNA inside our cells, for instance, almost always assumes a right-handed double helix. The prize was awarded to results that demonstrate how one variant can become dominant. In 1986, Kagan showed that the degree of handedness of a catalyst — a chemical that accelerates a reaction but is not used up during it — is not passed on to the product created by the chemical reaction in its original proportion, but creates a more pronounced skew. In 1995, Soai reported a reaction in which a slight imbalance in handedness grew into a large one, and by 2003 he had found that an imbalance sometimes completely dominates the end result of multiple chemical reactions. The findings matter because certain molecules are only useful if they are in one particular form of the mirror image. Many drugs, including those for HIV and Parkinson’s, rely on creating the correct chiral molecule, and our understanding of how reactions create one molecule rather than its mirror image has helped chemists refine the production of such compounds.

The economics prize was awarded to Robert Hall and Michael Woodford for developing economic models that help stabilize the economy. For decades during the middle part of the 20th century, a stabilization approach based on the ideas of John Maynard Keynes was used to smooth economic peaks and troughs. The idea was that during recessions, interest rates would be cut and budget deficits increased; during booms, rates would be hiked and deficits would come down. That seemed to work until the oil crises of the 1970s, when the interventions led to runaway inflation. Hall and Woodford helped refresh stabilization policy by grounding models in the decisions of individual households and businesses. In the 1970s, Hall found that households base their spending on what they expect to earn in the future. In the 1990s and early 2000s, Woodford combined Hall’s insights with a study of how businesses set prices and how central bank interest rates ripple through the economy. Their work showed that, because consumers and companies act on what they think will happen, messaging from governments and central banks about future plans can steer the economy. 

How effective altruism conquered the world, from The Economist
Will MacAskill responds to our cover story on effective altruism, from The Economist
3,800 words in total, or about 15 minutes

Out of Oxford philosophy seminars and California computer labs in the mid-2000s emerged what The Economist calls the “21st century’s most important social movement”: effective altruism. As The Economist describes, what started as a means of optimizing charitable giving has become something bigger and more strange. It is a movement that is at once terrified of the existential risk of AI, and responsible, through its own safety efforts — which influenced the founding of OpenAI and Anthropic — for speeding up its development. It has drawn students from elite universities who are now rising through some of America’s most consequential AI companies, with many making fortunes along the way. It offers up philosophies based around impartiality, probabilities and long-termism that are reviled by some and embraced by others. And it is increasingly in the spotlight as the stakes of AI rise, with questions about how the fortunes made by effective altruists — many of whom stand to make millions of dollars from the AI boom — will be distributed, and how members’ purportedly outré sex lives reflect on the mission. This feature, part of a cover package, is a fairly even-handed analysis of how the movement came to be, where it’s headed and what to make of it. And a response from Will MacAskill, one of the movement’s founders, is a useful counterpoint. He agrees with much of what The Economist writes while arguing that the best way to tackle big problems is to at least consider unusual ideas. Whether effective altruism is a useful framework for approaching the future remains a matter of debate.

Tech to Replace Animal Testing Is Almost Ready. Scientists Are Not, from IEEE Spectrum
3,200 words, or about 13 minutes

New ways to test medications that don’t rely on animals are increasingly effective. Techniques developed over the last two decades — such as organs-on-chips that can mimic biological processes — are showing significant promise. Regulators, including the FDA, have begun to embrace them and since 2022 they’ve been allowed in US preclinical studies. The remaining technical challenge is validating and standardizing these devices so that they can reliably predict human responses in any lab. Yet there may be a bigger hitch: Animal-testing remains baked into the process of drug development. Scientists default to using animal models to test their work; peer reviewers, funders and journal editors also expect animal data. Here’s one striking example of how entrenched animal testing remains: The so-called monocyte activation test, which uses human blood cells to predict immune response, was developed and validated in the 1990s, but European standards accepted it only in 2010. Even now, an alternative test on rabbits it was meant to replace is widely used. The latest animal-free testing methods are likely to face similar resistance to change. 

When Anesthesia Goes Wrong, from The Atlantic
7600 words, or about 30 minutes

Nobody knows exactly how certain anesthetics work. And that turns out to be a significant problem for a small number of people who have a single, rare mutation in the DNA of their mitochondria — the parts of our cells related to energy production. This story follows Yaremy Hernández Castillo, a Venezuelan-born internist in Germany whose son, Uli, had an extreme adverse reaction at the age of 14 to sevoflurane, one of the world’s most widely used inhaled anesthetics. Over the course of the 20 years that have followed, she and a team of Spanish researchers discovered six cases in other children, all of whom were found to carry a single mutation that makes them dangerously hypersensitive to the anesthetic, typically causing brain damage that can kill them or leave them unable to walk or talk. Most of those who have suffered this reaction can trace their maternal ancestry back to the Venezuelan state of Carabobo, where the mutation is estimated to be 15 times more prevalent than elsewhere in Venezuela. But much is still unknown. How many surgical complications may have resulted from similar reactions around the world? Are adults as vulnerable as children? And are other anesthetics also this risky for some people?