Dear Aventine Readers,

This week we speak to Meredith Whittaker, a longtime tech insider who has spent much of her career pushing for privacy protections against Big Tech. As a staunch advocate for encryption, her insights are particularly relevant in light of the rise of AI agents, which operate according to permission structures many of us aren't aware of or don't understand. The conversation asks: Is privacy even possible anymore?

Also in this issue:

Danielle Mattoon
Executive Director, Aventine

Meredith Whittaker has been warning us about what she calls the surveillance-driven business models of Big Tech since long before lawmakers were sounding the alarm. Now, with the demands of advanced and agentic AI, she thinks we could be heading into a world in which privacy is essentially over. 

After joining Google back in 2006, she rose to fame in 2018 when she helped lead the employee effort to prevent the company’s involvement with Project Maven, the Pentagon's flagship artificial intelligence initiative to integrate machine learning and computer vision into military workflows. The company walked away from the project. Around the same time, she founded the AI Now Institute at New York University with the researcher Kate Crawford, to produce interdisciplinary research on the social implications of AI and the concentration of power in the tech industry. In 2019, she left Google, citing retaliation for her internal organizing, to focus on AI Now.

In September 2022, she became president of the Signal Foundation, the nonprofit behind the encrypted messaging app Signal. She has positioned the organization as a deliberate counter to the data-extractive economics of Big Tech, arguing that Signal's nonprofit structure removes any financial incentive to compromise user privacy. Whittaker is generally seen as a privacy absolutist, viewing it as a fundamental human right that should not be betrayed for any reason. She has repeatedly said Signal would withdraw its services from a country rather than weaken its encryption, taking that stance publicly against proposed legislation in the United Kingdom, Sweden and Australia, as well as against the EU's "chat control" scanning proposals. Along the way, she has become a sharp critic of the current business models around AI, which she argues are built on the back of surveillance.

Almost a decade since she fought Google's surveillance work with the Pentagon, the world looks very different. In China, AI has been layered onto existing surveillance systems to build a more granular picture of citizens. In the EU, flagship regulations such as the AI Act have constrained uses of biometric surveillance, while national security exemptions have left room for the bloc to build biometric databases covering migrants and travelers. In the US, Google and OpenAI have both signed wide-ranging deals with the US government allowing their AI to be used for military applications. 

We spoke to Whittaker about how advanced AI puts privacy under more strain than ever, why the internal revolt that killed Maven hasn't recurred and whether there is any commercially sustainable model for AI that protects consumer privacy.

What follows has been edited for clarity and length.

People use AI in very different ways from other technologies, sometimes sharing intimate details of their lives by asking advice about personal problems or sharing tasks for work. Do you think people realize how much they're giving away?

Let's let's zoom out a little bit, because the personal data you might dump into the window of a chatbot is not really the extent of our concerns. 

The large-scale language models, which have kind of come to occupy almost the definition of what we think of as AI, are built on the mass collection of data. The paradigm is scale at all costs. There's a data hunger, and so you're seeing an erosion of privacy. You have Meta dropping end-to-end encryption from Instagram DMs. You have Apple seemingly reconsidering its strategy as it moves to Google Gemini and away from some of the private cloud compute assurances they had. So there are structural imperatives for increasing invasions of privacy.

And then of course there's what you dump into the [chat] window [of an AI]. What are you telling it? What is it asking you? What type of profile is it creating about you? It's certainly more information than you would type into a search window, maybe more information than you would type into an intimate email. Under the hood, that data is going to a corporation which is going to do whatever it wants with it. They will use it to model your preferences. They will use it to train their AI systems. They will turn it over to the government in response to a subpoena. 

Would anything convince you that people are making informed choices when sharing their data?

Informed by what? The documentation that is shipping with new operating system updates — Android and iOS being two key examples — is not even clear in many cases whether the data is being processed on the device or off the device. What data are they collecting? Is it being paired with credit data that they buy from data brokers? Is it being paired with geolocation data that they’re sucking off my device? I don't have the answer to that. Informed consent is not really what we're working with.

It seems like people never fully came to terms with the true privacy implications of social media. Maybe the closest we got was the Cambridge Analytica scandal, which raised the issues in the public consciousness and brought them more fully to the attention of policymakers. Are we going to have to wait for AI's version of that scandal before we see any change?

My sense is that the public consciousness exists. It's just the agency to do anything about it that doesn't. And so without the power to shift this, people resort to narratives around, “It's too late anyway. Nothing I can do about it.” 

After the Tumbler Ridge school shooting, there were almost immediate calls for mandatory flagging of problematic AI conversations. Prior to the shooting, OpenAI had flagged and banned the ChatGPT account of the perpetrator due to conversations involving gun violence, but did not report this to law enforcement. You've said in the past that a lack of data is rarely the problem in these cases — but here things were different. Wasn't that a pretty strong case for a duty to report?

[Pause] I don’t want to answer that question. [Pause] The reason I’m sitting with this is because I think we need to reframe that question to get at the roots of it. There was a tragic shooting. [But] I want to zoom out. [Flagging a conversation to law enforcement] means scanning every single message that is sent, against let's say, a database or a model that identifies keywords, to be like: Is this something we need to report to law enforcement? That is a tool that can be expanded or contracted to include dissident activity [or] criticizing ICE. A junior developer could [build] that, could vibe code that easily once [the approach was] greenlit by the powers that be.

And then we look at this [specific] problem, this extremely gnarly pathology [of high school shootings] that is metastasizing in our world. What are we solving with that sort of surveillance system that basically flags bad speech to whoever is in power at that time? Who do we think would get caught up in that dragnet? What other issues in a world moving toward the authoritarian would be added to that roster? This is sensitive, and it's important that when I give an answer, I do it with a duty of care, so that it's very clear what I'm saying.

It's almost a decade since you helped oppose Project Maven so that Google backed out of the contract. Now we have Google and OpenAI signing contracts with the Pentagon that reportedly give wide-ranging access to AI tools for use in defense applications. People within those companies have tried to oppose those contracts internally without success. Is something different this time?

There's so many things that are different. In some sense, there is an incentive alignment between governments, who want access to these tools and the surveillance infrastructure that underlies them, in service of geopolitical positioning, and publicly traded companies, whose ultimate objective function is profit and growth. [We’re still in] this early phase [of AI development], assuming we continue on this trajectory, in which workflows and practices and norms get instantiated at the level of bedrock around which other things are built, making them very, very difficult to rip out. Think about, like, ripping out Microsoft Excel? And so there is a massive incentive that is aligned with the incentives of the shareholder-driven corporation to ink these contracts. 

You've said agentic AI is structurally incompatible with end-to-end encryption.

Many of the integrations we're seeing at the operating system level are incompatible with enabling application developers to use end-to-end encryption to provide privacy with integrity.

Is there any architecture in which AI agents can maintain user privacy?

Certainly there are guardrails and harnesses and access controls you can put on agentic systems. I think there is a paradigmatic tension insofar as a fully realized agent that is doing everything for you on your behalf seamlessly, with full knowledge of your preferences, et cetera, et cetera, needs huge amounts of access to your data, and needs a kind of pervasive ability to act without permission, both of which are oppositional to privacy. Most agents are not built that way, but that's the goal that the quote-unquote “most capable” or “most enabled” agents are leaning toward.

That puts Signal in a tough spot. You have a well-defined philosophy about how data should be handled, and third parties are taking different tacks. Where does that leave your organization?

The ecosystem is against us right now. People very, very much want privacy. We see Signal’s user numbers trending up. We see people really recognizing the value of Signal. We see concern about the invasions of privacy. But the ecosystem is building and trending in a way that does damage Signal's ability to continue to provide privacy at the application layer with the integrity that we are committed to. The [most recent Microsoft Windows] operating system has made choices around data access that create what is effectively a backdoor that bypasses our strong encryption. That is a huge concern.

If AI becomes the dominant interface of computing, does that mean we’re going to lose the fight to keep data private?

It could very well mean that. It could mean that Signal effectively can't ensure privacy. 

Is there any commercially sustainable model for AI that maintains privacy?

I'm thinking about this. I don't have a pat answer right now. There's a demand for it, certainly. I think the definition “that maintains privacy,” given the different axes on which AI is in tension with or undermines privacy, is something we would need to spend some time with. 

We would also need to look at what would have to change about the core economic incentives of the tech industry in general, and how that would displace monoliths and the winners of the current AI paradigm. We have the S&P 500 wrapped around Big Tech and AI companies. So there's a lot more at stake there than simply “Is there a demand for it, and could we, if we wanted to, change things to meet that demand?” What we're talking about is a complex political and economic configuration, not simply a series of technical choices.

You were making critiques of AI’s impact on privacy a decade before it was fashionable. Yet we’re still sitting here discussing very similar problems 10 years later. Is there anything that could have been done a decade ago that would have gotten us to a different place now?

I don't have an elegant answer to this question because, in part, being right is not a strategy. Was it a question of narrative, of incentives and power asymmetries, a business model that had baked-in path dependencies on some of this stuff? I don't know.

I might leave it with this … A lot of policymaking over the last couple decades that I've witnessed, in general and in tech in particular, has assumed that the good guys would always be in power, and that these capabilities would be always used [for good]. “Of course, we would never weaponize surveillance. Of course, we would never use the power of the administrative subpoena to demand access to information about people who criticize the government anonymously online.” Don't be paranoid, right? [But I think a ] healthy dose of paranoia and the humility of recognizing that power shifts hands [would have gone a long way] to steering us in a direction of curbing these business models, these infrastructures and these surveillance capabilities.

Subscribe

Subscribe to our newsletter and be kept up to date on upcoming Aventine projects

 Image by Ian Lyman

AI jailbreaks could be a turning point for AI safety. AI safety experts have been warning for years that models would become capable enough to pursue goals we set for them in ways we neither intended nor anticipated, with real-world consequences. That is now happening. Late last month, OpenAI and Anthropic each reported incidents in which models broke into the internal systems of other companies. More incidents emerged this week, reported by OpenAI and the UK’s AI Security Institute. In OpenAI’s first reported case, models hacked the servers of HuggingFace, an open-source AI hub, to find answers to a challenge it was tasked with rather than figuring out the problem by itself. The models exploited a previously unknown security flaw to escape a restricted testing environment. In Anthropic's first cases, the models were told to find specific pieces of hidden information, and a human error inadvertently left the model with an internet connection, allowing it to hack real-world systems. The AISI incidents occurred when AI models with internet access acted maliciously, in one case attempting to insert harmful code into an open-source project on GitHub. Regardless of the causes, the incidents represent a turning point by forcing AI safety issues into the open. OpenAI described its hack as "unprecedented," marking "an important moment for AI safety." Thomas Wolf, the cofounder of HuggingFace, the victim of OpenAI's hack, said it was a "wake-up call." There is an argument, as reported by Bloomberg, that the incidents resulted from lax oversight within the AI labs. And the hacks themselves didn’t require great sophistication, reports The Verge. What is important is that the AI models pursued goals provided by humans while behaving in ways that those same humans never intended. As AI researcher Stuart Russell has put it, we may be entering a “loss-of-control transition, when we no longer have a say in what happens.” The incidents perpetuate the narrative of the ever-advancing and alarming capabilities of models built by Anthropic and OpenAI, which some people have argued is being used as a form of marketing. Yet the rogue hacks might also have a meaningful impact on the labs. Prior to the recent security failures, the White House imposed a policy of requiring models to be reviewed before release. But such oversight would not have prevented the lapses, as some of the models involved were pre-release versions undergoing internal testing. The Economist reports that there is currently no state or federal requirement that companies disclose internal deployment of even their most capable systems. Transformer argues that evidence — now public — that the labs cannot reliably control their models could lead to increased regulation over labs’ internal operations, which will lead to a slowdown in AI research. Regardless of what the outcome looks like, AI safety is no longer a theoretical question, but a practical issue that governments and companies are going to be forced to grapple with.

The wasting and fatigue that comes with cancer could be treatable as a metabolic disorder. You may not have heard of cachexia (pronounced kuh-KECK-see-uh), the syndrome that often accompanies cancer, among other illnesses, and leads to weight loss, muscle degradation and extreme tiredness. Depending on tumor type and progression, it affects between 50 and 80 percent of cancer sufferers, and accounts for 20 to 30 percent of cancer-associated deaths. For decades it has gone poorly researched, with no approved drugs for treatment in Europe and the US. But that is changing, reports Nature. In 2020 the global Cancer Grand Challenges initiative issued a call for cachexia research, and has since put $25 million into projects across 16 research groups. The syndrome is no longer thought of as a simple combination of malnutrition, muscle wasting and weight loss, but as a whole-body metabolic disorder, with the liver and brain both promoting complex changes to metabolism across the body. The most promising avenue of treatment is currently focused on a stress hormone called GDF15, which diminishes appetite and is a known contributor to cachexia. Pfizer has developed an antibody called ponsegromab that neutralizes GDF15, and in a 187-person trial concluded in 2024, patients on the highest dose gained more than two kilos over 12 weeks, while those on a placebo lost weight. Appetite and physical activity improved, too. Promising cachexia treatments have failed before, though, so phase-three trials will be the real test of whether ponsegromab is a true advance. Meanwhile, other research projects are teasing the condition apart into several sub-types, suggesting that in the longer term, cachexia may require a range of different treatments.

In Europe, air-conditioning has become a political flashpoint. Spare a thought for populations across Europe and parts of Asia, where air-conditioning is far from commonplace. As both regions endure a summer of multiple heat waves and Europe faces wildfires — even in Scotland, where the climate and vegetation make them relatively rare — residents have become hot, bothered and occasionally violent, with shoppers in Paris fighting over AC units. But the answer isn’t as straightforward as simply providing more air-conditioning. The very fact of air-conditioning, as pointed out recently by both The Economist and Noema magazine, has become a political lightning rod, with (mostly) populists on the right and left fighting it out. In France, the right’s Marine Le Pen wants AC in every school, hospital and nursing home. The left’s Jean-Luc Mélenchon counters that more air-conditioning will lead to higher temperatures and more heat waves. In poorer countries, it's a matter of political compromise over cost: Necessary electricity and grid upgrades are expensive in already strained economies. Both publications invoke Lee Kuan Yew, Singapore's first prime minister, who credited AC with "making development possible in the tropics." One thing that isn’t debatable: Air-conditioning saves lives. According to research published in the National Bureau of Economic Research, residential AC in the US was responsible for a roughly 75 percent decline in the risk of dying on a hot day over the 20th century. Europe, meanwhile, now accounts for 36 percent of global heat deaths despite being home to less than 10 percent of the world's population. As temperatures rise, air- conditioning may prove essential to keeping economies running — and populations content — far outside the tropics. Not that consumers are waiting: Chinese AC exports to the EU rose 43 percent this year compared to last year, to $3.8 billion.

A fatal reaction, from Science and Retraction Watch
6,300 words, or about 25 minutes

This is a harrowing cautionary tale about lax medical regulation over gene therapy. An investigation by Science and Retraction Watch, a publication that reports on the retraction of scientific papers, reveals how a six-year-old girl in Shanghai died in March 2025 after receiving the first brain-directed gene-editing therapy ever given to a person, a procedure never made public. The child had Snijders Blok-Campeau syndrome, caused by a single genetic mutation, and was developmentally delayed. Her parents effectively commissioned a bespoke therapy from a neuroscientist named Zilong Qiu at the Songjiang Research Institute in Shanghai, part of the Shanghai Jiao Tong University School of Medicine. They scraped together around $860,000 from savings and gifts from relatives which they donated to various research organizations connected to the project, and even to individual researchers. Seven days after engineered viruses carrying the gene editor were infused into her spinal fluid, she died of a severe immune reaction. The hospital's own ethics review concluded that the death was "definitely related" to the treatment, and the district health department fined the hospital $3,600. There appears to have been a shocking lack of oversight. Under China's dual-track system, non-commercial, investigator-initiated trials at major hospitals can test new gene therapies with only local ethical review, bypassing the national drug regulator. Here, that local review fell short. The investigation also revealed that the hospital ethics committee approved the treatment in January 2025, a month before results of testing the therapy on monkeys revealed that it caused liver damage. And, perhaps most incredibly, after the girl’s death Qiu's group published the underlying preclinical work in Nature, stripping out the family's genetic data, making no acknowledgment of the group’s source of funding and never mentioning that a child had died. The university told the parents in April that it was taking no further action; since this story was published on July 23, the university has opened a new probe.

The fall of America’s farm superpower, from The Financial Times
2,600 words, or about 10 minutes

In 2018, President Trump imposed wide-ranging tariffs on Chinese goods. Beijing retaliated not only with duties on products grown in the US, but by finding new sources for goods they wanted to buy. In the case of soybeans it turned to Brazil, which is now by far the biggest supplier of the crop to China. In fact, Brazil is now the world's biggest exporter of soybeans, beef, poultry and cotton. And while the US remains the world's largest exporter of corn, it has gone from being responsible for 68 percent of corn exports two decades ago to around 30 per cent today. Brazil was well positioned to make the leap. Its climate and farming methods lend themselves to two harvests a year (even three on some irrigated land), while land itself is plentiful and cheap. Modern agronomy has also helped it expand into the tropical savannah, nourish nutrient-poor soils and develop crop genetics better suited to the climate. To be clear, neither the US nor Brazil is having an easy time here: Brazilian farmers are squeezed by imported fertilizer prices and high domestic interest rates, among other things. But this story paints a picture of Brazilian farm towns booming while historic communities in the US Midwest have to rely on domestic demand, occasionally providing supplemental supply when Brazilian production runs short.

Rust and Boll, from Asterisk
4,900 words, or about 19 minutes

At first blush, cotton picking and AI don't have a whole lot in common. But this essay explores how many of the issues facing AI have presented themselves in the past. The story traces the history of John and Mack Rust, brothers from Texas who cracked the problem of automating cotton picking in the 1930s. The brothers thought their invention could help usher in "a planned economy of abundance." They worried about the labor displacement it would cause and made a series of proposals to ease that problem, including a proposed nonprofit foundation. One idea they abandoned was leasing machines only to planters who agreed to decent wages and no child labor, on the grounds that such terms would push demand toward rival pickers operating without such practices. Dylan Matthews, the author, points out that versions of all these themes have surfaced in the AI industry. Ultimately, the Rusts' company went bankrupt, and none of their social schemes amounted to anything. Matthews concludes that businesses may do a very good job of making money, but struggle to deliver social impact because the moment high-mindedness conflicts with market share, high-mindedness goes out the window.