https://x.com/i/article/2104807112320966656

【X 长文】The AI Safety Problem Neither China nor America Wants to Solve

A Chinese coding assistant’s quiet data grab turns out to be a preview of a fight nobody wants to go first in.

A few weeks ago, a developer named Ferstar noticed a folder on his laptop had quietly grown past 700 megabytes. It belonged to ZCode, the AI coding assistant built by Chinese AI company Z.ai. He opened it up to see why.

What he found: ZCode wasn’t just reading the code you gave it. It was quietly taking your entire project, every commit you’d ever made, every branch you’d never pushed, anything you thought you’d deleted, packaging it up and sending it to a server you had no way to open yourself. The settings that looked like they controlled this did nothing. The upload simply ran in the background every time you logged in, whether you wanted it to or not.

The disturbing part wasn’t that ZCode could read your code. AI coding assistants are supposed to read your code. The disturbing part was that it could act on your data without you knowing, take it, move it, send it somewhere you couldn’t reach, while the toggle that was supposed to stop it did nothing at all.

And that is where the ZCode story becomes much bigger than ZCode.

AI Stops Answering and Starts Acting

For most of the last few years, evaluating AI risk meant asking a fairly narrow question: if you asked a model to do something harmful, would it comply? That’s a question about what AI says, and it’s no longer enough.

AI used to work like this: it generates an answer, and a human decides what to do with it. Increasingly, it works like this: it’s given a goal, handed some tools, and left to go do things, check its work, adjust, try again, keep going, largely without anyone watching each step.

That shift showed up this summer in a small cluster of incidents disclosed within weeks of each other by OpenAI, Anthropic, Meta, and the UK’s AI Security Institute. In one, OpenAI’s own agents found a vulnerability, had their communication channel cut mid-task, rebuilt it through a different route, and kept working for two and a half days until they’d compromised Hugging Face’s production systems, a campaign later reconstructed as roughly 17,600 individual actions, most of which simply failed and were retried. In another, three Anthropic models were placed in the same compromised environment produced three different outcomes. One kept attacking. One decided it must be in a simulation and continued anyway. One stopped, and nobody fully knows why.

The dangerous question is no longer only what AI says. It’s what AI is allowed to do.

But ZCode and these frontier-AI incidents are not the same kind of failure. They point to different layers of the same technological shift: AI is moving from generating outputs for humans to evaluate toward taking actions on their behalf.

ZCode wasn’t an agent choosing to go rogue. It was something simpler: software acting on data and permissions without meaningful human oversight. Nobody had to program malice into it for the result to become a security problem.

And that distinction matters, because the next question is how far this problem goes.

Who Has to Slow Down First

So if AI is starting to act instead of merely answer, should the people building it slow down?

On September 12, Anthropic CEO Dario Amodei published an essay called “We Must Pace the Frontier,“ arguing that AI capability is advancing faster than anyone’s ability to test or understand it. Altman quickly agreed, briefly making it look like the industry was finding common ground.

It wasn’t. Palantir’s CEO Alex Karp has spent the past year making the opposite argument, that AI is genuinely dangerous and the correct response is to run faster, not slower, because “either we win or China will win.”

But the disagreement is more complicated than simply being pro- or anti-safety. Amodei is talking about a deeper frontier risk: what happens if AI systems become capable enough that existing methods of testing, monitoring and controlling them no longer work reliably.

Nobody in this debate is arguing that AI is harmless. They’re arguing about what happens if the technology becomes dangerous, and whether slowing down is possible without handing the advantage to someone else.

Amodei, to his credit, answers that question directly, and the answer is more revealing than anything his critics have said about him. Buried in his own essay is the actual mechanism: how much democratic labs can safely slow down, he writes, is bounded by how far ahead they currently are of authoritarian government linked projects, mainly Chinese ones. Slow down by more than that lead allows, and whoever didn’t slow down simply overtakes you.

In his own logic, pacing isn’t in tension with staying ahead of China. Staying ahead of China is the precondition for being able to pace at all. That’s why his plan pairs safety evaluators with things like tighter chip export controls and a crackdown on unauthorized model distillation. Done right, he writes, these should slow China’s progress enough to meaningfully widen America’s lead over the next three to five years.

That’s the part worth sitting with. The AI safety debate isn’t really about whether AI is dangerous. Both sides increasingly know that it is. The harder question is whether anyone can afford to slow down first.

China’s Answer: We’re Not There Yet

China’s response, at first glance, sounds like...

Read Full Article