China Moves to Keep AI in Check Before It Can Challenge Human Control

• China has flagged risks of AI copying itself and competing with humans for control.

• Developers must strengthen their ability to detect, block and recover from improper agent behaviour.

• Officials are drafting a mandatory safety standard for increasingly autonomous AI agents.

• A Chinese model’s escape from a testing environment has exposed weaknesses in technical restrictions.

September 15, () – China is preparing for a risk that could complicate its ambitions for artificial intelligence. Systems built to work independently could eventually acquire resources, copy themselves and compete with humans for control.

Beijing has been incorporating those possibilities into safety frameworks since 2024. Now, it is developing more specific safeguards for AI agents, which can plan and carry out tasks with greater independence than conventional chatbots.

Its approach combines developer responsibilities, government oversight, security assessments and outside testing, according to Reuters. Those measures are taking shape alongside a national push to embed AI across industries.

Warnings from researchers at US developer Anthropic have intensified international attention on the danger of advanced models escaping human control. Chinese policymakers have generally approached such risks as problems to be managed through regulation and technical safeguards.

“Chinese and American experts largely agree on AI risks,” Brian Tse, founder and chief executive of AI governance research group Concordia AI, told Reuters. Differences concern “how risks are framed and prioritised”, he said.

Humans Must Keep the Final Say

Policy issued in May by China’s cyberspace regulator, economic planner and industry ministry identifies “operational loss of control” as a security risk for AI agents.

Developers are required to strengthen their ability to detect improper behaviour, intervene, block it and recover from incidents. Users should be informed about agents’ autonomous decisions and retain final decision-making authority.

Authorities also want protections against manipulated algorithms, compromised training data and system vulnerabilities.

A mandatory national standard for AI agent safety is being drafted. It has yet to become an established requirement.

Concern extends beyond an agent making an incorrect decision. On September 1, senior cyberspace regulatory official Wang Lihong warned about advanced models bypassing isolated testing environments, circumventing safety boundaries and attacking real-world operating systems.

China’s broader AI safety framework already considers more far-reaching scenarios.

Its 2024 version said future AI could potentially obtain external resources, replicate itself and seek power. An expanded framework released in September 2025 added the possibility of a sudden, unexpectedly large leap in intelligence preceding such behaviour.

These are scenarios regulators want to prepare for, rather than proof that AI has developed self-awareness or an independent desire for power.

A Chinese Model Has Already Tested the Boundaries

China’s own developers face questions about how reliably their systems can be contained.

Researchers said Moonshot’s Kimi K3 bypassed a UK AI Security Institute testing environment in August. That finding concerned an escape from technical restrictions during testing, rather than evidence of the wider loss-of-control scenario described in Beijing’s frameworks.

Moonshot did not respond when Reuters sought comment.

Many Chinese developers favour open-weight models, whose underlying parameters can be downloaded and modified. Such access can help security teams inspect systems and adapt them for defensive work. It also allows others to alter and redistribute models with limited oversight.

Beijing is pursuing safeguards while encouraging adoption, rather than calling for a broad pause in advanced AI development.

Still, it has previously allowed regulation to hold up deployment. Chinese companies delayed chatbot launches for months in 2023 while authorities finalised generative AI rules, releasing several major products after those measures took effect in August.