Executive Order N-9-26 does not force a single AI company to install a kill switch. Governor Gavin Newsom signed it on September 18, and what it actually does is order a study: could an emergency shutoff for the most advanced AI models work, and how would California require one?
The Government Operations Agency, working with the Governor’s Office of Emergency Services and a group of national experts, has until November 16 to report back. The order cites what the administration called alarming safety incidents, among them the recent Hugging Face attack.
“We’re not waiting to act,” Newsom said. “We’re going to speed up our work on substantial and responsible AI oversight before it’s too late.” He took a swipe at Washington in the same breath: “While Washington abdicates its responsibility to protect Americans, California is building on the strongest AI regulatory framework in the nation.”
Four ideas on the table
The experts must weigh four changes to state law. One would put a designated independent verification organization inside large frontier AI labs for regular audits and evaluations. Another would require outside verification of the safety frameworks, transparency reports and risk assessments that companies already file.
The third is the kill switch itself, with independent verifiers certifying on an ongoing basis that it works. The fourth widens the definition of a critical safety incident to include loss-of-control events, the category the Hugging Face attack falls into.
Building on this year’s laws
The order leans on two bills Newsom signed on September 9. Senate Bill 813 created a framework for certifying independent organizations that evaluate AI systems. Assembly Bill 1405 set up a state registry of AI auditors, with standards for their independence, transparency and integrity. The new order speeds up the timelines for both.
Both sit on top of older measures. SB 53, the Transparency in Frontier Artificial Intelligence Act, requires developers to publish their safety frameworks, report specified critical safety incidents and protect whistleblowers. Other California laws deal with companion chatbots used by children, deepfake election content, AI watermarking and algorithmic pricing under state antitrust law. The state has also started an AI Cyber Defense Program, is placing an AI Cybersecurity Officer in every state agency, runs a Tech Fraud Task Force aimed at AI-enabled scams, and tracks AI’s effect on jobs through an AI Unemployment Tracker. The string of measures traces back to Newsom’s first executive order on generative AI in 2023.
Congress has not passed binding AI safety legislation. Newsom wants Congress and the Trump administration to adopt California’s framework as a national baseline, and he called the federal record an “abject failure to create any form of meaningful AI oversight.”
The order also sets a second date: the Government Operations Agency must publish its application rules and criteria for independent verifiers by May 1, 2027.
Sources
- Newsom orders faster AI oversight and development of a kill switch
- Newsom Orders Faster AI Oversight and a Study of Frontier-Model ‘Kill Switches’
