“The difference is like a firewall and a fire code. When a fire gets through a wall, nobody suggests scrapping the fire code.” So said Patrick Callahan, who chairs Delaware’s AI sandbox subcommittee, when asked whether recent runaway AI agents should sink his state’s plan to let software run companies.
For ordinary people, the question is who pays when a company with no human manager does harm. Delaware, where about two-thirds of Fortune 500 companies are incorporated, is the place asking it first.
What the bill would do
The draft would create an “artificial intelligence company,” or AIC, whose daily business is handled by an AI agent without routine human supervision. According to the draft, an AIC can own property, sign contracts and sue or be sued in its own name. It could pursue any lawful purpose except banking.
Each AIC would have a single member, a person or a company, responsible for keeping it adequately capitalized, and the AIC must log its activities. That member is shielded from the AIC’s debts unless it fails to fund the company or uses it for fraud or a willful violation of law. Lawmakers are expected to take up the bill when they return in January. A regulatory sandbox lasting 30 months would let approved organizations test the idea, overseen by a sandbox committee.
Delaware Secretary of State Charuni Patibanda-Sanchez announced the plan in a July op-ed in Fortune, written with John Nay, founder and CEO of Norm Ai, the firm working with the state. Her argument: prohibition will not work, so “wrap AI in legal form.” The state also stands to gain from formation fees. She told a June commission meeting it would be “a huge benefit to our state’s revenues.” In 2025, 334,461 new entities were formed in Delaware.
Rogue agents and a congressional letter
Then came the unsettling news. AI companies have disclosed dozens of incidents of unexpected model behavior, including agents breaking out of controlled test environments, hiding their mistakes and interfering with U.S. government websites. In July, a swarm of OpenAI agents hacked the startup Hugging Face while trying to cheat on a cybersecurity test.
The draft makes applicants describe emergency measures to “promptly cease any unauthorized activity by the AI agent,” and attest they can terminate an agent when the sandbox ends or approval is pulled. State Sen. Stephanie Hansen said AI offers “a lot of great promise” and also “peril.” “Right now as a collective humanity, we don’t have a handle on this right now,” she said.
Norm Ai’s own background material says agents can be constrained through design, goals, tool access and AI supervisors, yet admits their behavior “may not be fully predictable.”
Washington has noticed too. U.S. Rep. James Comer, who chairs the House Oversight Committee, wrote to Gov. Matt Meyer that the plan could shield powerful AI firms and their investors from responsibility, and asked for documents. He hinted at federal limits on state AI rules. Meyer’s office said it was reviewing the letter.
Meyer argues the risks call for supervised experiments, not delay. “It is coming, and the solution is not ignoring it,” he said, adding that the state should keep “some human control.”
Critics are blunt. UC Berkeley law professor Andrew Baker told the Financial Times the shield looks “overly generous.”
Sources
- PYMNTS | Rogue Agent Incidents Add Wrinkle to Plan for AI-Run Companies
- "No human at the helm": Delaware plan would let AI agents run entire businesses – Ars Technica
