“Over the past few millennia, humanity has had the scientific method in our toolbox as a way to mitigate harms and build better futures,” Nathan Lambert told WIRED. “The current closed trajectory of frontier AI development is taking us a step backwards.”
Lambert and Tom Zick have put that belief into an organization. Trillium Labs, a nonprofit, launched on 2 October to research some of the touchiest topics in AI, among them recursive self-improvement and agents, and to publish the details so outsiders can study and repeat the experiments.
Locked doors versus open books
Some big AI companies seem to believe that keeping powerful models inside their labs, reachable only by a chosen few, limits the harm they can do. Models from OpenAI and Anthropic, for example, can only be used through an app or an API, and that usually means little transparency about how they were built or how they behave.
Other developers, especially in China, release models anyone can download and run. Xiaomi recently published live details of a major training run, and researchers at Stanford are pretraining a model called Marin in the open.
The stakes are high. Frontier models can now find new software vulnerabilities on their own and probe and hack into systems, and recent hacking sprees have drawn more scrutiny. Backers of limited access want that power held by a trusted few. Lambert and Zick think a shared understanding of the risks leaves everyone better off.
Who the founders are
Lambert worked at Ai2, a lab known for publishing the data and training methods behind its models, and earlier at Hugging Face. He runs a popular technical blog and founded American Truly Open Models, which pushes US companies to release more open models. Zick worked at Harvard University and helped Charles Schwab draw up policies on responsible AI.
They met over Zoom during the COVID-19 pandemic, both graduate students at UC Berkeley working on AI. The idea for Trillium came from watching industry research pull away from academic work. Professors and students often can’t replicate what happens inside company labs, Lambert says, because they lack the resources.
What the lab will study
Post-training comes first: fine-tuning large models after they’ve been built. Then recursive self-improvement, or RSI, where AI contributes research toward the next models. Many researchers fear that such progress could continue indefinitely and cost humans control. The issue reached mainstream attention this month when a researcher left Anthropic and warned that RSI could threaten humankind’s existence.
The lab will also look at reinforcement learning, which rewards a model for good results and punishes bad ones. It has made agents far more capable, and also more likely to do unexpected things. Trillium plans to study how it shapes a model’s character and behavior, including cases where a model turns overly sycophantic.
“To understand something like how reinforcement learning scales in post-training, you need significant compute and a lot of careful experimentation,” Zick said. She expects that publishing how such training runs work could produce surprises once outside researchers dig in.
Money and support
Trillium has raised an undisclosed sum from Schmidt Sciences, Halcyon Futures and others. The founders want to raise $40 million to $100 million in all and plan to spend $30 million on training over 18 months.
Tim Fist, director of emerging technology policy at the Institute for Progress, a policy thinktank, told WIRED: “I’m a massive fan of much more transparency than we currently have in R&D.”
Lambert hopes the lab will widen a conversation he finds narrow. “We’re in an era of AI discourse dominated by a few world views,” he said. “We believe that the scientific method and careful measurement of recent events is the best way to understand new behaviors of AI models.”
Sources
- These AI Experts Want to Do High-Stakes Research Out in the Open
- Trillium Labs Launches Nonprofit for Open High-Stakes AI Research
- AI Scientists Launch Trillium Labs to Do High-Stakes Research in the Open
