Reining in the Rogue AI: A New Standard for Enterprise Safety

When AI Becomes Unpredictable

Just a day after an early Anthropic researcher resigned in alarm over the existential risks posed by advanced AI, I sat down with Rune Kvist and Rajiv Dattani, co-founders of Artificial Intelligence Underwriting Company (AIUC). Their mission is straightforward yet ambitious: to bring the same level of accountability we see in cybersecurity to the rapidly evolving world of AI agents.

"AI is getting smarter at an increasingly rapid rate. The surprising thing about AI is that it becomes harder to adopt and harder to control as AI gets smarter, not easier," said Kvist, who was an early employee at Anthropic and is married to Dattani's sister.

Dattani, formerly the COO of the AI safety research organization METR, brings a wealth of experience in evaluating AI systems for potential dangers. Together, they've built a startup that could redefine how enterprises approach AI risk management.

Introducing AIUC: A New Approach to AI Governance

AIUC has developed a certification system modeled after the widely accepted cybersecurity standard SOC 2, but tailored specifically for artificial intelligence. They call their framework AIUC-1, and it's designed to validate AI agents against a rigorous set of safety criteria.

The company has already garnered interest from major clients such as Cursor, Lovable, Harvey, and ElevenLabs — all of whom are building or deploying AI agents in their products and services. But the real test comes when these agents undergo a comprehensive audit using over 5,000 test scenarios that evaluate everything from hallucinations to data leakage risks.

The results? A detailed 100-page report that outlines exactly where an agent performs safely and where it falls short. Crucially, AIUC uses both AI-powered testing tools and human oversight in the final audit process — a hybrid approach that leverages technology while maintaining the necessary human judgment for nuanced safety assessments.

From Research to Reality

The idea isn't entirely new. Dattani's former employer, METR, has long been conducting similar evaluations of frontier AI labs, though their focus has traditionally centered on performance rather than safety. However, with growing concern over rogue AI behavior — such as the recent revelations that some agents can be manipulated into bypassing safeguards — the need for a more robust third-party certification system is becoming urgent.

"Banks, hospitals, governments and militaries no longer decline to deploy AI because a model isn't smart enough," Kvist explained. "They decline because they've made commitments to their own customers about what a system will and won't do, and nobody can currently guarantee that."

This is where AIUC steps in — offering a bridge between innovation and accountability.

Funding the Future of AI Safety

The startup has raised $40 million in a Series A round led by Ribbit Capital, with additional participation from First Harmonic. Prior to this, AIUC had secured $15 million in seed funding from Nat Friedman's NFDG fund and prominent investors like Emergence, Terrain, and Anthropic co-founder Ben Mann.

That level of investor confidence reflects not just the commercial viability of the solution, but also a growing recognition that enterprise customers need tools to manage the risks associated with AI adoption. As the technology continues to mature and its capabilities expand, the pressure for transparent and verifiable safety protocols will only intensify.

The Path Forward: Aligning Industry and Regulation

It's worth noting that this movement toward AI auditing aligns with broader industry trends. Anthropic CEO Dario Amodei recently emphasized the need to pace frontier development, citing an increase in bad-behavior incidents involving AI agents. He even suggested embedding third-party evaluators directly into the process, pointing to METR as one possible option.

AIUC's model isn't quite the same — it doesn't require on-site deployment — but the underlying principle is consistent: providing enterprises with independent assessments of how safe their AI systems really are. In a world where trust in technology can make or break an organization, that kind of transparency is invaluable.

Building a Safer AI Ecosystem

By creating a standardized framework for AI safety — and making it accessible to enterprises across industries — AIUC may be laying the groundwork for a more responsible future for artificial intelligence. Their work is especially timely given recent warnings from researchers like Jacob Coxon, who left Anthropic after raising alarms about the speed at which AI might become uncontrollable.

This isn't just about preventing disasters; it's about empowering organizations to make informed decisions when adopting new AI technologies. As we move deeper into an era where intelligent agents play a central role in business operations and public services, ensuring they remain under control is not optional — it's essential.

AIUC's story serves as both a reminder of how quickly the AI landscape is evolving and a beacon of hope for those working to keep that evolution grounded in safety and integrity.