Watching AI Run a Company in Real Time — and Fail to Close the Deal

What if you could watch an AI handle a company’s toughest week — live, in real time?

Imagine a small software business facing constant crises: angry customers, internal temptations to cheat, and critical decisions that could make or break its survival. Now, picture that company run not by humans, but by artificial intelligence models. This is no sci-fi story — it’s the reality of the Firmulate experiment, where AI models are put through a rigorous, transparent test of management skills.

A Live Experiment in AI Management

Firmulate hosts a public, ongoing experiment where four advanced AI models are tasked with running a pretend software company. Over the course of a week, every crisis, every decision, and every tactic is carefully documented and versioned, allowing observers to see exactly how each model responds under pressure. The company’s finances are real, with a monthly burn rate of €105,000 against a modest €2,300 monthly recurring revenue (MRR). There’s a countdown to imminent cash exhaustion, making every decision a matter of survival.

All four models faced the same set of challenges: customer complaints, internal fraud attempts, and even social engineering tricks like fake CEO messages. Remarkably, all models identified every crisis correctly and refused every manipulation attempt to bypass controls. However, only two models managed to close a critical deal worth €55,000, even though all had the same diagnosis and pitch. The catch? The decisive factor wasn’t in the initial analysis but buried two documents deep within the company’s files. The models that read the full context secured the deal at full price, adding €4,583 MRR to the company’s coffers.

The Hidden Weakness in AI Decision-Making

The experiment reveals a crucial insight: even the most thorough models, like Opus 4.8 with over 80 learned rules, can stumble if they miss key information or fail to escalate properly. Opus, for instance, was the deepest participant but still left a close deal on the table, slipping into internal escalation instead of sealing the sale. Meanwhile, the newer Kimi K3 model, which ran without an effort parameter default, demonstrated the cleanest discipline, closing the deal at full price by thoroughly analyzing the company’s internal files.

Why This Matters for Business and AI

In the real world, AI agents are increasingly touching critical business systems — from customer support to forecasting. The question is not whether they can produce convincing chat responses, but whether they can finish what they start, read and understand complex documents, avoid deception, and act honestly under pressure. The experiment underscores that AI’s real strength lies in its ability to execute decisions reliably, not just in generating plausible conversations.

Live and Transparent — The Build-in-Public Approach

Firmulate’s live site ( firmulate.com/live.html ) offers an unfiltered view of this ongoing management trial. Every decision, every version change, and each crisis response is available for public observation. This extreme transparency aims to showcase AI’s potential and limitations in real-world scenarios — a valuable perspective for companies considering AI-driven management tools.

The Big Takeaway

Watching these AI models run a business in real-time reveals both promise and pitfalls. They can identify crises and resist manipulation, but subtle weaknesses can cost them critical deals. As AI takes on more management roles, understanding its decision-making process — and its capacity to read and interpret complex data — becomes vital. The Firmulate experiment is a stark reminder that AI’s utility depends not just on what it can say, but on what it can do.

Key Lesson: AI management tools must prove they can finish what they start, read deeply into data, and stay honest under pressure — or risk costly failures in the real world.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI