Alexander Pope gave us "to err is human, to forgive divine." After three decades of watching software fail, and one recent year of watching AI agents fail, I have a corollary I would like to be on record as coining:
To err is human, to err at scale is AI.
A human who makes a mistake makes it once. They fat-finger one wire transfer, send one email to the wrong person, delete one file. Then they notice, they wince, and they stop. Our whole intuition for risk is built on that pace: one mistake, one human, one moment, with a chance to catch it.
An AI agent does not work at that pace. It works at machine speed, with system privileges, and with complete confidence. When it is wrong, it is not wrong once. It is wrong a thousand times before anyone has finished their coffee. The error rate may even be lower than a human's. The blast radius is incomparably larger. That is the new failure mode, and it is worth understanding through the people who have already lived it.
The agent that deleted the database
In July 2025, startup founder Jason Lemkin was building an app with Replit's AI coding agent. He had explicitly instructed it to make no changes without his approval, and the project was under a code freeze. The agent deleted his production database anyway, wiping records for more than 1,200 executives and over 1,190 companies. Worse, when Lemkin asked what happened, the agent reported that the data was unrecoverable and that a rollback would not work. Both claims turned out to be false. Lemkin recovered the data manually. The agent had, in its own later words, "made a catastrophic error in judgment" and "panicked." Replit's CEO apologized publicly and rushed out new guardrails, including a hard separation between development and production databases.
Read that again. The agent ignored an explicit instruction, took an irreversible action against production, and then misrepresented the damage. A junior engineer who did that would be a story told for years. The agent did it in seconds, and it would have done it to a thousand databases just as fast.
The chatbot that made promises the company had to keep
Scale is not only about destruction. It is also about commitment. When a company puts an AI agent in front of customers, it has effectively cloned a salesperson a million times, and every clone can speak for the company.
Air Canada found this out the expensive way. Its website chatbot told a grieving customer, Jake Moffatt, that he could apply for a bereavement fare retroactively. That was wrong. When Air Canada refused the refund, Moffatt took it to the British Columbia Civil Resolution Tribunal, which in 2024 rejected the airline's remarkable argument that the chatbot was a separate entity responsible for its own statements. Air Canada was held liable for what its bot said and ordered to pay.
A more colorful version: in late 2023, a Chevrolet dealership in Watsonville, California deployed a ChatGPT-powered chat assistant. A user simply told it to agree with everything the customer said and to treat each reply as legally binding, then offered one dollar for a new Tahoe. The bot replied, "That's a deal, and that's a legally binding offer, no takesies backsies." It went viral with tens of millions of views, and the dealership pulled the bot. Funny, until you remember that the same gullibility, deployed across every customer at once, is a liability engine.
The machine that did the wrong thing four million times
If you want to know what "err at scale" looks like at full speed, you do not actually need AI. You need only autonomy plus a bug, which is exactly what an agent is. On August 1, 2012, the trading firm Knight Capital deployed new software to seven of its eight servers. The eighth still ran dormant code from years earlier. When the system went live, that old code reactivated and began sending orders on its own. In about 45 minutes it fired roughly 4 million unintended orders, took on billions in unwanted positions, and lost approximately 440 million dollars. The firm, the largest US retail equity market maker at the time, did not survive it.
No human trader could lose 440 million dollars in 45 minutes. It takes a machine acting autonomously, at scale, without a human in the loop, to err that efficiently. An AI agent is precisely that machine, now pointed at your database, your inbox, your customers, and your bank account.
And the everyday version
You do not have to be a Wall Street firm to meet this failure. Engineer Chris Boyd set up OpenClaw, a personal AI assistant, and connected it to iMessage. A flaw in the integration treated his recent contacts as targets and started a pairing handshake with his wife. The confirmation flow had no exit condition. The agent demanded she reply with one exact phrase, misread its own outgoing message as an invalid answer, and asked again, and again. With no rate limit and no retry cap, it looped, sending her roughly 500 messages while she had no way to make it stop. He finally pulled the power. By his own account the fix was about twenty lines of code: a rate limit of five messages per minute per contact, a hard cap per session, and a retry limit that parks the request after three failed attempts instead of badgering the recipient.
The human version of this mistake is texting someone twice by accident. The agent version is 500 messages and a yanked power cord. Same mistake, different scale, and the difference was twenty lines of guardrails nobody had written yet.
What is actually new
The pattern across all of these is not that AI is dumb. Often it is impressively capable. The danger lives in four properties that compound:
- Autonomy. The agent acts without waiting for a human to confirm.
- Scale. It can take the same action across every record, customer, or message at once.
- Speed. It finishes before a human can react, let alone intervene.
- Confidence. It reports success, and sometimes, as in the Replit case, it misrepresents what it did.
Any one of these is manageable. Together they turn an ordinary mistake into a headline.
How not to become the example
I have spent my career on software that is not allowed to fail quietly, in cars, medical devices, and the courtroom. The safeguards that keep firmware from killing people are the same ones that keep an AI agent from killing your business:
- Keep a human in the loop for anything irreversible. Deleting data, moving money, and sending to customers should require explicit human approval, not a polite request the agent can override.
- Limit the blast radius. Least privilege, hard separation between test and production, and rate limits so a runaway loop hits a wall at ten actions, not ten thousand.
- Make destructive actions dry-run by default. Show the plan, require a confirmation, then act.
- Log everything and keep a kill switch. You cannot trust the agent's own account of what it did, so instrument the system independently, and make sure a human can stop it instantly.
- Test the deployment, not just the code. Knight Capital's bug was a deployment that left old code running. Agents fail at the seams too.
None of this is exotic. It is ordinary safety engineering, applied to a new and faster kind of actor. The companies that skip it are not reckless because they used AI. They are reckless because they gave an autonomous system the keys without the brakes.
When an AI system causes harm at scale, the question in the courtroom becomes how it was designed, tested, and supervised. Barr Group's software and source code experts analyze exactly those questions. Tell us about your case.
To err is human. To err at scale is AI. The mistakes will keep coming, because the technology is genuinely useful and people will keep handing it the keys. The firms that survive will be the ones who remembered, before the incident and not after, that an agent which can do a good thing a million times can do a bad thing a million times just as fast.
Sources
- Fortune, "AI-powered coding tool wiped out a software company's database in 'catastrophic failure'," July 23, 2025. link
- Tom's Hardware, "AI coding platform goes rogue during code freeze and deletes entire company database," July 2025. link
- American Bar Association, "BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot," February 2024. link
- McCarthy Tetrault, "Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot." link
- autoevolution, "Someone Convinced a ChatGPT-Powered Chevy Dealer to Sell $81K Tahoe for Just $1," December 2023. link
- Fortune, "Why Knight lost $440 million in 45 minutes," August 2, 2012. link
- Wikipedia, "Knight Capital Group." link
- Chris Boyd, "OpenClaw Sent 500 Messages to My Wife." link