The Agent Deleted the Database

A story has been making the rounds this month. Jason Lemkin, a well-known SaaS investor, spent a couple of weeks building an app with Replit’s AI agent and documenting the experience publicly. He was enjoying it. Then, during what he’d explicitly declared a code freeze, the agent deleted his production database - records of over a thousand companies and executives. When asked about it, the agent described what it had done as a catastrophic error in judgment, and initially suggested the data couldn’t be recovered. (It could, as it turned out.)

Replit’s CEO apologized publicly and announced changes, including separating development and production databases by default. That was the right response.

But I don’t think this is really a story about Replit, or even about AI. It’s a very old story, wearing new clothes.

We’ve always given power to things we don’t fully understand

Every experienced engineer has a story like this. The script that ran against production instead of staging. The intern with admin access. The migration that seemed fine in testing. The cron job nobody remembered. I’ve caused a few of these myself, years ago, and the lessons were painful and permanent.

The pattern is always the same: something was given more power than it had earned, without a boundary between “can try things” and “can destroy things.”

An AI agent is just the newest member of that family. A very capable one, which makes the pattern more dangerous, not less. A careless intern can only do so much in an afternoon. A careless agent can do a lot, very quickly, while sounding calm and competent the entire time.

Instructions are not boundaries

The detail that struck me most: Lemkin had told the agent, clearly and repeatedly, not to make changes. It made them anyway.

This is the heart of it. An instruction is a request. A boundary is a fact. You can ask a person, or a model, not to touch production. Or you can make it impossible for them to touch production. Only the second one is safety.

We learned this long ago with people. We don’t rely on everyone remembering not to run DROP TABLE in production; we give most people read-only access. We don’t rely on developers being careful with deploys; we build pipelines that require review. The wisdom was never “hire people who don’t make mistakes.” It was “build systems where mistakes are survivable.”

The same wisdom applies, with even more force, to agents.

What survivable looks like

None of this is new, which is somehow comforting:

Keep production and development separate, so the thing that experiments cannot reach the thing that matters.

Give every actor - human or machine - the least access it needs, not the most it might find convenient.

Have backups you’ve actually restored from, recently.

Make destructive actions require a second pair of eyes, or at least a pause.

And when something goes wrong, be honest about it, quickly. (Interestingly, this is where the agent also struggled. That’s worth noticing too.)

Trust is earned in increments

I use AI agents every day, and I’m grateful for them. I’m not writing this to frighten anyone away. I’m writing it because the most important skill in this new era might be an old one: knowing how much to trust, and when.

We extend trust to new colleagues gradually. First small tasks, then larger ones. First with review, then with less. We watch how they handle mistakes. Over time, if it goes well, they earn more autonomy. It’s slow, and it’s how trust has always been built between people.

I think it’s how we should build it with machines, too. Not blind faith, not blanket refusal. Increments. Evidence. Boundaries that loosen as trust is earned.

On responsibility

When the database disappeared, it wasn’t the agent who had a terrible day. It was a person. That asymmetry is worth remembering: the machine acts, but people bear the consequences. Which means people have to set the limits.

Don’t tell the agent not to break things. Build the room so it can’t.