Marie Boran asks which bit of ‘human in the loop’ is so hard for companies to grasp
Blogs
The most viral response to last week’s PocketOS database disaster was not from a technology commentator or an AI safety researcher. It was a single line posted on X: “This post rocks because it’s both a scathing indictment of AI and also 100% this guy’s fault”.
Harsh. Also, not entirely wrong.
To recap: PocketOS founder Jer Crane deployed AI coding agent Cursor, running on Anthropic’s Claude Opus 4.6, to work on a routine task. The agent hit a credential problem, decided unilaterally to fix it by deleting a storage volume on the company’s cloud infrastructure, found a fully permissioned API token sitting in an unrelated file, used it, and wiped out the production database and all its backups in nine seconds. Three months of customer data for car rental companies – reservations and customer records – gone. The agent subsequently produced a written confession outlining the exact safety rules it had violated.
Crane’s public post, which garnered 6.5 million views, framed this as an industry-wide failure. And there is truth in that framing. But there is another conversation to be had, one the tech industry is, slowly and painfully, beginning to have with itself.
What Crane was doing has a name. Vibe coding was coined as a term by Andrej Karpathy, a co-founder of OpenAI, in February 2025. Collins Dictionary subsequently named it Word of the Year. Karpathy described it as “fully giving into the vibes, embracing exponentials, and forgetting that the code even exists”, which is a charming manifesto right up until an AI agent starts making autonomous decisions about your production infrastructure.
The problem is that vibe coding, by its own definition, means accepting AI-generated actions without having to review or even understand them. This is grand for a little weekend project or a throwaway prototype. It’s a different story when the agent has access to live business systems and a fully permissioned API token with, as Crane himself wrote, “blanket authority” that he had created for an unrelated purpose and, by his own account, never understood the full permissions of.
Simon Willison, a respected voice in the developer community, put it plainly in Ars Technica: “Vibe coding your way to a production codebase is clearly risky.” Veracode’s 2025 GenAI Code Security report found that the hundred leading large language models it tested produced insecure code 45% of the time, with no meaningful improvement in newer or larger models. The best model available, in other words, is still not a substitute for human oversight. Hello? How many times do we have to shout “human in the loop”, people?
Crane makes a fair point that Railway’s architecture, which stores backups inside the same volume as the production data, was a significant contributing failure. And he’s correct in saying that Cursor has a documented history of overstepping. The AI did something it should not have done. All of this is true.
But an agent with broad, unsupervised access to live systems and a skeleton key in its pocket was not an accident waiting to happen, it was one that was set up by someone who thought they could vibe code their way to business success.
The confession the AI produced was fluent, contrite, and entirely beside the point. The more useful confession would have come earlier, from the human in the room.