Pillar article

Who is accountable when an AI agent gets it wrong?

By Pat Villaceran

I hold the business that deployed the agent responsible, every time. Inside that business, the person who set the agent to work is accountable. That's my answer whenever someone asks me who is accountable when AI makes a mistake: a tool can't own a decision, so the human who prompted it, approved it or shipped its output owns the result, good and bad.

When an AI agent gets something wrong, Pat Villaceran's view is that a person is accountable. An agent can take the action, but it cannot be sued, fired or asked to explain itself to a regulator. A person in the loop only helps if that person understands how the work was done. Klarna shows the cost of getting this wrong: in May 2025, its CEO told Bloomberg it was recruiting human customer service staff again after a cost-led approach left lower quality. Her advice is to decide in writing where the agent's work ends.

For me, whoever deployed it answers for it

Picture the end of a long chain. Agents have done a pile of work. Nathan set it up like this: someone gets "a little prompt pop up, they click yes or no. If they click yes, are they responsible for everything that that agent has done? Good and bad."

My answer: "the person doing the prompting is one hundred percent accountable." Then, straight after: "Nathan, on every single thing, on every single thing, right?"

This wasn't theory. Nathan and I had been testing agent workflows the week we recorded. "Maybe it's hallucinated... But if I didn't catch that, it is my accountability". The numbers don't match the data set? That's mine. Something wrong goes out under my name? That's mine too.

Technology is not the enemy. I said it more than once in that episode. But "the model did it" is not an answer I accept from anyone on my team. Including me.

An agent can take the action, but a person is accountable

An agent drafts a contract. It checks the clauses. It sends it. Then something goes wrong. Can the agent be sued? Fired? Embarrassed in front of a client? Asked to explain itself to a regulator?

No. Every one of those lands on a person. And so the decision sits with a person too. When we talked about contracts, I said: "There's someone still controlling the agents. That's the person that signs the thing."

That is why accountability is one of our core beliefs. Early in the episode I said: "For me, Nathan, innovation is always about accountability. You can always blame someone else." And then: "You can always blame the market, you can always blame the thing, but that's the problem solving, that's where innovation really happens." With agents, the temptation to blame something else grows. Why? The agent hands you a very convincing excuse.

One click at the end isn't accountability

Five agents have each done a step. The person at the end sees the last output, not the steps before it. The deadline is in an hour. The draft looks fine. They click yes.

Something breaks. They were "in the loop". On paper, the process was supervised. In practice, nobody looked at step two. That's where the error crept in. Mostly, that click puts the blame on one person. It doesn't give them what they need to carry it.

A person in the loop only helps if that person understands how the work was done. That is why I said "everyone in the chain of the process should now be knowledgeable on base foundational theories on how AI technologies work." Everyone. Even the people who never touch the code.

In the loop, the human is the driver

Think about a long drive. The driver doesn't check the car once at the end of the trip. The driver is there the whole way. "So human in the loop is the driver."

Klarna cut costs, and quality dropped

Nathan brought up Klarna. Everyone knows that story. In May 2025, Klarna's CEO told Bloomberg that the company was recruiting human customer service staff again. Fortune reported the words of Sebastian Siemiatkowski, the chief executive: "As cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality."

I want to be straight with you about the caveat. A Klarna spokesperson told Fortune, "we're very much still AI-first", and CIO reported the same mixed messages: a pilot of new human staff, alongside a statement that the focus on AI hadn't changed. Klarna didn't give up on AI. The story is a decision made mostly on cost. Then the people at the top owned what it did to quality.

I gave the Klarna team credit for exactly that when we talked about it. They changed the narrative from the top, and they said every customer should be able to talk to a human being. That's accountability. At company level.

Decide where the agent's work ends

A customer writes in: "Where's my order?" An agent can answer that. Another customer writes in. They're upset. It's about money. That one needs a person. "You want a human being to be solving more complex questions than it's gonna arrive on Thursday."

For us, it's about being able to say "where do we put the line in to make sure" "That our clients and our customers are still served." So draw the line in writing, before a customer finds it for you. That's the move.

This matters more now. Agents are starting to hand work to each other. Since 9 October 2026, in beta, an agent on Anthropic's Claude Managed Agents can write a workflow, which Anthropic's release notes define as "a program that runs many agents in phases and combines their results". Every handover is another gap. A question that needed a person can slip through it. If the line only lives in someone's head, who checks it at the handover? Nobody.

Draw it this week.

1. Pick the agent that talks to your customers. Support replies, booking changes, refund requests, the chat window on your website. Start where a wrong answer reaches a person outside the company.

2. Read a week of what it handled. Sort the requests into two piles. One pile is the "where's my order?" kind, where the answer sits in your data and nobody has to judge anything. The other pile is everything that needed judgement: complaints, money, exceptions, anyone upset.

3. Write the line down. One short page. The agent answers the first pile. The second pile goes to a person, with the conversation so far attached so the customer doesn't have to repeat themselves. Add one rule for doubt: if the agent would have to guess, it hands over.

4. Leave a door open. Give every customer a visible way to reach a human at any point. After the company's cost-led push on AI support, Siemiatkowski said in the same Bloomberg interview: "[I]t's so critical that you are clear to your customer that there will be always a human if you want."

5. Check the numbers against the data. Once a week, you, the person who set the agent to work, take a handful of its answers and check them against the source. Do the dates, amounts and order details match what's in your system? My test was exactly this: "If the statistics are not matching the data set, it's my accountability." Where they don't match, move that kind of request across the line.

Your first page will be wrong somewhere. Fine. A "simple" request turns out to need a person? Move it across. With the line on paper, you can see which side a mistake came from.

I count agents in the workforce and keep human capital for people

Agents are getting job titles now. For example, on org charts. I said it to Nathan: they are part of the workforce, but I do not think they are human capital. If a firm supplies the agents, the accountability for how they perform goes back to that firm. If you deploy them yourself, it's yours.

Before you decide where agents go at all, answer the questions above the workflow: what your culture is about and what you're building for. You'll be able to hear the whole conversation once the episode is out.

One more read

The read from this episode is called "The Yes Button".

Sources

The button below puts you on the waitlist.

Become an Impact Maker

← All articles