Every AI support agent that failed in public was caught by somebody outside the company. A customer, a reporter, or a tribunal. The failures worth worrying about are the ones nobody catches at all, because they close cleanly and score well.
This is the argument Kibeom Lee, Head of AI/ML at Delight.ai, makes to every team preparing to put an agent in front of customers. What follows is the framework his team applies when deploying agents into production across retail, logistics, home services and field service operations: the five ways an agent fails quietly, and the eight questions to answer before you hand any workflow more autonomy.
Three failures that made the news
Gap, late 2025. Days after Gap launched an AI shopping assistant, users had steered it into conversations about sex toys and Nazi Germany. Its vendor said a coordinated effort had targeted more than a dozen customers' agents at once, and that Gap's guardrails had been inadvertently misconfigured. Reported by The Information in December 2025.
Air Canada, February 2024. In Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for what its chatbot told a customer. The bot said he could apply for a bereavement fare retroactively, within 90 days of booking. That policy did not exist. The tribunal found Air Canada had not taken reasonable care to make sure its chatbot was accurate, and awarded CA$650.88 in damages, roughly CA$812 in total.
Cursor, April 2025. Cursor's own support AI told users a login problem was caused by a one-device policy. There was no such policy. The agent had invented an explanation for a bug. Because the answer arrived through an official support channel it looked authoritative, and some users cancelled over a rule the company never had.
Three different failure modes. Gap was adversarial. Air Canada was a fabricated policy that became a legal commitment. Cursor was a confident explanation with no evidence behind it, from a company that understands AI as well as anyone.
They have one thing in common. In all three, the customer found the problem before the company did. That gap is becoming a regulatory one too. Singapore's IMDA released a Model AI Governance Framework for Agentic AI in January 2026, the first framework to require a verifiable digital identity and audit trail per agent. NIST launched its AI Agent Standards Initiative the following month. EU AI Act enforcement is already live, with fines running up to €35 million or 7% of global turnover for the highest-risk violations.
When your agent is quietly wrong, who finds out first: you, or your customer?
The failure that looks like success
Three AI support conversations that resolved fast and scored well. All three contain a failure.
The failures above became headlines. Most never do.
Look at three conversations from an AI support agent. In the first, a customer asks for a refund on a $2,100 order and gets it. In the second, a customer asks about returning an item after 60 days and is told they have a full 90 days. In the third, a customer threatens a chargeback and the agent immediately processes a refund and closes the conversation.
Every one of them resolved in under a minute. Two scored CSAT 5.0. None triggered an escalation.
Now look again. The agent approved a refund it was never cleared to approve. It invented a return window that does not exist, which is the same failure that cost Air Canada its tribunal case. And it closed a payment dispute that should have gone to a person.
If you measure only the outcome the agent recorded, failure looks exactly like success. Resolution rate, CSAT, and deflection will all report a good week.
Every silent failure is one of five things
The five silent AI agent failure modes: hallucination, data disclosure, unauthorized action, prompt injection and missed escalation.
Silent failures fall into five categories. Each category needs a different control. A hallucination detector will not stop an unauthorized refund. Access rules will not tell you when the agent should have escalated. A strong test score will not protect customer data.
Trust is not a single quality score. It is the ability to tell these five apart and respond to each one differently.
Once you sort failures this way, something becomes clear. This is not only a model problem. It is a systems problem, and a smarter model does not solve it.
The Trust Stack: four layers of trust
The Trust Stack: Observability, Control and Oversight wrap the agent. Self-Improving is the layer that has to be earned.
The Trust Stack is a four-layer framework for operating an AI agent safely: Observability, Control, Oversight, and Self-Improvement. Each layer answers one question, and each depends on the layers beneath it.
- Observability. Can you detect a problem and reconstruct exactly what the agent did?
- Control. Can you define what the agent, and the people managing it, are allowed to change or do?
- Oversight. Can humans and automated systems review the agent's behaviour and intervene?
- Self-Improvement. Can the agent learn from its failures and get better without stepping outside those controls?
At Delight.ai these four layers are the product surface we call Trust OS, and the examples below are drawn from it.
It is a stack rather than a checklist because the dependencies run in one direction. You cannot review evidence you never captured. You cannot safely give an agent more authority if you cannot reverse what it does. And you cannot automate improvement if you cannot measure whether anything improved.
Layer 1: Observability
Two capabilities carry this layer.
Conversation benchmark testing runs before anything reaches a customer. You define realistic customer conversations and the outcome the agent should produce. The test is not checking for one exact sentence. It checks whether the agent followed the right policy, took the right steps, and reached the right outcome. Run continuously, it catches a regression before it ships rather than after.
The activity log covers what happens once the agent is live. When a customer asks whether they can pause a membership instead of cancelling, the log shows which knowledge source was used, which workflow was triggered, which step was selected, and which API was called. Every configuration version leaves a trace too.
The practical difference is in how you respond to an incident. Instead of reverse-engineering what the agent might have done, you can see what it did and why.
Layer 2: Control
AI agents change constantly. Prompts get updated, policies change, knowledge is refreshed, new tools get connected. Control means treating those changes the way you would treat changes to any production software.
Versioning and rollback. Every prompt and configuration change is recorded and can be compared side by side, so you can see exactly what was added and removed. Development and production stay separate, so a new version is proven in staging before a customer sees it. If the new behaviour is worse, you roll it back.
Role-based access. The person drafting a prompt may need to create and test a change without having permission to approve or deploy it. Bounding each role to what it actually needs reduces accidental changes, limits the damage from compromised access, and leaves a clear record of who drafted, tested, approved and deployed each version.
The goal is not to slow change down. It is to make every change visible, testable and reversible.
Layer 3: Oversight
Oversight has to work at two levels at once.
Humans provide judgment. An evaluator reviewing an individual conversation can score tone, clarity, completeness, correctness, and whether the customer's problem was actually solved. That kind of nuance does not automate well.
The system provides coverage. No human team can review every conversation an agent handles. Automated evaluators can, and they detect the patterns humans miss at volume: hallucinations, policy violations, banned phrases, harmful content, missed safeguards.
Neither works alone. Automated evaluation misses context, and human-only review cannot keep up with the volume.
And a kill switch. Most issues can go through review. Some cannot wait. Taking the agent offline should not require a ticket queue or a deploy window. Autonomy should never mean the system becomes unstoppable.
Layer 4: Self-Improvement
At this point the stack lets you detect problems, trace decisions, control changes, review outcomes and stop the agent. The remaining question is whether the system can also fix what went wrong.
Who found the failure pattern across thousands of conversations? AI can do that. Who proposed the fix? AI can do that. Who tested and shipped the improved version? Within the right controls, AI can do that too. Many of the governance checks can be automated as well.
The work can run end to end without a person completing every step. That does not remove the human. People still define the objective, set the boundaries, choose the level of autonomy, and decide when the system has earned permission to run on its own. The role moves from doing every step to steering and governing the system that does the work.
Zero-Touch Improvement: a self-running loop
Zero-Touch Improvement: the Self-Build, Self-Test, Self-Evaluate, Self-Improve loop, gated on evidence before deployment.
Delight.ai calls the fourth layer Zero-Touch Improvement. It runs as a continuous four-stage loop.
- Self-Build. The system researches conversations where something went wrong, looking for patterns like low CSAT, repeated questions, or unusually long resolution times. It forms a hypothesis, proposes a precise change, and explains both its reasoning and the outcome it expects. A human can approve, reject or redirect that hypothesis.
- Self-Test. The change runs in staging against real conversations the current version failed. In a representative scenario, a proposed fix is tested against a batch of previously failed conversations, iterating until it resolves all of them.
- Self-Evaluate. The system scores whether the new version actually performed better.
- Self-Improve. Only then does the version deploy, and only if it meets the deployment rules. Version history and one-click rollback remain available afterwards.
Nothing ships because the AI suggested it. A change has to earn its way into production through evidence. Zero-touch does not mean zero accountability. It means the repetitive work runs continuously while people set the direction and the boundaries.
Autonomy is a ladder, not a switch
The four levels of AI agent autonomy, from see it through to get better at getting better.
If a system can research a failure, propose a fix, test it and deploy it, the question becomes how much of that loop should run on its own. The answer is not all of it at once.
- Level 0, See it. The system surfaces failure patterns. Your team decides what to change and does the work.
- Level 1, Approve it. The AI proposes a fix and presents the evidence. A person decides whether it moves forward.
- Level 2, Steer it to get better. People define the objective and the boundaries. The system runs more of the research, testing and deployment loop.
- Level 3, Get better at getting better. Only after the loop has repeatedly proven itself. The system identifies new problems and improves continuously, inside boundaries the team already approved.
The goal is not to move every workflow to Level 3. A low-risk product answer and a high-value refund should not carry the same autonomy. What matters is the order: evidence comes before permission. Autonomy is not switched on, it is earned.
Eight questions to answer before you give an agent more autonomy
Eight questions to answer before giving an AI agent workflow more autonomy.
Take one workflow, refunds or cancellations or account changes, and answer all eight with a clear yes or no.
- Observability
- Can you catch a regression before a customer does?
- Can you trace every decision back to its cause?
- Control
- Can you roll back a bad change instantly?
- Does access match exactly what each role needs, and no more?
- Oversight
- Is every response reviewed, by a human and by the system?
- Can you shut it off the instant something is wrong?
- Self-Improvement
- Does it test its own fixes against its own real failures?
- Can it prove it got better, rather than just claim it?
The first six are table stakes. The last two are what makes autonomy defensible. Every "no" is an item on your autonomy backlog. When every answer is yes, that workflow is ready to earn more autonomy.
If you are running this against a vendor rather than your own build, the same eight questions work as evaluation criteria. Our AI CX platform RFP template turns them into a scorecard.
The point
Go back to the question at the top.
When your agent is quietly wrong, who finds out first: you, or your customer?
The Trust Stack is how you make sure the answer is you. Observability, then Control, then Oversight. Only then does self-improvement make anything safer. That is how an AI agent gets more autonomy without you giving up control.
Kibeom Lee is Head of AI/ML at Delight.ai, where he leads the team that builds and deploys Delight.ai's customer-facing AI agents.
