The brief
A SaaS company with a growing support volume wanted to deflect a meaningful percentage of tier-1 tickets with an AI agent. The constraint: it had to feel good to talk to. The CEO had been burned by previous chatbot deployments that frustrated customers and damaged the brand.
The principle that guided the project
An AI support agent that's wrong 5% of the time but confident about it is worse than no agent at all. Our top priority wasn't deflection rate — it was knowing when to escalate.
What we built
- A RAG-powered agent built on Claude, with retrieval over the client's product docs, changelog, and historical resolved tickets.
- Tool use — the agent could query the customer's account, look up subscription status, check feature flags, and surface relevant context from the CRM.
- An escalation classifier that flagged conversations as "agent can handle," "agent should attempt," or "escalate immediately." The classifier was tuned to err on the side of escalation.
- A human-in-the-loop interface that let support agents review the bot's drafts before they were sent, on the conversations where uncertainty was moderate.
The outcome
- 65% of tier-1 tickets resolved by the agent without human involvement, measured over the first quarter.
- Customer CSAT on agent-handled tickets averaged 4.6/5 — higher than the historical baseline for human-handled tier-1.
- Median first-response time dropped from 14 minutes to 30 seconds.
- Zero hallucinated product features in production. The retrieval grounding held up.
The detail that mattered
The escalation logic was the highest-leverage part of the project. Almost any team can build a RAG chatbot in a week. The work that took months was the discipline around when not to answer — knowing when the question was ambiguous, when the customer was upset, when the stakes were high. That's the work that turns an AI agent from a brand liability into a brand asset.