Case Study
AI that audits support,
and the bots that answer it
Built for FloBiz (myBillBook), a GST billing app for Indian small businesses. I built an auditor that grades support chats, a WhatsApp bot that answers only from the help docs, and an eval for voice AI on collection calls.
How a chat gets graded · illustrative
Parse, judge, score
Support audit
Illustrative chat · invented, not a real customer
00 · raw transcript
One chat in. Bot and agent mixed together.
Chat text and scores in this demo are invented to show the mechanics.
The design constraint
The politest chat had a skipped check
The warmest, most polite chat in the corpus was also one where the agent skipped verification. A rubric that rewards tone ranks it near the top. So I flipped the weights: empathy is worth 5 points out of 100, and a failed gate caps the leg at 40 however good the rest of it reads.
Architecture
The LLM judges. Python counts.
Parse · pure code
Splits each transcript into legs: the bot, then every human agent who touched the chat. No model involved, so the same chat always parses the same way.
Judge · LLM
For each check the model gives a verdict and the verbatim line that proves it. It never hands back a number.
Score · pure Python
Python computes every point, applies the gates and caps a leg at 40 when one fires. Bot and agent get separate scorecards and are never averaged.
Trace + redact
Every scorecard row links back to the chat it came from. Evidence is redacted before it is published, and the corpus never leaves my machine.
What it surfaced
The most common auto-fail was a template, not a person
In the first audit, the most frequent gate traced back to the handoff template. It had no field for internal notes, so notes ended up wherever there was room. The fix was a tooling change. No coaching needed. That is what I want an audit to do: point at the system first.
Numbers
The first audit
48
Chats audited
first audit
50
Legs scored
bot and agents separately
12
Agents
each with their own scorecard
113
Automated tests
parse and score are pure code
5/100
Empathy weight
politeness can't hide a miss
40
Gate cap
one failed gate, always red
The other side
The bots that answer it
Grounded WhatsApp bot
Internal test · not customer-facing yetA support bot that only says what the docs say
RAG plus a rules layer. It answers from the help docs and cites the doc it used. Building the 318-question knowledge base surfaced 65 places where the docs contradicted each other.
318
KB questions
65
contradictions found
0%
grounding violations in testing
~$0.64
per 1,000 chats (≈ ₹55), projected
Voice-AI eval
Cheaper model not at parity yetPicking a voice model for collection calls
I compared two voice models for an automated collections agent: a 20-call like-for-like run plus 227 simulated calls. The eval caught a voice-activity detection setting that stopped the agent from pausing when a caller cut in. It also caught a template placeholder being spoken aloud in over half of real calls, which I confirmed by transcribing them. The cheaper model costs about 5× less per minute. It isn't at parity on judged quality yet, and I say so.
20
like-for-like calls
227
simulated calls
₹3.66 → ₹0.75
per minute
Not yet
at quality parity
Vendor map
MappedWhat in-house AI can replace
I mapped 18 vendor tools against what in-house AI could do instead. Most of them are support, telephony and CRM add-ons, which is exactly where the auditor and the bot already live.
18
vendor tools mapped
3
categories: support, telephony, CRM add-ons
Decisions
Why I built it this way
Why doesn't the LLM do the math?
Models are shaky at arithmetic and drift between runs. So the model only says what happened and quotes the line. Python adds it up. Same verdicts in, same score out, every time.
Why score the bot and the agent separately?
They fail in different ways and different people fix them. Average a weak bot with a strong agent and both problems disappear into one middling number. Two scorecards tell each owner what is theirs.
Why is empathy worth only 5 points?
The warmest, most polite chat in the corpus was also one where the agent skipped verification. Under a tone-heavy rubric it would have ranked near the top. Courtesy still counts. It just can't buy back a failed check.
Why a hard cap and not a big deduction?
A deduction can be outweighed by points elsewhere. A cap can't. If a gate fires, the leg lands at 40 or below, and nobody has to argue about it.
Why insist every row traces to its chat?
An audit people can't check is an audit people won't trust. Any score can be traced back to the exact line that produced it, which makes disagreements quick to settle.
Stack
What powers it
Python
Parse + score
LLM judge
Verdicts + quotes
pytest
113 tests
Redaction
Before publish
RAG
Grounded answers
Rules layer
Bot guardrails
Bot channel
Speech-to-text
Call transcripts
Want your support audited, or answered?
I build AI that checks its own work and shows the line it came from. Tell me about your support queue.
© 2026 Sarthak Goel