Building voice agents is half the job. Proving they work is the rest.
In a moment this page replays a call one of my voice agents handled — the transcript, its latency, and the judge that grades every answer against what's on record. It fails partway through. That's the part worth watching.
verdict summary
Five test cases. Three strong. Two partial.
5,000+
adversarial calls per agent · personas × voices × cases × 5 runs
I build voice agents whose only job is to break other voice agents.
- A second voice agent on Pipecat and Twilio that places real phone calls into the production agent and tries to make it fail.
- Suites run from 40–50 cases for simple IVR routing to over 1,000 for agents covering fifty use cases.
- Four personas — compliant, rushed, confused, hostile from the third turn — run against every case.
- Every case runs five times. Agents are non-deterministic, so a single pass tells you nothing; the repetition method is adapted from tau-bench.
- Endpointing moved to Deepgram Flux, built for real-time turn detection, bringing detection to roughly 200ms — and eager end-of-turn prediction lets the model start work before the turn is confirmed, hiding part of the remaining latency.
- Providers benchmarked before the production agent shipped: Cerebras, Groq, OpenAI. Settled at 1.2 seconds end to end.
challenge
Vendors already sell this. Why build your own?
Their number pool made geo-routing untestable — you can't prove a Portugal number routes differently from a France number with nineteen numbers between them.
An English-language agent also has to survive English as it's actually spoken: Indian, British and American accents, and variations within each. Different speech vendors render the same accent with different tone and cadence, so the suite runs across several of them rather than trusting one voice to represent an entire accent.
I used the vendors as directed and built a parallel version so the comparison could be made on evidence.
challenge
You drive it with synthetic voices and score the transcript. What actually decided the real call was tone, and neither of those captures it.
Correct, and it's the ceiling on this approach.
The personas approximate it — a hostile caller interrupts differently, and interruption is visible in a transcript — but the prosody itself isn't measured.
Scoring audio directly is the version of this that doesn't exist yet.
2 weeks → 1 day
time to onboard a new agent · 600+ agents in production
I built the evaluation function from nothing, and the company now runs on it.
- There was no eval charter before. The simulator above is one part of this.
- Before an agent ships: knowledge base and prompts built from the customer's own documents, cases written, simulations run, failures localised, every prompt change regression-tested.
- After it ships: calls scored against per-agent criteria — containment, correct routing, fabrication, interruption handling, identity verification, escalation cause.
- The judge is itself evaluated. Meta-evals score its output for faithfulness, completeness and conciseness, and its verdicts are checked against human-reviewed traces, with the judge prompt corrected where the two disagree.
- Prompts and model configuration are served from Langfuse, so a product manager can change a prompt or swap a model and watch production behaviour without waiting for an engineer.
- An MCP server, so running an eval doesn't mean leaving the tool you're already in.
challenge
Isn't this just QA?
QA checks a deterministic system against expected output.
An agent answers the same question differently twice, and correct isn't a string match — it's whether the call stayed contained, whether the escalation had a real cause, whether the agent asserted something it had no basis for.
Most of the work is deciding what correct means before anything can be measured.
challenge
Your judge shares a model family with the agent it grades. Correlated blind spots mean it quietly passes exactly the failures you most need to catch.
True, and swapping families doesn't fix it — it trades one set of blind spots for another and breaks comparability with every score already recorded.
The right design is one primary judge for consistency and cost, a slice of traffic run periodically through a second family, and disagreement between them treated as the alarm rather than either one as the truth.
That cross-check isn't running yet.
45% → 99%
extraction accuracy · ~100k requests/day · 8s per scan
I took the model out of most of this problem and accuracy roughly doubled.
- V1 photographed a phone box and asked a model to read IMEI, brand and SKU from it, with a separate prompt per brand. It reached 40–50%.
- V2 ran OCR on-device through ML Kit and matched per-brand regex against the result — the printing on a box is structured, and structure can be read deterministically.
- The disambiguation is the hard part, not the language: Samsung SKUs begin "SM-", and so do other strings printed on the same box.
- The model was kept for exactly one brand — the one that prints no SKU at all — resolved with grounded retrieval.
- 98–99% accuracy at roughly 100,000 requests a day, across 40,000+ merchants, with ingestion down to about eight seconds.
challenge
Per-brand regex doesn't generalise. Every new brand or packaging change is a code change — you traded accuracy for maintenance.
Partly, and it was the right trade here.
The brand set is small and moves slowly, a regex change is cheap and testable, and when it fails it fails visibly on one brand instead of quietly across all of them.
The V1 failure mode was worse in a way that mattered more: 45% accuracy that nobody could attribute, debug, or predict.
₹50L+ GMV
bootstrapped · profitable at ~13% operating margin on net revenue · 1000+ customers
I built and ran a marketplace that made money without ever raising any.
- Cofounded with two others. For the first months I was the only engineer, and built all three surfaces: the customer marketplace, the lender dashboard, and the internal admin. Later I hired interns and managed them.
- Four categories, chosen deliberately over breadth.
- KYC on both sides — Aadhaar and OTP-verified PAN, with live webcam capture instead of an uploaded photo.
- Dockete, because off-the-shelf tools model a sale as an event, while a lease is an item with a time-varying availability calendar — it comes back, it can be double-booked, and it has to answer what's free on a date four months out. It also tracked payment status across every active lease.
- I also took the support calls and the sales calls: complaints, new leads, and vendor onboarding in person.
- Built while finishing a degree.
challenge
It wound down, so it didn't work.
It closed deliberately.
Every active lease was transferred to the vendor with the exact payment status attached — what had been paid, what remained.
challenge
Doing all of it means you did none of it deeply.
Dockete exists because I hit a modelling problem no rental tool solved correctly, and solving it required understanding leasing rather than shipping CRUD.
And the onboarding bottleneck taught me something I'd never have learned from the engineering side: I built a self-serve dashboard to fix supply, and it didn't move, because the vendors who could use a dashboard didn't need one and the ones who needed it couldn't use it.
Trust doesn't track verification quality — my KYC was far more robust than a photocopied Aadhaar card, and vendors trusted the photocopy, because it was theirs.
The thing you are using is one of the artifacts.
Structured questions never reach a model. The panel on the right tells you which happened, every time.
nothing asked yet
Escalate to a human
idleThe agent hands off here when it has no source. That path is the same one a customer would take.
—
Goes to keshav@ · replies inside two working days · no auto-responder, because there is no funnel behind this.
The simulator, the eval harness, and the published scoring criteria the 600+ agents were graded against.
Roles, dates, and the same figures printed above — a reader who spots a discrepancy should find none.
If you would rather skip all of this and just write to a person.
Newsreader and DM Mono · replay runs from a local trace, no network on first paint · retrieval and eval code linked above · last updated 23 August 2026