There is a large and growing gap between a voice AI demo and a voice AI system. A demo needs to survive one conversation in a quiet room. A production system answers thousands of calls from drivers standing next to running engines, speaking Hindi mixed with English, asking about money — and it has to be right, fast, every time, with no human watching.
This is the breakdown of how we built the second kind: the voice agent that handles inbound driver support for Battery Smart, India's largest EV battery-swapping network. In-scope calls — battery status, station availability, swap history, account and payment queries — are resolved end-to-end with zero human handoff.
The constraints that shaped everything
Three constraints drove every architectural decision:
- Hindi first, not Hindi translated. The callers are drivers whose working language is Hindi, freely code-switched with English ("battery ka status check karo"). A translate-to-English pipeline adds latency and mangles exactly the phrases that matter.
- Latency is the product. On a phone call, silence longer than about a second reads as a broken line. Whatever the architecture, the total loop — hear, understand, look up, decide, speak — has to feel conversational.
- Wrong answers have physical consequences. If the agent tells a driver a station has charged batteries and it doesn't, someone is stranded on the road. Answers must come from live systems, and the agent must refuse to guess.
The architecture
The simplified flow, every stage streaming:
Five stages, but the design rule is one sentence: treat the phone call as a distributed system. Each stage can be slow, fail, or return garbage, and the call has to survive all of it gracefully.
The speech layer: why Sarvam AI
We chose Sarvam AI for both speech-to-text and text-to-speech because it is built for Indian languages rather than adapted to them. That mattered in three specific ways:
- Code-switching is native. Hinglish isn't an edge case in this domain — it's the default register. The STT handles mid-sentence language switches without falling apart.
- Noisy audio is the norm. Roadside calls mean traffic, engines, and wind. The recognition had to hold up on real-world telephone audio, not studio recordings.
- The voice sounds local. TTS quality decides whether callers stay on the line. A natural Hindi voice keeps the interaction feeling like service, not a robocall maze.
Both directions run streaming: partial transcripts flow to the agent as the caller speaks, and synthesized audio starts playing before the full response is generated.
The agent layer: an LLM with tools, not an intent tree
The classic approach here is an intent classifier feeding a decision tree — IVR 2.0. We went with an LLM agent layer instead, for one reason: real callers don't speak in intents. They interrupt themselves, ask two things at once, and refer back to something they said a minute ago. A dialogue-state-carrying LLM handles that naturally; a tree becomes unmaintainable.
The agent layer owns three jobs:
- Understanding in context. It maintains dialogue state across turns, so "aur wahan se pehle wala?" resolves against what was already discussed.
- Tool calling. Every fact the agent can state maps to a tool — a thin, typed wrapper over a production API. Driver profile, battery telemetry, station availability, swap history, payments. If there's no tool for it, the agent cannot claim it.
- Knowing its limits. The in-scope query list is explicit. Anything outside it — disputes, refunds, anything contractual — routes to the escalation path with the transcript attached. The prompt is written so the agent's failure mode is "let me connect you", never a confident guess.
The most important line in the system prompt is not what the agent should do. It's what the agent must never do: invent an answer it didn't retrieve.
Live data, defensively
Tool calls hit the same production APIs that power the driver app — which means answers reflect reality at answer time, and also means the agent inherits every failure mode those APIs have. So every tool carries a timeout, a retry policy, and a spoken fallback ("I'm having trouble checking that right now — one moment"). A slow upstream service degrades the conversation; it never breaks it.
Latency engineering
The latency budget is end-to-end: caller stops speaking → agent starts speaking. Ours lands at ~200ms on average. Getting there was less about any single fast component and more about never doing work in sequence that could happen in parallel:
- Streaming partial transcripts into the agent so understanding starts before the caller finishes.
- Kicking off independent tool calls concurrently instead of one by one.
- Starting TTS on the first complete sentence of the response rather than waiting for the whole answer.
- Short verbal acknowledgements while a slow lookup completes, so the line never goes dead.
What production actually required
Everything above gets you a working agent. These are the parts that let it run unsupervised:
- Barge-in. Callers interrupt. The agent stops talking, listens, and re-plans mid-turn.
- Full observability. Every call produces a transcript, the tool calls made, and a resolution tag. That's the raw material for everything else.
- Evals before rollout. We built an evaluation set from real call scenarios and scored the agent against it on every change — retrieval correctness, scope discipline, tone.
- Staged rollout. Query types were enabled one at a time, each watched in transcripts before the next was switched on. "Zero human handoff" was earned scope by scope, not declared on day one.
What we deliberately didn't build
Two roads we considered and rejected, because both optimise for the demo instead of the operation:
- An all-in-one voice platform. Off-the-shelf voice AI platforms are genuinely fast to stand up — until you need Hindi-native speech quality, tool calls into private production APIs, and custom escalation logic. At that point you're fighting the platform's abstractions instead of building on them. Owning the pipeline meant we could swap any stage independently as better components ship.
- Open-ended conversation. It's tempting to let the model "just talk" — it makes for an impressive first call. But an agent that can discuss anything can be led anywhere, and a support line is an adversarial environment. Bounded scope isn't a limitation we accepted; it's the feature that made unsupervised operation possible.
Results
In-scope customer queries are resolved with zero human handoff, 24/7, at 1000+ calls per month, 70% of total inbound volume. Support executives stopped scaling with customer growth, which was the point. Less hiring. More postive EBITDA.
What we'd tell anyone building one
- Scope discipline beats model horsepower. A modest model that refuses to leave its lane outperforms a frontier model that improvises.
- Go language-native. If your callers code-switch, translation pipelines will fail them at the exact moments that matter.
- The call is a distributed system. Timeouts, retries, fallbacks, and observability aren't add-ons; they're the architecture.
- Evals from real transcripts are your safety net. You cannot supervise every call. Your eval set supervises for you.
This is the engineering behind our AI voice agents service, and the same tool-calling, live-data approach powers our RAG & AI integration work. The full case study, including the business results, is here: Battery Smart voice agent case study →