Engineering notes, product updates, and voice AI insights from the team building TechLadder.
A slow first response can disrupt an entire voice conversation. We found that prompt design and cache reuse mattered more for latency than we expected. By restructuring prompts, improving cache hits and routing requests more carefully, we cut first-token latency from about 1,780 ms to about 350 ms and carried 2.3× more calls per GPU in our tests. In this article, we share the engineering lessons, benchmarks and practical techniques that helped us improve voice LLM performance under load.
Our voice agent can tell every speaker apart and still spend the whole call talking to the wrong person. Let's know why the loudest speaker is often not the one your agent should follow, and how we solved it.
Every Voice AI vendor tells you how many calls their agent handles alone. Almost none tell you what happens when it can’t. That’s the number customers remember. They don’t call back because the bot confirmed an appointment. They call back because they said “cancel” twice and still weren’t understood. This playbook explores that boundary: what Voice AI should do when it reaches the edge of what it can or should handle.