Coaching that arrives after the call is coaching that arrives too late
Post-call QA (like my call-audit system) tells an agent what went wrong yesterday. But a sales call is won or lost in the moment — when the customer raises a price objection, the agent has seconds to respond with the right fact. Telio moves the coaching into the call itself.
- Hinglish, two speakers, streaming. Live code-mixed Hindi-English audio must become usable text in real time — with the agent and customer kept on separate channels so the system knows who said what.
- A late suggestion is a worthless suggestion. The latency budget spans streaming ASR → LLM reasoning → UI push. Every stage must fire on the customer's turn, not on a polling clock.
- Silence breaks naive designs. When the customer goes quiet, a turn-triggered copilot goes dead exactly when the agent is monologuing off script and needs it most.
- Concurrency multiplies everything. Every call is two parallel ASR streams — a floor of concurrent calls hits vendor quotas twice as fast as the call count suggests.
- Insights must be for the agent's ear. Business intelligence about the customer has to arrive as internal guidance the agent can act on — not lines to read out loud.
An event-driven pipeline that thinks between turns
Three services: a backend that ingests audio and runs the AI pipeline, a live dashboard, and a call simulator that replays real recordings as live calls for testing. The pipeline is triggered by conversation events, not timers.
Dual-channel audio ingest
Each live call streams the agent and the customer as two separate WebSocket audio channels into the backend, which lazily creates a call session on first contact and tracks its lifecycle end to end.
Streaming ASR with a resilient key pool
Per-channel streaming speech-to-text translates Hinglish audio into English text as it's spoken. API keys are drawn from a round-robin pool that spreads concurrent streams across keys and automatically retires exhausted ones — so one dead key never stalls the floor.
One-shot business-intelligence enrichment
At call start, the session fetches the customer's business and competitor metrics once from an internal API, normalizes the payload, and compresses it into a dossier — search demand, competitor density, gaps in the customer's own listing — that the LLM can reason over.
Turn-based LLM coaching
A main agent fires on each customer utterance, producing a next-best-action with its reasoning and a verified proof point from the dossier. If the customer goes silent, a 15-second idle fallback re-runs the agent on the agent-side transcript — the panel never goes dead.
Live dashboard + embeddable assist panel
Suggestions, key insights, and a script-adherence tracker are pushed over WebSocket to a React dashboard — and to a stripped-down iframe embed that drops the same three panels into the dialer application the agents already use.
What the agent actually sees
Three live panels, updated as the conversation moves — inside the dashboard and embeddable in the dialer app the agents already work in.
Live Assist per customer turn
A next-best-action suggestion generated on every customer utterance — with its reasoning shown in small print and a verified number from the customer's own business data as a proof point. Guardrails strip unusable proof points (nulls, bare numbers) before they reach the screen.
Key Insights business intelligence
The 3–4 most important facts about this customer's market, phrased in simple Hinglish as internal guidance for the agent — "customer ko photo upload karwana chahiye, profile views badhenge" — never as lines to read aloud, and never exposing internal scores.
Script Adherence live tracker
A stage-by-stage view of whether the agent is following the sales script — which stages are covered, which were skipped, and where the conversation currently is — updated live from the agent-side transcript.
Resilient ASR key pool concurrency
Round-robin distribution of streaming-ASR keys across concurrent calls, with automatic retirement of credit-exhausted keys persisted across restarts — built so a full floor of simultaneous calls (two streams each) degrades gracefully instead of failing loudly.
Idle fallback silence handling
A self-resetting 15-second timer that re-engages the coaching agent when the customer stops talking — passing an avoid-list of prior suggestions so the panel offers something new instead of repeating itself.
Call simulator test harness
A FastAPI service that replays real recorded calls as live dual-channel streams — with one-click presets — so the whole real-time pipeline can be exercised, demoed and load-tested without a single live customer on the line.
Where it stands
What building a real-time copilot taught me
- Real-time inverts LLM product design. A continuously-updating agent sounds smarter but arrives late and repeats itself; firing once per customer turn — with the turn as the unit of thought — made suggestions both faster and more relevant.
- Silence is a state, not an absence. The copilot's worst failure mode wasn't a wrong suggestion — it was going quiet during customer silence, exactly when the agent was improvising. Designing an explicit idle state fixed what better prompts never could.
- Classify vendor errors precisely. An over-broad "credits exhausted" matcher was retiring healthy API keys on unrelated quota errors. In a key-pool design, one sloppy regex silently shrinks your capacity.
- Write for the agent's ear, not the customer's. The same insight fails as a script line ("humare paas 30,000 searches hain") and lands as internal guidance ("customer ko batao is category mein demand high hai"). Phrasing is a product feature.
- Validate identifiers at every boundary. A look-alike ID passed verbatim into an upstream API caused expensive scans on someone else's system. IDs that flow across services need format checks at the edge, not trust.