Hiding extraction latency behind parallel agent nodes
The AI Voice Agent Receptionist answers live inbound calls from real estate sellers over Twilio and qualifies them the way a human receptionist would: holding a conversation, asking the right follow ups, and getting the lead into the CRM by the time the caller hangs up. A phone call gives you no room to think. There is no spinner a caller will tolerate, no "one moment please" that reads as anything other than the system being slow. The budget I designed against was sub-second to first audio, every turn, for the whole call.
That budget is the whole story here. It is short enough that anything I add to a turn has to earn its place, and long enough that a naive implementation blows through it without doing anything obviously wrong.
Where the time actually goes
Walk one turn end to end. The caller finishes speaking. Their audio, arriving over Twilio Media Streams, gets transcribed by faster-whisper. The transcript goes into a LangGraph agent running on Google Gemini, which reasons about where the conversation is and produces a reply. That reply is synthesised back to speech through a Kokoro ONNX TTS pipeline, streamed, and sent back down the same Twilio Media Streams connection the caller is still listening to.
Every one of those stages is individually reasonable. Transcription is fast. A single Gemini call is fast. Kokoro streaming synthesis starts emitting audio before the whole utterance is generated. None of them, looked at alone, looks like a latency problem worth writing about.
The sum is not reasonable. Four stages, each contributing its own latency, chained in sequence, is a pipeline whose total time is the sum of its parts. A sub-second budget for first audio does not have room for four sequential stages plus whatever else gets added to the chain. The moment I ask the agent to do anything beyond "transcribe, reason, speak," that addition lands directly on top of the budget, because the pipeline is a straight line and every node on it sits on the critical path by default.
The sequential trap
The agent has two jobs on every turn, not one. It has to hold a natural conversation, and it has to extract structured CRM data from what the caller just said, whatever fields the conversation has revealed so far. Both jobs need the same input: the transcript and the conversation state.
The obvious way to structure that is two steps. Reply first, then extract from the same turn once the reply is on its way. Or extract first, then reply once the fields are pulled, so the extraction can inform how the agent responds.
Either ordering has the same problem, because either way the two steps run one after the other, and one of them is on the critical path the caller actually experiences. Extract-then-reply delays the reply directly: the caller is waiting through a full model round trip for structured output before they hear anything back. Reply-then-extract looks safer at first, since the reply goes out first, but the graph still has to wait for the extraction step to finish before it can consider the turn done and move on, and that tail latency shows up as a longer gap before the agent is ready to hear the caller again. There is no ordering of two sequential steps where only one of them costs time. The caller pays for whichever one runs second, or for the turn overall taking longer to close out, in either arrangement.
Running both branches at once
The fix was to stop treating "reply" and "extract" as a sequence at all. The LangGraph graph fans both nodes out from the same turn state (the transcript, the conversation history, whatever fields have already been captured) and runs them concurrently. The conversational reply node calls Gemini for a natural response. The extraction node calls Gemini, separately, for structured output against the CRM schema. Both start from the same snapshot of state and neither waits on the other to begin.
The reasoning that makes this safe is about what is actually on the critical path. Only the reply is on the critical path the caller experiences, because the reply is the only output that becomes audio. The extraction branch writes its result into the turn state and into PostgreSQL, but nothing downstream of it needs to finish before the agent can speak. As long as the extraction branch completes before the next turn needs that state, its latency is invisible. It does not need to be faster than the reply node in some absolute sense. It needs to be finished by the time it would next be read, which is a much looser constraint than finished before the reply goes out. Capturing structured data stopped costing perceived response time, because the graph stopped forcing something off the critical path to sit on it.
What concurrency costs you
Running two branches instead of one is not free, and it is worth being honest about where the cost lands.
Both branches call the model on every turn, so cost roughly doubles per turn compared to a single reasoning call. That is a straightforward trade against the alternative, which is a caller experience that fails the latency budget.
The extraction branch reads the same turn state the reply branch reads, and it has to be constrained to never mutate that shared state while the reply branch is still reading it. The two branches are concurrent, not coordinated, so any write one of them makes has to be scoped so it cannot race with what the other is doing with the same snapshot.
And a slow extraction branch is a hidden cost until it is not hidden anymore. Its latency is invisible only while it stays inside its window: the time until the next turn actually needs its output. If a Gemini call for extraction runs long, that latency eventually surfaces, either as stale fields on the next turn or as a bottleneck if something downstream blocks on it. The extraction node needs a timeout, and the graph needs a merge strategy for what happens to the turn's state when extraction does not finish in time. Dropping the partial result cleanly is a real decision, not something to leave implicit.
Barge-in is a separate problem
Callers interrupt. They talk over the agent mid sentence, and the agent has to stop talking and listen. Barge-in handling is orthogonal to the graph shape: it does not run inside the reply or extraction nodes, and parallelising those two nodes does nothing to solve it. What barge-in actually requires is cancelling in-flight TTS audio the instant new caller speech is detected, not just cancelling the reasoning that produced it. A reply node that finishes its Gemini call and hands off to synthesis has already done the reasoning work; the audio still streaming down the Twilio connection is what has to be cut, immediately, or the caller hears the agent talk over them.
The general lesson
None of this is specific to voice agents or to LangGraph. On a hard latency budget, the question that matters is which work is actually on the critical path the user experiences, not which work exists. Everything else, however necessary it is to the product, belongs off that path, running concurrently with what the user is waiting on, as long as it finishes before its result is next needed. The budget does not get bigger by asking the agent to do less. It gets bigger by asking fewer things to wait on each other.
This came out of the AI Voice Agent Receptionist.