Voice Agent Latency Engineering: Turns and Barge-In

TLDR

A low-latency voice agent is a distributed real-time system whose correctness depends on more than streaming. It must distinguish an unstable ASR hypothesis from a committed user turn, isolate every response with a generation epoch, know what audio the client actually played, and measure latency from named events on comparable clocks. Barge-in is a cancellation and state-reconciliation protocol, not a single VAD callback. Production evaluation must combine conversational timing with task completion, policy compliance, tool effects, and recovery.

Table of Contents

• Key Takeaways
• The Production Correctness Model
• Define Latency as Event Intervals
• Reference Architecture and Clock Boundaries
• ASR Hypotheses and Turn Commit
• Turn Taking Is More Than Voice Activity
• Generation Epochs and Playback Truth
• A Playback Aware Barge In Protocol
• Transport and WebRTC Observability
• Security Privacy and Tool Effects
• Evaluation Contract
• Architecture Selection
• Best Practices
• FAQ
• Summary

Key Takeaways

• A partial ASR result is a hypothesis, not conversation truth. Use it only for work that can be discarded without an external effect.
• Turn commit is an application event. VAD speech-stop, ASR finalization, and semantic completion are evidence for that decision, not interchangeable names for it.
• Cancellation needs an epoch and a playback acknowledgement. Otherwise late model tokens, synthesized chunks, or client-buffered audio can leak into the next turn.
• Measure user-perceived latency on one clock when possible. Do not subtract unrelated browser and server wall clocks without a synchronization model and uncertainty.
• Overlap has multiple meanings. A true interruption should yield the floor; a backchannel, side conversation, or background speaker often should not.
• Voice quality and task correctness are one release decision. A fast, natural agent that performs the wrong tool action is still a failed agent.

The Production Correctness Model

A production voice agent should model one user turn and one assistant generation as a sequence of explicitly owned state transitions. “Listening, thinking, speaking” is too coarse because it cannot answer whether text was tentative, whether audio was only synthesized or actually heard, or whether an external action completed.

Keep at least four records:

Tool execution is a separate state machine. A committed transcript can propose an action, but only trusted application code can authorize it. The system also needs an outcome_unknown state when a timeout occurs after dispatch: retrying blindly could duplicate a payment, booking, or message.

Define Latency as Event Intervals

Voice latency becomes actionable only when every metric names its start event, end event, clock, and inclusion rules. A single “response latency” value hides endpointing delay, model work, synthesis, transport, and client buffering.

Useful intervals include:

endpointing = turn_committed - user_speech_end
decision = first_accepted_unit - turn_committed
synthesis_and_delivery = first_playable_audio - first_accepted_unit
perceived_response = playback_started - user_speech_end
barge_in_stop = playback_stopped - interruption_speech_start

Do not publish a universal budget for these intervals. Establish a reproducible workload contract with language, accent, task, codec, device, network profile, concurrency, model revision, and tool path. Report distributions by slice. A p95 that combines clean broadband greetings with lossy mobile payment flows is not an operational SLO.

An acknowledgement is useful only when it is semantically honest. “I am checking” may reduce silence before a slow tool, but it must not be counted as a successful first answer or played before the application has decided that the tool is allowed.

Reference Architecture and Clock Boundaries

The voice gateway should carry media, while the orchestrator owns turn commit, authorization, epochs, cancellation, and trace correlation. The client owns the strongest evidence of user-perceived timing because it observes capture and playback on the same device.

Within one process, record durations with a monotonic clock. Across browser, gateway, model provider, and tool service, preserve each event’s clock domain and correlation ID. Wall-clock timestamps help align traces, but subtracting two machines’ wall clocks is invalid unless the trace records clock synchronization and its error bound.

For end-to-end perceived response and interruption stop time, prefer two events observed on the client’s monotonic timeline. Server spans still explain where time went; they should not pretend to be the same clock.

ASR Hypotheses and Turn Commit

Streaming speech recognition produces hypotheses that can change as more audio arrives. A partial can warm a cache, retrieve candidate documents, or begin cancelable decoding, but it must not become durable history or trigger an irreversible action.

type Hypothesis = {
  revision: number;
  text: string;
  audioEndMs: number;
  providerFinal: boolean;
};

type UserTurn =
  | { state: "candidate"; hypothesis: Hypothesis }
  | {
      state: "committed";
      turnId: string;
      transcript: string;
      audioStartMs: number;
      audioEndMs: number;
    };

function mayStartWork(kind: "retrieval" | "tool_effect", turn: UserTurn): boolean {
  if (kind === "retrieval") return true; // Result stays speculative until commit.
  return turn.state === "committed";     // Authorization is still required later.
}

Provider “final” is not always application “committed.” The application may wait for endpointing evidence, merge a correction, require push-to-talk release, or reject an empty or low-confidence segment. Conversely, manual control may commit an audio boundary before the final transcript arrives.

The OpenAI Realtime VAD guide currently documents server and semantic VAD modes plus speech-start and speech-stop events. Those are provider-specific controls, not a universal turn protocol. LiveKit turn management similarly distinguishes VAD, endpointing, semantic detection, manual clear, commit, and interrupt operations. Use these APIs as implementations of your contract, not as the contract itself.

Turn Taking Is More Than Voice Activity

Voice activity answers whether speech-like audio exists; it does not answer who is speaking, whom they address, or whether they want the floor. Treating every detected voice as an interruption causes false cutoffs and makes the agent brittle in homes, cars, offices, and calls.

Test at least four overlap classes:

Full-Duplex-Bench v1.5 formalizes these four controlled scenarios and defines stop latency from user-speech start to model stop, and response latency from user-speech end to model restart. Its exact results belong to its tested models and audio protocol; the reusable lesson is to evaluate scenario-specific behavior rather than optimize one interruption rate.

An overlap classifier can use acoustic direction, enrolled-device context, lexical cues, duration, dialogue state, and explicit controls. It will still make mistakes. Design a repair path: restore an incorrectly ducked response when possible, ask a short clarification when addressee confidence is low, and never infer authenticated identity from voice characteristics alone.

Generation Epochs and Playback Truth

Every assistant response needs an immutable generation epoch so that stale output can be rejected at every boundary. Increment the epoch when the user commits a new turn, policy changes, or an interruption invalidates the current answer.

Each text or audio unit should carry:

{
  "sessionId": "sess_123",
  "turnId": "turn_009",
  "generationEpoch": 18,
  "unitId": "unit_004",
  "text": "Your order is scheduled",
  "audioStartSample": 38400,
  "audioEndSample": 62400
}

The model may have generated ten units, TTS may have synthesized seven, the server may have sent five, and the client may have played only three. Persisting the generated answer as if the user heard it poisons later context. Update durable conversation history from playback progress or a final playback acknowledgement, not from generation completion.

If a playback acknowledgement is lost, record the heard boundary as uncertain. Do not invent an exact transcript prefix from server send time. The next response can avoid references such as “as I just said” until state is reconciled.

A Playback Aware Barge In Protocol

Correct barge-in coordinates detection, classification, cancellation, client playback, and state reconciliation. Clearing the server TTS queue is insufficient because audio may already be in transit, decoded, or playing.

from dataclasses import dataclass
from typing import Awaitable, Callable, Optional


@dataclass(frozen=True)
class PlaybackAck:
    epoch: int
    last_played_unit_id: Optional[str]


class VoiceSession:
    def __init__(
        self,
        cancel_model: Callable[[int], Awaitable[None]],
        cancel_tts: Callable[[int], Awaitable[None]],
        flush_client: Callable[[int], Awaitable[PlaybackAck]],
    ) -> None:
        self.epoch = 0
        self.cancel_model = cancel_model
        self.cancel_tts = cancel_tts
        self.flush_client = flush_client
        self.committed_transcript: list[dict[str, str]] = []

    async def interrupt(self, overlap_class: str) -> Optional[PlaybackAck]:
        if overlap_class != "user_interruption":
            return None

        invalid_epoch = self.epoch
        self.epoch += 1  # Late chunks from invalid_epoch must now be rejected.

        await self.cancel_model(invalid_epoch)
        await self.cancel_tts(invalid_epoch)
        ack = await self.flush_client(invalid_epoch)

        # Resolve unit_id through the epoch's immutable text-audio manifest.
        if ack.last_played_unit_id is not None:
            self.committed_transcript.append({
                "role": "assistant",
                "heardThrough": ack.last_played_unit_id,
            })
        return ack

The client must reject any media chunk whose epoch is no longer active. A practical sequence is:

1. detect overlapping user speech;
2. classify it as interruption, backchannel, side speech, or background speech;
3. invalidate the current generation epoch;
4. cancel the model stream and TTS synthesis cooperatively;
5. send a client flush command with the invalid epoch;
6. stop or duck playback and return the last played unit or sample cursor;
7. commit only the confirmed assistant prefix;
8. continue collecting the user’s revisable ASR hypotheses;
9. commit the new user turn;
10. authorize and start a new epoch.

Put timeouts around every cancellation step. A timeout changes the state to degraded or unknown; it does not prove cancellation succeeded.

Transport and WebRTC Observability

Choose transport from the media environment rather than from a slogan. WebRTC provides browser real-time media APIs, congestion behavior, jitter buffering, echo-control integrations, and getStats(); WebSocket framing can be simpler when the application controls both endpoints; SIP introduces telephony codecs and infrastructure; native mobile stacks expose platform-specific audio routing.

The WebRTC 1.0 Recommendation defines browser real-time communication APIs. It does not define a voice-agent pipeline or an application SLO. The WebRTC Stats specification defines fields for RTP, packet loss, jitter, candidate pairs, and audio playout, but the current document is a Candidate Recommendation Draft and browsers may not expose every field.

For cumulative stats, sample twice and compute deltas over the interval:

async function sampleInboundAudio(peerConnection) {
  const report = await peerConnection.getStats();
  return [...report.values()]
    .filter((stat) => stat.type === "inbound-rtp" && stat.kind === "audio")
    .map((stat) => ({
      id: stat.id,
      timestamp: stat.timestamp,
      packetsLost: stat.packetsLost,
      jitter: stat.jitter
    }));
}

Feature-detect fields and preserve raw samples. Network metrics explain packet delivery; they do not replace application events such as turn commit, accepted semantic unit, flush request, playback acknowledgement, or tool outcome.

Security Privacy and Tool Effects

Speech is untrusted input, not identity or authorization. A voice agent must resolve the authenticated principal from the session, check tenant and object ownership in trusted code, validate exact tool arguments, and require action-bound confirmation for high-impact effects.

Apply these boundaries:

• partial ASR can only start reversible computation;
• committed text can propose a tool call but cannot authorize it;
• tool execution uses idempotency keys and an explicit outcome state;
• confirmations name the exact action, object, amount, and destination;
• interruption cannot silently change the target of an already confirmed action;
• audio, transcripts, traces, and replay artifacts have consent, retention, redaction, and access policies;
• speaker recognition, when separately deployed, is not a substitute for session authentication.

Do not retain raw audio merely because it helps debugging. Keep the minimum artifact needed for the declared purpose, restrict replay access, and make deletion cover derived transcripts, caches, indexes, and evaluation datasets.

Evaluation Contract

A release gate should reproduce both the acoustic experience and the business task. The recent tau-Voice benchmark combines grounded task completion, full-duplex interaction, and realistic audio. Its reported model scores are specific to its tasks and simulator; its durable contribution is the joint evaluation shape.

workload:
  languages: [en-US, zh-CN]
  devices: [laptop-headset, mobile-speaker]
  codecs: [opus, telephony-narrowband]
  networkProfiles: [clean, jitter, loss, reconnect]
  taskSlices: [information, retrieval, read-only-tool, state-changing-tool]
  overlapScenarios: [interruption, backchannel, talking-to-others, background-speech]

timingEvents:
  - user_speech_end
  - turn_committed
  - first_accepted_unit
  - first_playable_audio
  - playback_started
  - interruption_speech_start
  - playback_stopped

releaseMetrics:
  - task_completion
  - tool_argument_accuracy
  - policy_adherence
  - false_cutoff_rate
  - false_interruption_rate
  - recovery_success
  - stale_audio_after_cancel
  - perceived_response_p50_p95_p99_by_slice
  - cost_per_successful_turn

Run deterministic audio fixtures in CI and sampled end-to-end sessions before release. Keep model, prompt, endpointing, codec, and client revision in the result. Compare candidates on paired trials when possible. A regression in policy adherence or task completion must block a latency “win.”

Architecture Selection

Cascaded and audio-native systems expose different control surfaces, but neither removes the need for the application contract.

Benchmark both with the same workload. Do not conclude that a cascade is always more controllable or that an audio-native model is always faster or more natural. Provider implementation, model revision, prompt, language, network, client buffering, and task policy can reverse the result.

Best Practices

1. Name every event before setting an SLO. A metric without start and end semantics cannot diagnose a regression.
2. Keep hypotheses separate from committed state. Speculation may save time only when it is cheap to discard.
3. Attach an epoch to every output unit. Reject stale text and audio at model, TTS, gateway, and client boundaries.
4. Track the heard prefix. Generated, synthesized, sent, and played are different states.
5. Classify overlap before yielding. Tune interruption, backchannel, side-speech, and background-speech behavior separately.
6. Use monotonic clocks for durations. Correlate cross-service events without pretending unrelated wall clocks are exact.
7. Authorize effects outside the model. Voice and transcript content cannot grant permissions.
8. Gate on task and interaction quality. Report distributions by real workload slice, not one blended average.

FAQ

What latency should a production voice agent target?

There is no universal target. Define user_speech_end, turn_committed, first_accepted_unit, first_playable_audio, and playback_started, then measure their intervals on representative devices, languages, networks, and tasks. Report p50, p95, and p99 with task success and reprompt rates. A fast filler phrase is not equivalent to a useful answer.

Can an ASR partial transcript trigger a tool call?

It may prepare reversible work, such as retrieval or a cache lookup. It should not enter durable history or cause an external side effect because later audio may revise it. After turn commit, trusted code must still validate identity, object access, arguments, policy, and any required confirmation.

What does correct barge-in handling require?

Correct barge-in invalidates the response epoch, cancels model and TTS work, stops audio already buffered by the client, and records the assistant prefix the user actually heard. It then waits for the new user turn to become stable before authorizing another response. Clearing only the server queue leaves stale in-flight and client-buffered audio.

Should teams choose a cascade or an audio-native model?

Test both under one contract. A cascade exposes transcripts and independent components; an audio-native model may retain acoustic and prosodic context. Both need playback-aware cancellation, tool authorization, traceability, privacy controls, and end-to-end evaluation. Architecture labels alone do not establish latency or quality.

How should full-duplex behavior be tested?

Separate true interruption, backchannel, speech directed to another person, and background speech. Measure whether the agent responds or resumes appropriately, how quickly audible output stops when it should, how quickly it recovers, and whether stale audio or tool effects escape cancellation. Pair these metrics with task completion and policy adherence.

Summary

Reliable voice-agent latency engineering starts with state semantics, not a universal millisecond budget. Commit user turns explicitly, isolate responses with generation epochs, reconcile conversation history with client playback, and treat barge-in as a distributed cancellation protocol. Measure perceived timing on comparable clocks, test distinct overlap scenarios, and release only when conversational quality, task correctness, policy compliance, and recovery pass together.

Related Resources

• Speech Recognition Glossary
• Latency Glossary
• AI Agent Development Guide
• Agent Observability Engineering
• Multimodal AI Pipeline Engineering
• WebRTC 1.0 Recommendation
• WebRTC Statistics API
• Full-Duplex-Bench v1.5
• tau-Voice