NVIDIA's VoiceChat 11B collapses the traditional ASR → LLM → TTS pipeline into a single streaming network . A Fast Conformer speech encoder ingests 16 kHz audio, the Nemotron Nano v2 hybrid Mamba/Transformer backbone reasons over it, and a TTS decoder emits 22.05 kHz audio — all in one loop . It listens while it speaks, handles interruption in roughly 480 ms, and has a separate output channel for tool calls .
The technical numbers are impressive. 448 ms smooth turn-taking latency. 0.82 turn-over rate on Full-Duplex-Bench 1.0. Ranked #2 among open full-duplex models on VoiceBench . Trained on roughly 550,000 hours of audio .
But the numbers that matter for most businesses aren't on the benchmark sheet.
The cost comparison tells the real story. A single H100 GPU runs about $2-3 per hour on demand. A continuous inference instance costs roughly $1,500-2,200 per month . VoiceChat's weights are 44.4 GB in float32, requiring an 80 GB GPU — A100, H100, H200, B100, or B200 .
Compare that to the API model. OpenAI's Realtime API charges per audio minute. VoiceChat's cost is fixed amortization. For any organization running voice agents at scale, the per-hour economics flip dramatically .
This is the first open full-duplex model with live tool calling. When the model triggers a function call, it emits a <TOOLCALL> block on a side channel while continuing to speak, using operator-defined "on-hold" lines to fill the gap while the API runs . It's the voice-agent equivalent of "let me look that up for you" — without dead air.
But the gap between the demo and production is substantial. On AU Harness BFCL-v3, VoiceChat averages 56.1% on spoken tool calling — 58.5% simple tasks, 62.5% multiple tasks, 42.5% parallel, 89.6% on irrelevance detection. On Full-Duplex-Bench v3, tool selection hits 82.5%, but argument accuracy drops to 44.2%, with a pass@1 of just 33% .
NVIDIA is transparent about the limits. Five tools maximum per session. No reliable parallel tool calls. Users cannot interrupt during tool execution. ASCII-only system prompts and tool responses .
The release is marked "for research purposes only" . It's English-only, capped at a two-minute audio context, can degrade into non-recoverable gibberish after several turns, and has documented cases of "runaway continuation" — the model keeps speaking or starts new turns without user input . It's not suitable for noisy environments, and background speech causes degraded output .
For contact centers, clinical intake, or regulated support desks, these are showstoppers. For researchers building the next generation of voice agents, they're known constraints to be engineered around .
The architecture matters more than the demo clip. VoiceChat's unified streaming design removes one big source of orchestration debt — the cross-service glue code that makes cascaded stacks brittle. But it shifts the bug pattern: less API handoff error, more pressure on prompt discipline, turn-state handling, and runtime guardrails .
VoiceChat is the first open release that treats duplex conversation and agentic tool use as one problem rather than two. That's the signal. The 448ms number is the hook. The economics are the reason it matters .
P.S. The real question isn't whether VoiceChat is production-ready today — it isn't. The question is whether the economics of self-hosted voice agents will force the market to accept the gap between "good enough" and "API perfect" faster than anyone expects. The answer depends on how fast NVIDIA can close the 33% pass@1 gap.
Frequently Asked Questions
Q: What is NVIDIA NemotronLabs VoiceChat 11B?
A: VoiceChat 11B is an 11-billion-parameter open-weight, full-duplex speech model released by NVIDIA on August 9, 2026. It processes and generates speech in a single streaming network (speech→reasoning→speech) with live tool calling. It runs on a single 80GB GPU, was trained on roughly 550,000 hours of audio, and has 32,000 vocabulary word pieces.
Q: What does "full-duplex" mean?
A: Full-duplex means the model can listen and speak simultaneously, handling interruptions and overlapping speech. The traditional voice pipeline (ASR → LLM → TTS) is half-duplex or near-duplex — one direction at a time. VoiceChat's unified network enables true conversation, with smooth turn-taking latency of 448 ms.
Q: What is the cost advantage of self-hosting VoiceChat?
A: OpenAl's Realtime API charges roughly $11+ per conversation hour for flagship models. Self-hosting VoiceChat on a single H100 GPU costs $2–3 per hour in compute amortization — fixed cost regardless of usage volume. For organizations running voice agents at scale, this is a massive cost shift from variable API charges to fixed hardware amortization.
Q: Can VoiceChat call tools and APIs?
A: Yes. VoiceChat has a separate output channel for tool calls. When the model triggers a function call, it emits a <TOOLCALL> block on a side channel while continuing to speak, using operator-defined "on-hold" lines to fill the gap. It supports up to five tools per session, but cannot reliably call multiple tools in parallel yet.
Q: Is VoiceChat production-ready?
A: No. NVIDIA explicitly marks it "for research purposes only." Known limitations include: English-only, two-minute audio context limit, degradation into non-recoverable gibberish after several turns, and documented "runaway continuation" cases where the model keeps speaking without input. It's not suitable for noisy environments or background speech.
Q: How accurate is its tool calling?
A: On Full-Duplex-Bench v3, tool selection accuracy is 82.5%, but argument accuracy drops to 44.2%, with a pass@1 of 33%. On the benchmark for spoken tool calling, it averages 56.1% (58.5% on simple tasks, 62.5% on multiple tasks, 42.5% on parallel tasks, 89.6% on irrelevance detection).
Q: What hardware does VoiceChat 11B require?
A: The model weights are 44.4 GB in float32. It requires a single 80GB GPU: A100, H100, H200, B100, or B200. Inference runs in FP16 or lower precision, but NVIDIA has not published FP16/INT8 memory requirements.
Q: How does the architecture differ from traditional voice agents?
A: Traditional voice AI uses three separate models: ASR (speech→text), LLM (text→text reasoning), and TTS (text→speech). VoiceChat collapses all three into a single unified network: Fast Conformer speech encoder → Nemotron Nano v2 backbone → TTS decoder. This reduces orchestration debt and eliminates cross-service API handoff errors.
Q: When can we expect voice agents to use this type of model at scale?
A: VoiceChat is currently a research prototype with significant limitations. For enterprise voice agents to deploy this model at scale, NVIDIA needs to improve tool argument accuracy (from 33% to over 80%), extend audio context (from 2 minutes to hours), add multi-language support, and clean up gibberish/stability issues. Progress is likely in the next 6-12 months, as NVIDIA trains larger and more robust versions.
