Dissecting Streaming TTS Inference Bottlenecks

This post is the second entry in the TTS Inference Deep Dive series.
Part 1: Introducing OmniVoice covered the microservice architecture and core APIs.

Related Series

• TTS Inference Deep Dive — Part 1: Introducing OmniVoice
• TTS Inference Deep Dive — Part 2: Dissecting the Bottlenecks in Streaming Inference (current post)

---

Introduction

When I first wired up the /tts/stream endpoint, the mental model was straightforward: TTS generates audio in lockstep with the LLM's output, and the user hears the first word before the response is even finished.

Reality was different. The perceived latency was nearly identical to waiting on a single non-streaming request for the full text — and in some cases, streaming was actually slower.

Streaming is enabled. Why is it still slow?

There's no single culprit. Bottlenecks stack up across four layers: the structural characteristics of the diffusion model, OmniVoice server concurrency constraints, the await chain in the backend adapter, and the frontend audio queue. This post dissects each of the four.

---

First, Let's Be Clear: What Does "Streaming" Mean Here?

This is where the confusion starts.

The typical image of streaming audio is chunk-level streaming — audio data flowing out in hundreds-of-milliseconds increments as it's generated. AR (autoregressive) TTS models — architectures like VALL-E or XTTS that predict audio tokens one at a time — can do this. Playback can start the moment the first token is produced.

OmniVoice is a diffusion-based model. It starts from Gaussian noise and reconstructs a spectrogram through num_step full reverse-diffusion passes. There is no intermediate output to extract. Whether it's 16 steps or 32, audio data only exists after the final step completes. Chunk-level streaming is architecturally impossible.

What /tts/stream actually implements is sentence-level streaming.

text → sentence split → [sentence 1] → OmniVoice → audio 1 → NDJSON chunk
                      → [sentence 2] → OmniVoice → audio 2 → NDJSON chunk
                      → [sentence 3] → OmniVoice → audio 3 → NDJSON chunk

As soon as one sentence is synthesized, the result is sent immediately and synthesis of the next sentence begins. The main benefit is that TTFA (Time To First Audio) is bounded by the synthesis time of a single sentence rather than the full text.

This is where the outline of the first bottleneck comes into view.

---

Bottleneck 1: The Original Sin of Diffusion Models — A High TTFA Floor

In an AR model, TTFA is roughly the time to generate the first token. With a well-optimized model, that can be under 100 ms.

In a diffusion model, TTFA is the full synthesis time for the first sentence. On a Pascal-class GPU (GTX 1070), with num_step=16 and a sentence around 20 characters, that's roughly 1.5–3 seconds.

On TTFA alone, diffusion looks worse. But it has clear advantages in overall throughput and clone quality. Understanding the trade-off before drawing conclusions is a very different thing from just calling it "slow."

Lowering the Floor with Adaptive num_step

The adaptive num_step logic introduced in Part 1 is an attempt to reduce this TTFA floor:

# server/engine.py
TEXT_STEP_TABLE = [
    (10,  6),    # ≤10 chars   → 6 steps  (exclamations, short reactions)
    (25, 10),    # 11–25 chars → 10 steps (short sentences)
    (50, 14),    # 26–50 chars → 14 steps (medium sentences)
    (None, cfg_num_step),   # 51+ chars  → use configured value
]

def adaptive_num_step(text: str, cfg_num_step: int) -> int:
    n = len(text)
    for threshold, step in TEXT_STEP_TABLE:
        if threshold is None or n <= threshold:
            return min(cfg_num_step, step)
    return cfg_num_step

Processing a 3-character exclamation like "어머!" at 6 steps instead of 16 is roughly 2.5× faster. In streaming scenarios where short sentences appear in rapid succession — Geny's VTuber response pattern is exactly this — the cumulative effect is significant.

This isn't a fundamental fix. It's trimming. Sub-hundred-millisecond TTFA is structurally off the table for diffusion architectures.

---

Bottleneck 2: The Serialization Hell of MAX_CONCURRENCY=1

Part 1 mentioned that the default value of OMNIVOICE_MAX_CONCURRENCY is 1. The impact this has on streaming performance is larger than it might seem.

Scenario: an LLM response with 5 sentences, each taking 2 seconds to synthesize.

T+0s    sentence 1 synthesis starts
T+2s    sentence 1 done → sent immediately  ← user hears first audio
T+2s    sentence 2 synthesis starts
T+4s    sentence 2 done → sent
T+4s    sentence 3 synthesis starts
...
T+10s   sentence 5 done → sent

There's a 2-second silence gap between sentences. If audio playback and synthesis aren't pipelined, this feels more choppy than a single non-streaming request.

The ideal is pipelining: sentence 2 synthesizes while sentence 1 is playing.

T+0s    sentence 1 synthesis starts
T+2s    sentence 1 done → playback starts (2-second audio)
T+2s    sentence 2 synthesis starts          ← simultaneously
T+4s    sentence 1 playback ends
T+4s    sentence 2 done → plays immediately  ← seamless

For this pipelining to hold, RTF (Real-Time Factor) ≤ 1.0 is required. Synthesis time must not exceed audio playback duration.

Even if the RTF condition is met, there's one more trap. The HTTP round-trip time between when sentence 1 completes and when the sentence 2 request arrives at the server leaves the GPU idle:

T+2.000s  sentence 1 synthesis done, HTTP response returned
T+2.050s  backend receives it → sends sentence 2 HTTP request
T+2.100s  server receives it, acquires semaphore, starts synthesis
            ↑ 100ms GPU idle

Over 5 sentences, that's 500 ms of accumulated GPU idle time. In practice, this is surprisingly noticeable.

Fix: Expanding Concurrency

If GPU VRAM allows, setting OMNIVOICE_MAX_CONCURRENCY=2–3 is the most direct solution.

# docker-compose.yml
environment:
  OMNIVOICE_MAX_CONCURRENCY: "2"

With concurrency=2, two sentences are processed simultaneously:

T+0s    sentences 1 & 2 synthesis starts (concurrent)
T+2s    sentences 1 & 2 done
T+2s    sentences 3 & 4 synthesis starts (concurrent)
T+4s    sentences 3 & 4 done
T+4s    sentence 5 synthesis starts
T+6s    done
→ serial: 10s → parallel: 6s

Note that bumping concurrency to 2 doesn't yield exactly 2× throughput. Sharing the same GPU increases individual synthesis times. OmniVoice occupies 6–8 GB VRAM in float16, so on a 24 GB system, concurrency=2–3 is a realistic ceiling.

---

Bottleneck 3: The Giant Bubble Between LLM and TTS

This is the most commonly underestimated bottleneck.

The ideal streaming TTS flow:

LLM tokens: ──[sent1 complete]────[sent2 complete]────[sent3 complete]──
                      ↓                    ↓                    ↓
TTS synthesis: ──[sent1 synth]────[sent2 synth]────[sent3 synth]──
                ↑ LLM and TTS pipelined ↑

The actual Geny architecture:

LLM Agent → TTSService.synthesize_stream(full_text)
                 │
                 └── OmniVoiceAdapter.stream(full_text)
                               │
                               └── POST /tts/stream

synthesize_stream receives the full text only after the LLM has finished generating, then calls /tts/stream. There is a bubble at the front of the pipeline equal to the full LLM generation time:

Actual timeline:

  ├────────── LLM response generation (2,000ms) ──────────┤
                                                          ├─[s1]─┤─[s2]─┤─[s3]─┤
                                                          ↑
                                                   TTS only starts here

Effective TTFA = LLM generation time + first sentence synthesis time. If the LLM bubble dominates, enabling streaming changes nothing perceptible.

Fix: Live-Split Pipelining

Subscribe to the LLM token stream and start a TTS task immediately upon detecting a sentence boundary.

async def live_tts_pipeline(llm_token_stream, tts_adapter):
    buffer = ""
    tts_tasks: list[tuple[int, asyncio.Task]] = []
    idx = 0

    async for token in llm_token_stream:
        buffer += token

        if is_sentence_boundary(buffer):
            sentence = buffer.strip()
            buffer = ""
            # Fire off TTS task as soon as sentence is complete — don't await
            task = asyncio.create_task(
                tts_adapter.synthesize(sentence, index=idx)
            )
            tts_tasks.append((idx, task))
            idx += 1

    # Handle remaining buffer
    if buffer.strip():
        task = asyncio.create_task(
            tts_adapter.synthesize(buffer.strip(), index=idx)
        )
        tts_tasks.append((idx, task))

    for i, task in sorted(tts_tasks, key=lambda x: x[0]):
        yield await task

Live-split timeline:

LLM: ──[sent1 complete]────────────[sent2 complete]────────[sent3 complete]──
              ↓                            ↓                       ↓
TTS: ──[sent1 synth]────────────[sent2 synth]────────[sent3 synth]──
       ↑ TTS is already running while LLM is still generating ↑

TTS starts the moment the LLM finishes the first sentence. While the LLM generates the second sentence, the first synthesis runs in parallel.

The Pitfalls of Sentence Boundary Detection

is_sentence_boundary() is trickier than it looks:

TERMINATORS = frozenset("。！？.!?…")
MIN_SENTENCE_CHARS = 8

def is_sentence_boundary(text: str) -> bool:
    stripped = text.rstrip()
    if len(stripped) < MIN_SENTENCE_CHARS:
        return False
    last = stripped[-1]
    if last not in TERMINATORS:
        return False
    # Prevent treating decimal points as sentence terminators — e.g. "3.14는 원주율"
    if last == "." and len(stripped) >= 2 and stripped[-2].isdigit():
        return False
    return True

It's not perfect, but it meaningfully reduces false positives in production. A naive implementation works reasonably well for Korean-only text, but in a mixed-language environment like Geny's, these guards are necessary.

---

Bottleneck 4: Head-of-Line Blocking

Say we've applied the improvements from bottlenecks 2 and 3 and are now dispatching sentences in parallel. A new problem surfaces.

Diffusion inference time is non-deterministic. Even when synthesizing two sentences concurrently, you can't know exactly which one will finish first:

Sentence 1 (8 chars,  6 steps):  ────────── done (250ms)
Sentence 2 (52 chars, 14 steps): ──────────────────── done (480ms)
Sentence 3 (6 chars,  6 steps):  ─────── done (220ms)

Sentence 3 finishes before sentence 2. If the playback queue processes chunks in arrival order, audio comes out in the sequence 3→1→2, breaking the narrative.

Add index-based sorting to enforce ordering, and you get:

Sentence 1 done → plays immediately ✓
Sentence 3 done → must wait for sentence 2 ✗  ← Head-of-Line Blocking
Sentence 2 done → sentences 2 and 3 play back-to-back ✓

A fast sentence gets held up by a slow one.

Fix: Index-Based Ordering Buffer

Use heapq to drain completed sentences in index order:

import heapq
import asyncio

class OrderedAudioBuffer:
    """Drains audio in index order regardless of completion order."""

    def __init__(self):
        self._heap: list[tuple[int, bytes]] = []
        self._next_expected = 0
        self._lock = asyncio.Lock()
        self._event = asyncio.Event()

    async def put(self, index: int, audio: bytes) -> None:
        async with self._lock:
            heapq.heappush(self._heap, (index, audio))
            self._event.set()

    async def drain(self, total: int):
        while self._next_expected < total:
            await self._event.wait()
            async with self._lock:
                while (
                    self._heap
                    and self._heap[0][0] == self._next_expected
                ):
                    _, audio = heapq.heappop(self._heap)
                    self._next_expected += 1
                    yield audio
                self._event.clear()

Completed audio is pushed onto a min-heap; only entries matching _next_expected are popped. Sentences that finish early sit in the buffer and are drained immediately once their turn arrives.

Prefetch for Proactive Loading

A prefetch strategy — keeping N chunks buffered ahead of the playback position — helps mitigate head-of-line blocking. If sentences 2 and 3 have already been synthesized and are waiting in the buffer while sentence 1 is playing, a slow-arriving sentence 2 won't interrupt playback.

PREFETCH_COUNT = 2   # synthesize 2 sentences ahead of the currently playing one

async def prefetch_pipeline(sentences, tts_adapter):
    dispatch_sem = asyncio.Semaphore(PREFETCH_COUNT + 1)
    buffer = OrderedAudioBuffer()
    total = len(sentences)

    async def _fetch(idx: int, sentence: str):
        audio = await tts_adapter.synthesize(sentence)
        await buffer.put(idx, audio)
        dispatch_sem.release()

    tasks = []
    for idx, sentence in enumerate(sentences):
        await dispatch_sem.acquire()
        tasks.append(asyncio.create_task(_fetch(idx, sentence)))

    async for audio in buffer.drain(total):
        yield audio

    await asyncio.gather(*tasks)

A higher PREFETCH_COUNT improves the chances of seamless playback but increases TTFA by the same amount. Tune it based on your service's requirements.

---

Frontend: Audio Queue Management

Once audio chunks arrive at the frontend over SSE or WebSocket, the frontend must play them back without gaps.

With the Web Audio API, each chunk must be decoded into an AudioBuffer and scheduled on the timeline:

const source = audioContext.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioContext.destination);
source.start(nextPlayTime);   // ← mismanaging this value causes gaps or overlap
nextPlayTime += audioBuffer.duration;

Poor nextPlayTime management causes two problems simultaneously:

• Gap: if the next chunk arrives late but nextPlayTime was calculated from a fixed offset, silence appears after the previous chunk finishes.
• Overlap: if a chunk arrives earlier than expected, the next audio starts while the previous one is still playing.

decodeAudioData is an async CPU operation. When chunks arrive in rapid succession, the decode queue backs up. Requesting PCM format (audio_format=pcm) skips this step entirely:

// Direct PCM handling — no decodeAudioData needed
const int16 = new Int16Array(rawBuffer);
const float32 = Float32Array.from(int16, v => v / 32768.0);
const audioBuffer = audioContext.createBuffer(1, float32.length, 24000);
audioBuffer.copyToChannel(float32, 0);

No WAV header parsing, no decoding overhead. In a streaming environment where low latency matters, PCM has a clear advantage over WAV.

---

Before and After

Test sentences: "어머!" / "그거 정말 신기하다!" / "나도 그런 거 본 적 있어." / "같이 가보자고!"
Environment: GTX 1070, float16, Geny internal benchmark

Adaptive num_step alone cuts serial streaming total time by roughly 36%. The live-split + prefetch combination delivers the largest overall gain, reducing total completion time by 55% compared to a single concatenated request.

---

Bottleneck Summary

Recommended Order of Adoption

These bottlenecks are interconnected. Increasing concurrency raises the likelihood of head-of-line blocking; introducing live-split increases parallel requests, which in turn surfaces the concurrency constraint again. The following order is the most practical approach:

Step 1  Refine adaptive num_step
        → A few lines of code, immediate effect, zero risk

Step 2  Introduce index-based ordering buffer
        → Required prerequisite before adding parallelism

Step 3  Set MAX_CONCURRENCY=2 (when VRAM headroom allows)
        → Configuration change; verify memory trade-offs

Step 4  Live-split pipelining
        → Increases coupling with the LLM component; introduce carefully

---

Closing Thoughts

The answer to "streaming is enabled — so why is it still slow?" comes down to this.

Diffusion TTS cannot stream at the chunk level by design. Sentence-level streaming does lower TTFA, but it does not automatically guarantee gap-free, seamless playback. Layer on top of that MAX_CONCURRENCY=1 serialization, the LLM→TTS bubble, and head-of-line blocking, and you have four independent layers of latency stacking simultaneously.

Fixing any single layer is not enough. Each layer needs to be addressed independently, and the full pipeline timeline must be measured empirically.

The next post will cover voice profile management and emotion control — design patterns for managing multiple character voices in production, and techniques for weaving emotional state naturally into TTS output.