TTS Inference Deep Dive — Part 1: Adopting OmniVoice

This post is the first entry in the TTS Inference Deep Dive series —
a series on the ins and outs of speech synthesis inference, grounded in real production experience.

Introduction

Geny is a multi-agent system with a TTS pipeline that lets a VTuber character read agent responses aloud in real time. GPT-SoVITS handled that role for a long time, but as the service grew, we ran into fundamental limitations.

Because the setup depended on an external Docker image (xxxxrt666/gpt-sovits), we had zero access to the internals. Wanted to improve timeout handling? Fix the per-emotion reference audio logic? No luck — the container was a black box. Being unable to debug a running service turned out to be a far bigger constraint than it sounds.

Discovering k2-fsa/OmniVoice was an opportunity to tackle that problem head-on. Zero-shot TTS supporting 600+ languages was compelling, but what mattered most was that we could vendor the source directly and take full control.

This post walks through the microservice architecture, core API, and generation parameters we designed while integrating OmniVoice into Geny.

---

What is OmniVoice?

OmniVoice is a zero-shot TTS model developed by k2-fsa (the open-source speech processing group that carries on Kaldi's lineage). Given a single reference audio clip, it can clone that voice and speak in 600+ languages. Internally it uses a diffusion-based architecture, and the checkpoint is downloaded from HuggingFace as k2-fsa/OmniVoice.

OmniVoice offers three generation modes:

The license is Apache-2.0, so commercial use is permitted.

---

Microservice Design in Geny

Vendoring Strategy: omnivoice_core + server

The first decision was how to bring OmniVoice in. Installing the package via pip install would reproduce the same black-box problem we had with GPT-SoVITS. The strategy we chose instead was source vendoring.

Inside Geny's omnivoice/ directory, two Python packages coexist:

omnivoice/
  omnivoice_core/   # point-in-time snapshot of k2-fsa/OmniVoice
  server/           # 100% Geny-owned FastAPI wrapper

omnivoice_core/ is a copy of the upstream inference code frozen at a specific commit. The only modification made there is rewriting import paths — omnivoice.utils.* → omnivoice_core.utils.* — to avoid collisions with any omnivoice package installed system-wide. This directory is treated as immutable; ad-hoc patches are off-limits. When we want to track upstream, we replace it wholesale following the procedure defined in upstream_sync.md.

server/ is entirely Geny-owned code. It houses the FastAPI app, the process-wide model holder, the concurrency semaphore, the voice profile scanner, the audio encoder, and so on. Whenever we add new functionality — metrics, authentication, batch endpoints — we touch only server/.

This separation enforces the principle: don't touch upstream code, but freely build on top of it.

Container Architecture

Here is an architecture diagram of the geny-omnivoice container's internals:

┌──────────────────────────────────────────────────────────────────────┐
│ geny-omnivoice container                                             │
│                                                                      │
│  uvicorn (server.main:app)                                           │
│   └── FastAPI(lifespan=...)                                          │
│        ├── lifespan()  ─────► server/engine.py :: load(settings)     │
│        │                       OmniVoice.from_pretrained(...)        │
│        │                       (≈ multi-GB checkpoint, GPU resident) │
│        │                                                             │
│        └── Routes (server/api.py)                                    │
│              GET  /health       → loading | ok                       │
│              GET  /voices       → voice profile list                 │
│              GET  /languages    → supported language list            │
│              POST /tts          → synthesize (single response)       │
│              POST /tts/stream   → synthesize (per-sentence NDJSON stream) │
│                                    │                                 │
│                                    ▼                                 │
│                           asyncio.Semaphore(MAX_CONCURRENCY)         │
│                                    │                                 │
│                                    ▼ run_in_executor                 │
│                           OmniVoice.generate(...)                    │
│                                                                      │
│  Volumes:                                                            │
│    /voices (ro) ◄── backend/static/voices  (Geny shared volume)      │
│    /models (rw) ◄── geny-omnivoice-models  (HF cache)               │
└──────────────────────────────────────────────────────────────────────┘
                                ▲
                                │ HTTP (httpx)
┌───────────────────────────────┴──────────────────────────────────────┐
│ geny-backend container                                               │
│   omnivoice_engine.py  ←── TTSService  ←── omnivoice_config.py       │
└──────────────────────────────────────────────────────────────────────┘

The model is loaded in FastAPI's lifespan hook (OmniVoice.from_pretrained(...)). This involves downloading a multi-GB checkpoint from HuggingFace and placing it resident in GPU memory, so the initial startup takes a noticeable amount of time. The /health endpoint transitioning from loading to ok is the signal that the model is ready.

Volume Mount Strategy

There was one important design choice in the volume setup: mount voice profiles from a shared volume, or copy them into the container?

We went with a shared read-only mount:

volumes:
  - ./backend/static/voices:/voices:ro   # same source as the Geny backend
  - geny-omnivoice-models:/models        # HuggingFace cache (persistent)

Mounting /voices read-only means the omnivoice container immediately sees any profiles the backend adds or modifies. A copy-based approach would have required explicit synchronization every time. /models is stored on a persistent read-write volume to avoid re-downloading the checkpoint on restart.

---

Core API Walkthrough

GET /health — Model Readiness Check

{
  "status": "ok",
  "model": "k2-fsa/OmniVoice",
  "device": "cuda:0",
  "dtype": "float16",
  "sampling_rate": 24000,
  "auto_asr": false,
  "max_concurrency": 1
}

status takes one of three values: loading, ok, or error. The backend adapter's health_check() reads this field and returns False if the value is not ok, at which point TTSService falls back to edge_tts. The relevant portion of the adapter:

async def health_check(self) -> bool:
    ...
    resp = await client.get(f"{api_url}/health")
    body = resp.json()
    # newer servers use `phase`; older servers use `status` (backwards-compatible)
    phase = body.get("phase")
    if phase is not None:
        return phase == "ok"
    return body.get("status") == "ok"

POST /tts — Single Synthesis Request

The core request structure:

{
  "text": "안녕하세요, 저는 파이몬입니다.",
  "mode": "clone",
  "ref_audio_path": "/voices/paimon_ko/ref_neutral.wav",
  "ref_text": "The actual transcript of the reference audio",
  "language": "ko",
  "speed": 1.0,
  "num_step": 16,
  "guidance_scale": 2.0,
  "denoise": true,
  "audio_format": "wav",
  "sample_rate": 24000
}

The response body is the raw audio bytes (binary), with metadata attached as headers:

Content-Type: audio/wav
X-OmniVoice-Sample-Rate: 24000
X-OmniVoice-Mode: clone

Errors are clearly distinguished by HTTP status code:

How the Three Modes Actually Behave

clone mode — the most commonly used mode. Pass ref_audio_path and ref_text and the server synthesizes in that voice. If ref_text is omitted and the server has OMNIVOICE_AUTO_ASR=true, Whisper transcribes the reference automatically. In Geny, the prompt_text field from a voice profile's profile.json is passed as ref_text.

design mode — describe the desired voice in natural language via the instruct field, e.g., "female, low pitch, british accent". An empty instruct returns a 400 error. Because a new voice is generated on every request, output is inherently inconsistent across calls.

auto mode — uses a random voice with no reference audio or description required. Useful when you just want to quickly verify that synthesis is working.

Audio Formats

The server supports four output formats:

Understanding Generation Parameters

Parameter tuning matters more than you might expect when running a TTS model in production. Let's walk through OmniVoice's key parameters one by one.

num_step — The Two Faces of Diffusion Steps

OmniVoice is a diffusion-based TTS system. num_step controls the number of outer-loop iterations in the diffusion process, and GPU time scales linearly with this value.

• Upstream default: 32 (best quality)
• Geny's choice: 16 (recommended for clone mode on Pascal-class GPUs)
• Fast demo: 8 (roughly 4× faster, with a slight quality drop)

One interesting observation: for short utterances, lower step counts produce almost indistinguishable quality. We exploited this with an Adaptive num_step strategy:

cfg_num_step = max(1, int(config.num_step))
text_len = len(request.text or "")

if text_len <= 30:
    adaptive_num_step = min(cfg_num_step, 12)   # 짧은 문장: 최대 12
elif text_len <= 80:
    adaptive_num_step = min(cfg_num_step, 16)   # 중간 문장: 최대 16
else:
    adaptive_num_step = cfg_num_step            # 긴 문장: 설정값 그대로

This dynamically caps the step count based on text length. Utterances of 30 characters or fewer sound perfectly natural at 12 steps — pushing it higher gains nothing in perceived quality while burning extra GPU time. This technique meaningfully improves RTF (Real-Time Factor) in environments like streaming TTS where short sentences arrive in rapid succession.

guidance_scale (CFG)

This is the Classifier-Free Guidance scale. Higher values impose stronger conditioning on the reference audio (or instruct signal). The default is 2.0. In a TTS context, the 2–4 range is typically appropriate. Going too high can actually produce unnatural pronunciation.

denoise

Whether to apply post-synthesis noise reduction. Default is true.

Troubleshooting tip: if the output audio is silent or corrupted, toggling denoise=false sometimes fixes it. Counterintuitive as it sounds, overly aggressive denoising can destroy the audio in certain cases.

speed vs duration_seconds

speed is a playback rate multiplier (default 1.0); duration_seconds pins the output audio to a fixed length in seconds. When duration_seconds is set to a non-zero value, it overrides speed and synthesizes to that exact duration.

The way the adapter handles both values is worth noting:

payload = {
    "speed": float(config.speed) * float(request.speed or 1.0),
    "duration": float(config.duration_seconds) if config.duration_seconds > 0 else None,
}

The final speed is computed by multiplying the config-level speed with the request-level speed. This lets you apply an operator-level adjustment (e.g., a global 1.1× rate) and a per-request adjustment (e.g., an excited-emotion 0.9× modifier) independently.

---

Voice Profile Compatibility Design

The biggest risk when migrating from GPT-SoVITS to OmniVoice was having to rebuild every existing voice profile from scratch. We avoided this by preserving the exact same directory layout that GPT-SoVITS used.

backend/static/voices/<profile_id>/
  profile.json
  ref_neutral.wav
  ref_happy.wav
  ref_sad.wav
  ...

Structure of profile.json:

{
  "display_name": "파이몬 (한국어)",
  "emotion_refs": {
    "neutral": {
      "file": "ref_neutral.wav",
      "prompt_text": "레퍼런스 오디오에 담긴 발화 텍스트"
    },
    "happy": {
      "file": "ref_happy.wav",
      "prompt_text": "기쁠 때 발화 텍스트"
    }
  }
}

emotion_refs[<emotion>].prompt_text is passed directly to OmniVoice as ref_text. As a result, registered voice profiles work with both engines — switching between them requires changing only the provider setting.

Emotion Ref Resolution Logic — Fallback Chain

The _resolve_emotion_ref() function implements a fairly careful fallback chain:

def _resolve_emotion_ref(profile, emotion):
    for candidate in (emotion, "neutral"):    # 1. 요청 emotion → neutral 순으로
        result = _try(candidate)
        if result: return result

    for emo in emotion_refs:                  # 2. emotion_refs의 첫 번째 항목
        result = _try(emo)
        if result: return result

    # 3. 파일명 컨벤션에 최후 의존 (ref_{emotion}.wav)
    return (f"/voices/{profile}/ref_{emotion}.wav", "", "")

Missing emotion references are handled gracefully — as a last resort, the function falls back to a filename convention and passes the path through. If the file doesn't actually exist, returning a 400 is delegated to the server.

---

Persistent HTTP Client Pool

One design decision in the backend adapter stands out: a module-level httpx client pool.

_clients: dict[tuple[str, float], httpx.AsyncClient] = {}

async def _get_client(api_url: str, read_timeout: float) -> httpx.AsyncClient:
    key = (api_url.rstrip("/"), float(read_timeout))
    client = _clients.get(key)
    if client is not None and not client.is_closed:
        return client
    # Double-checked locking으로 중복 생성 방지
    async with _clients_lock:
        ...
        client = httpx.AsyncClient(timeout=timeout, limits=limits)
        _clients[key] = client
        return client

Previously, a new httpx.AsyncClient was created on every request. In workloads with frequent short utterances, the TCP handshake and HTTP connection setup overhead was measurable. By caching and reusing clients keyed on URL + timeout, requests run over warm connections.

Connection pool settings:

limits = httpx.Limits(
    max_keepalive_connections=8,   # 따뜻한 커넥션 8개 유지
    max_connections=64,            # 최대 64개 (동시 발사 케이스 흡수)
    keepalive_expiry=60.0,
)

max_connections=64 is sized to absorb worst-case scenarios where GPU concurrency is 8 and the frontend fires ~16 sentences simultaneously in a single turn.

Why We Removed the Adapter-Level Lock

Notably, we removed the module-level asyncio.Lock that was present in the previous adapter (the GPT-SoVITS pattern). The reasoning might not be immediately obvious.

The OmniVoice server itself serializes GPU access with an asyncio.Semaphore(max_concurrency). Having a lock in the adapter on top of that means:

1. A second request waits behind the lock until the first request fully completes, blocking the HTTP request from even being sent.
2. This makes it impossible to pipeline into the upstream semaphore.

If the server already owns serialization, a client-side lock is pure redundancy — and it actively hurts performance. This is a concrete example of how "defensively adding a lock" can actually reduce throughput.

---

Environment Variables and Operational Settings

OMNIVOICE_DTYPE=float16 is the primary lever for reducing GPU VRAM usage. It uses roughly half the VRAM of float32 with negligible quality difference. This is the first setting to check when you hit a GPU OOM.

OMNIVOICE_MAX_CONCURRENCY=1 is the safe default for a single-GPU host. If you have VRAM headroom or a multi-GPU setup, bumping it to 2–4 increases parallel synthesis throughput. That said, this value is directly tied to the streaming bottleneck covered in part 2 — simply raising it isn't always beneficial.

---

Common Issues

Here are problems we encountered (or anticipated) while running OmniVoice in Geny.

/health stuck in loading state
The model is still downloading, or something went wrong during the download. Check progress with docker logs geny-omnivoice.

503 model_not_ready
Calling /tts before the model finishes loading. Restart the service or check GPU memory.

Silent or corrupted TTS output
Toggle denoise=false first. If that doesn't help, try increasing num_step to 32.

GPU OOM
Set OMNIVOICE_DTYPE=float16 and confirm OMNIVOICE_MAX_CONCURRENCY=1.

Timeouts
The first CUDA inference on Pascal-class GPUs (GTX 1070, etc.) can easily exceed 60 seconds due to compilation cache generation. Setting Config.timeout_seconds to 180 or higher resolves this. The adapter enforces a minimum as well:

# 첫 번째 CUDA 추론이 60초 기본값을 초과할 수 있으므로 최솟값 강제
timeout = max(float(config.timeout_seconds or 0.0), 180.0)

---

Wrapping Up

Part 1 covered the motivation for adopting OmniVoice and the microservice architecture we built in Geny. The key takeaways:

• Escaping GPT-SoVITS's black-box constraints → adopted a source vendoring strategy
• Dual-layer structure of omnivoice_core (upstream snapshot) + server (Geny-owned wrapper) gives us both control and the ability to track upstream
• Identical Voice Profile layout to GPT-SoVITS enables a zero-downtime migration design
• Adaptive num_step improves RTF for short utterances
• Removing the adapter-level lock allows pipelining into the server's semaphore

Part 2 will cover bottleneck analysis for TTS streaming inference. Starting from the question "why is streaming slower than a single request?", we'll dig into four structural bottlenecks and examine mitigation strategies for each.