Building an AI VTuber Agent, Part 1: Overview

Related Series

• AI VTuber Agent Build Log Part 1: Overview (current)
• AI VTuber Agent Build Log Part 2: LLM and Persona
• AI VTuber Agent Build Log Part 3: Memory
• AI VTuber Agent Build Log Part 4: STT
• AI VTuber Agent Build Log Part 5: TTS
• AI VTuber Agent Build Log Part 6: Live2D
• AI VTuber Agent Build Log Part 7: Streaming Integration
• AI VTuber Agent Build Log Part 8: Production

---

Introduction

In 2021, something unusual quietly appeared on Twitch. Neuro-sama — created by UK developer Vedal987 — was an AI VTuber that chatted with viewers in real time, played rhythm games, and cracked its own jokes. The responses were rough and often off the wall, but that unpredictability was exactly what drew people in. As it improved over time, it amassed hundreds of thousands of followers and posed a sharp question to the AI community:

"Can an AI actually become a character?"

Years later, with the explosive growth of large language models (LLMs), the widespread availability of local inference engines, and mature voice and animation technology, the answer is tilting decisively toward "yes." Thousands of starred open-source projects now form a thriving ecosystem, and individual developers have everything they need to build their own AI VTuber.

This series is not a simple tutorial. It is an end-to-end journey through designing and building an AI assistant agent with a VTuber avatar — from analyzing the open-source ecosystem and implementing code to making architectural decisions and thinking through what it takes to ship this as a real service.

---

What Is a VTuber AI Assistant?

VTuber (Virtual YouTuber) originally referred to streamers and creators who perform as virtual avatars. The culture started with Kizuna AI in 2016 and has since grown into a global ecosystem of thousands of active VTubers. Neuro-sama's arrival in 2021 opened a new paradigm: an AI itself becoming the VTuber.

The AI VTuber assistant we are building is an organic fusion of the following components:

When all of these are integrated into a single pipeline, the result is not just a chatbot — it is an AI presence that feels alive. It does not merely generate text; it speaks with a voice, shows facial expressions, makes gestures, remembers past conversations, and responds emotionally to individual users.

---

Why Now — Three Major Technology Shifts

To understand why building this system is so much more feasible today than it was three years ago, you need to look at three major technology shifts.

1. Local LLMs Go Mainstream

In early 2023, the weights for Meta's LLaMA model leaked, sending shockwaves through the AI ecosystem. The subsequent arrival of inference engines like llama.cpp and Ollama made it possible to run 7B–13B parameter LLMs in real time on consumer GPUs — or even CPU-only hardware.

The implications for VTuber AI are clear:

• Unlimited conversation with no API costs
• Privacy — personal data never leaves the machine
• Full customization — fine-tuning for a specific character's persona
• Latency control — local inference with no network round-trip

The quality gap between local models and top-tier options like GPT-4o or Claude 3.5 Sonnet is real. But for maintaining a character persona and handling everyday conversation, current local models are more than adequate.

2. Voice Technology Becomes Accessible

OpenAI Whisper (2022) was a watershed moment for STT. Before it, high-quality speech recognition meant paying for Google or Azure APIs. Whisper delivered multilingual recognition for free, running locally. Optimized variants — faster-whisper, whisper.cpp, sherpa-onnx — followed, making real-time STT a practical reality.

The TTS side saw its own revolution. Models like VITS (2021), XTTS-v2 (2023), and GPT-SoVITS (2024) made it possible to reproduce a specific character's voice from only a small sample of audio. RVC (Retrieval-based Voice Conversion) took this further, enabling real-time conversion of any TTS output into a target voice.

3. Web-Based Animation Frameworks Mature

Live2D Cubism SDK and VRM have been staples of the VTuber industry for years. What changed is that WebGL/WebGPU-based implementations capable of running these formats directly in the browser have matured to the point where a VTuber avatar can be rendered without any separate client software.

PixiJS, Three.js, and the newer WebGPU API enable smooth 60fps animation in the browser, while Web Workers and WebAssembly offload CPU-intensive processing to the background.

---

Deep Dive into the Open-Source Ecosystem

The following is a detailed look at the most notable open-source projects in this space. The goal is not just to introduce them but to understand the architectural philosophy and technical choices behind each one.

1. Open-LLM-VTuber — The Most Complete Python Implementation

GitHub: https://github.com/Open-LLM-VTuber/Open-LLM-VTuber
⭐ 7,385 | Python | Started November 2023

Open-LLM-VTuber is the most mature and long-running Python implementation in this ecosystem. Its README states:

"The reason it's called Open-LLM-VTuber instead of Open-LLM-Companion is because the project's initial development goal was to use open-source solutions that can run offline on platforms other than Windows to recreate the closed-source AI VTuber Neuro-sama."

The explicit goal, then, is an open-source recreation of Neuro-sama.

Key features:

① Fully offline operation
Every component can run locally. Ollama-based LLM and sherpa-onnx-based STT/TTS are the default options; cloud APIs are optional. The whole system works without an internet connection.

② Hands-free voice interruption
This is the project's flagship feature. If the user starts talking while the AI is speaking, the AI stops its current utterance and responds to the new input. It works without headphones — echo cancellation prevents the TTS output from feeding back into the microphone.

③ Broad model support

LLM: Ollama, OpenAI, Gemini, Claude, Mistral, DeepSeek, vLLM, LM Studio...
ASR: sherpa-onnx, FunASR, Faster-Whisper, Whisper.cpp, Groq Whisper, Azure...
TTS: MeloTTS, GPTSoVITS, XTTS, Bark, CosyVoice, Edge TTS, Fish Audio, Azure...

④ Browser-based Live2D rendering
Live2D avatars run directly in the browser without a separate viewer application. A built-in system automatically changes avatar expressions based on emotional state.

⑤ Desktop pet mode
A "desktop pet" mode renders the avatar over the screen with a transparent background. It can be dragged anywhere on screen, and a click-through option keeps it from interfering with other work.

Currently rewriting as v2.0
The team is in the middle of a complete rewrite for v2.0, addressing the architectural limitations of v1 and redesigning for greater flexibility and extensibility. This series will continue tracking how v2.0 evolves.

---

2. moeru-ai/airi — The Most Ambitious TypeScript Ecosystem

GitHub: https://github.com/moeru-ai/airi
⭐ 38,946 | TypeScript | MIT License | Started December 2024

moeru-ai/airi is by far the most-starred project in this space (~39k stars). The fact that it achieved this starting in December 2024 reflects explosive community interest.

The project's tagline:

"Re-creating Neuro-sama, a soul container of AI waifu / virtual characters to bring them into our world."

The goal is the same as Open-LLM-VTuber's, but the approach is entirely different.

① Web-first architecture

airi was designed from day one around web technologies as the foundation: WebGPU, WebAudio, Web Workers, WebAssembly, and WebSocket. This means:

• Runs directly in the browser — no installation required
• PWA support — works on mobile
• Platform-agnostic — Windows, macOS, Linux, iOS, Android

The desktop app is built on Tauri, leveraging native NVIDIA CUDA and Apple Metal acceleration. "Web tech without performance compromise" is the core message.

② Gameplay capability

This is what sets airi apart from every other project in the space. It does not just hold conversations — it plays actual games.

• ✅ Minecraft (Mineflayer-based)
• ✅ Factorio (RCON API + custom mod)
• ✅ Kerbal Space Program (announced)
• 🔄 Helldivers 2 co-op (in development)

This is not mere screen sharing. The AI reads the game state, makes strategic decisions, and executes in-game actions. This is the direction from "assistant" toward "companion."

③ A sprawling monorepo ecosystem

airi is not a single project — it is a large monorepo ecosystem. Key sub-projects that have spun out of it:

unspeech deserves special attention. It is a proxy server that uses OpenAI's /audio/transcriptions and /audio/speech API interfaces as a standard, letting you swap in different STT/TTS backends behind a consistent interface. This is a design pattern worth referencing throughout this series.

④ Memory system in development

A memory system called Memory Alaya is currently under development, planned to support in-browser DuckDB WASM and PostgreSQL pgvector. Part 3 of this series, which covers memory systems, will analyze this design in depth.

---

3. handcrafted-persona-engine — An RVC-Focused C# Engine

GitHub: https://github.com/elevenyellow/handcrafted-persona-engine
⭐ 1,230 | C#

Built in C# rather than Python or TypeScript, this project specializes in RVC (Retrieval-based Voice Conversion) integration. It brings Live2D, LLM, ASR, TTS, and RVC together in a single engine optimized for Twitch streaming.

The choice of C# is intentional: Unity ecosystem integration, strong performance on Windows desktop, and low-latency audio processing via .NET are all real advantages.

RVC will be covered in depth in Part 5 (TTS). In short, RVC takes audio produced by any TTS system and converts it in real time to sound like a target character's voice. A few dozen minutes of the target character's voice data is enough to reproduce it convincingly.

---

4. Other Notable Projects

From the list of related projects in airi's README, a few that are technically interesting:

• kimjammer/Neuro: Recreated Neuro-sama in 7 days. Impressive balance of implementation speed and completeness.
• semperai/amica: Strong VRM and WebXR support, pointing toward 3D avatars and metaverse use cases.
• elizaOS/eliza: An excellent example of how to integrate AI agents across multiple platforms.
• ardha27/AI-Waifu-Vtuber: Strong Twitch API integration.

Overall System Architecture

Let's take a bird's-eye view of the system we'll build throughout this series. Now that we've analyzed the best open-source projects in the space, it's time to design our own from scratch.

Architecture Diagram

┌────────────────────────────────────────────────────────────────┐
│                       사용자 인터랙션 레이어                       │
│            마이크 입력  /  텍스트 채팅  /  감정 반응               │
└─────────────────────────┬──────────────────────────────────────┘
                          │ 오디오 스트림
┌─────────────────────────▼──────────────────────────────────────┐
│                       STT 파이프라인                              │
│         VAD (음성 감지) → Faster-Whisper → 텍스트 정규화          │
│              [인터럽트 신호 감지 병렬 처리]                        │
└─────────────────────────┬──────────────────────────────────────┘
                          │ 텍스트 + 컨텍스트
┌─────────────────────────▼──────────────────────────────────────┐
│                    AI 코어 (LLM + 메모리)                         │
│  ┌──────────────┐  ┌───────────────┐  ┌──────────────────────┐ │
│  │  단기 메모리  │  │   장기 메모리   │  │     페르소나 관리      │ │
│  │  (컨텍스트)  │  │  (벡터 DB/RAG) │  │  (캐릭터 상태 머신)  │ │
│  └──────────────┘  └───────────────┘  └──────────────────────┘ │
│              ↓ 통합된 프롬프트                                    │
│         LLM 추론 (Ollama / Claude API)                          │
│              ↓ 응답 텍스트 + 감정 메타데이터                       │
└──────────┬──────────────────────────┬──────────────────────────┘
           │ 텍스트                    │ 감정 메타데이터
┌──────────▼──────────┐    ┌──────────▼──────────────────────────┐
│    TTS 파이프라인    │    │         VTuber 애니메이션 엔진          │
│  텍스트 → XTTS/     │    │   Live2D / VRM + 립싱크 + 표정 제어    │
│  GPT-SoVITS → 오디오│    │   자동 모션 + 아이 트래킹              │
└──────────┬──────────┘    └──────────┬──────────────────────────┘
           │ PCM 오디오               │ 렌더링 명령
┌──────────▼──────────────────────────▼──────────────────────────┐
│                         출력 레이어                               │
│              오디오 출력 (스피커) + 화면 렌더링 (WebGL)            │
└────────────────────────────────────────────────────────────────┘

Data Flow and Latency Analysis

In a real-time conversation system, latency is what makes or breaks immersion. Here are the expected latency figures for each stage:

Mic → VAD detection:            ~50ms   (nearly instant)
VAD → Whisper STT:             ~200ms  (faster-whisper, short utterance)
STT complete → LLM first token: ~300ms  (local Ollama, time-to-first-token)
LLM streaming → TTS start:     ~100ms  (after chunk accumulation)
TTS → audio output:            ~200ms  (XTTS synthesis time)
────────────────────────────────────
End-to-end (first utterance):  ~850ms – 1.2s

About one second of end-to-end latency is a realistic target with current technology. Both Open-LLM-VTuber and airi hit this mark. Of course, the actual numbers depend on the model size and hardware you're running on.

Strategies to reduce latency:

1. LLM streaming: Start TTS as soon as the first sentence is complete — don't wait for the full response.
2. TTS chunk processing: Split output into sentences and stream them through the TTS pipeline one at a time.
3. Audio buffering: Pre-synthesize the next TTS chunk and hold it in a buffer.
4. VAD tuning: Optimize voice-detection thresholds to eliminate unnecessary wait time.

---

Core Design Principles

These are the design principles we identified across every open-source project we analyzed, plus a few we're adding ourselves.

Principle 1: Local-First

Not depending on cloud APIs is the default. APIs are optional backends. This achieves three things at once:

• Cost: Running an LLM API 24/7 adds up fast.
• Privacy: Conversation data never leaves the local machine.
• Customization: No restrictions on model fine-tuning or system prompts.

Principle 2: Modularity

Every component must be a swappable plugin. Replacing Whisper with sherpa-onnx for STT should not require changes anywhere else in the system. airi's unspeech project is a good example of this principle applied to the STT/TTS layer.

# Bad: depending directly on an implementation
import whisper
result = whisper.transcribe(audio)

# Good: depending on an interface
class STTBackend(ABC):
    @abstractmethod
    async def transcribe(self, audio: bytes) -> str: ...

class WhisperSTT(STTBackend): ...
class AzureSTT(STTBackend): ...
class SherpaSTT(STTBackend): ...

Principle 3: Real-time First

Latency is the top priority in every design decision. This often requires trading off accuracy or quality.

• STT: Fast response over perfect recognition.
• LLM: Reasonable quality with a fast first token over maximum quality.
• TTS: Low latency over the most natural-sounding output.

Principle 4: Interruptibility

This is what fundamentally separates this system from a simple chatbot. The user must be able to interrupt the AI at any point while it's speaking. Every layer of the system must therefore support immediate cancellation.

User speech detected → Stop current TTS immediately → Cancel current LLM inference → Start processing new input

This is tricky to implement correctly. It requires cancellation logic at multiple levels: async cancellation handling, audio buffer flushing, and early termination of LLM streaming. We'll implement this mechanism in detail in Part 7 (User Interaction).

Principle 5: Extensible Character System

Rather than a single hardcoded character, we manage character profiles as external files. It should be easy to add new characters and switch between them.

---

Technology Stack Decisions and Rationale

Here is the finalized tech stack for this series, along with the reasoning behind each choice.

Backend: Python + FastAPI

Choice: Python 3.11+ / FastAPI / WebSocket
Why:    The standard language of the AI/ML ecosystem.
        Every library — hugging face, torch, whisper — is Python-first.
Alt:    TypeScript (airi's choice) — better for web integration,
        but weaker ML library bindings.

LLM: Ollama + Claude API Hybrid

Choice: Ollama (local) + Anthropic Claude API (cloud fallback)
Why:    Run llama3, mistral, etc. locally via Ollama.
        Fall back to Claude API when higher quality is needed.
        Both interfaces speak OpenAI-compatible format, so switching is easy.
Models: Llama 3.1 8B (default) / Claude 3.5 Haiku (cloud)

STT: faster-whisper

Choice: faster-whisper (CTranslate2-based optimized Whisper)
Why:    4× faster inference than the original Whisper.
        Memory-efficient with INT8 quantization.
        Built-in VAD filter.
        Supports real-time streaming.
Alt:    whisper.cpp (C++ implementation, even lower latency),
        sherpa-onnx (ONNX runtime, very lightweight)

TTS: XTTS-v2 + GPT-SoVITS

Choice: XTTS-v2 (default) / GPT-SoVITS (character customization)
Why:    XTTS-v2 does zero-shot voice cloning from just a 3-second audio sample.
        Multilingual support (including Korean).
        GPT-SoVITS enables few-shot training for a more natural character voice.
Alt:    Edge TTS (fast and stable, but no customization),
        Fish Audio API (highest quality, but paid)

Avatar: Live2D Cubism Web SDK

Choice: Live2D Cubism Web SDK (PixiJS-based)
Why:    Optimized for Japanese animation style with 2D illustrations.
        Industry standard in VTubing. Rich, polished expression and motion support.
        Runs directly in the browser.
Alt:    VRM + Three.js (3D avatars with XR support)
        airi supports both formats.

Memory: Qdrant + SQLite

Choice: Qdrant (vector search) + SQLite (relational data)
Why:    Qdrant supports local mode with fast ANN search.
        SQLite is lightweight, file-based, and requires no separate server.
        Combining both implements short-term (SQLite) + long-term (Qdrant) memory.
Alt:    Chroma (simpler, but lower performance),
        pgvector (most powerful, but requires PostgreSQL)

Frontend: React + PixiJS

Choice: React 19 + PixiJS 8 + TypeScript
Why:    React handles UI; PixiJS handles Live2D rendering.
        WebSocket for real-time communication with the backend.
        Vite + Bun for a fast development environment.

---

Full Series Roadmap

This series consists of 8 parts, each covering one technical domain in depth.

Each part starts with the theoretical background for that domain, moves into analysis of open-source implementations, and ends with writing actual code. Code built in one part carries forward into the next.

---

What This Series Will Produce

By the end of the series, we'll have the following.

1. A fully working AI VTuber assistant

• Holds voice conversations, remembers names, and recalls previous exchanges
• Changes facial expressions based on emotion; mouth moves while speaking
• Stops instantly when the user interrupts and responds to the new input

2. A modular codebase

• Each component independently reusable
• New STT/TTS/LLM backends easy to plug in

3. Deep technical understanding

• Why each architectural choice was made
• How each component works internally
• The real trade-offs that come up in a production environment

4. Readiness to contribute to the open-source ecosystem

• Deep enough understanding to submit meaningful PRs to the projects we analyzed

---

The Question Neuro-sama Left Behind

Let's close with a brief reflection on Neuro-sama — the project that started all of this.

Neuro-sama, built by Vedal987, was never just a tech demo. Through it, he posed a fundamental philosophical question: can an AI exist as a genuine presence? What caused viewers to form emotional connections with Neuro-sama wasn't just that it was technically impressive — it was the sense of presence it conveyed.

Neuro-sama's source code is not public. Vedal987 has said almost nothing about how it was built. But the impact of what he created has materialized in dozens of open-source projects.

This series aims to distill the best of that open-source ecosystem into something anyone can use to build their own AI VTuber. Beyond the technical implementation, we'll also think about what kind of AI we want to create and what that existence means.

"If an AI could become a genuine presence, what should that look like?"

Let's write our own answer to that question in code.

---

Coming Up Next

Part 2: LLM and Persona — Giving the Character a Soul

Going far beyond "speak like a cute character" system prompts, we'll take a deep dive into how to design a true AI character with a complex personality, consistent memory, and emotional state.

Topics covered:

• Character design fundamentals: Structuring personality, speech style, values, and backstory
• System prompt engineering: Techniques for organizing long character prompts effectively
• Character state machine: Managing a persona that shifts dynamically with conversation context
• Local LLM vs. cloud API: A practical comparison from the perspective of character consistency
• Actual implementation: Building a character conversation server with Ollama + FastAPI