Building an AI VTuber Agent Part 6: Live2D

Related Series

• Building an AI VTuber Agent, Part 1: Overview
• Building an AI VTuber Agent, Part 2: LLM and Persona
• Building an AI VTuber Agent, Part 3: Memory
• Building an AI VTuber Agent, Part 4: STT
• Building an AI VTuber Agent, Part 5: TTS
• Building an AI VTuber Agent, Part 6: Live2D (current)
• Building an AI VTuber Agent, Part 7: Stream Integration
• Building an AI VTuber Agent, Part 8: Production

---

Introduction

In Part 5, Haru learned to speak. We built a voice with XTTS-v2, cut latency with a sentence-level pipeline, and finished off a TTS system that varies speed and pitch by emotion.

But the screen is empty.

If audio is coming out of speakers while nothing is on screen, that's a radio, not a VTuber. A VTuber's identity comes from being a visual presence. When Neuro-sama talks, her mouth moves. When she laughs, her eyes curve. When she's idle, she breathes. That's what makes her a character.

Part 6 puts Haru on screen. We'll render a Live2D avatar in the browser, animate the mouth in sync with speech, change facial expressions based on emotion, and keep the eyes blinking even during silence.

Topics covered:

1. Live2D Cubism Web SDK — WebGL-based model rendering setup
2. Lip sync — real-time mouth parameter control via TTS audio waveform analysis
3. Emotion → expression mapping — automatic Live2D Expression switching from Part 2's EmotionState
4. Automatic motion system — eye blinking, breathing, and gaze tracking
5. WebSocket synchronization — precise synchronization between audio playback timing and animation

---

Live2D Cubism Web SDK Overview

Live2D is a technology for animating 2D illustrations as if they were 3D. Models are distributed as .moc3 binary files, and the SDK renders them via WebGL.

The official SDK must be downloaded directly from the Live2D official site. It is not published on npm. The downloaded archive contains Core/ and Framework/ directories.

An alternative is the pixi-live2d-display library — a wrapper on top of PixiJS, installable via npm with a much simpler API. The tradeoff is that SDK version support lags behind and fine-grained parameter control is limited. Because this series requires direct manipulation of lip sync and expression parameters, we'll work with the official SDK.

Model File Structure

A Live2D model consists of several files.

assets/haru/
├── haru.model3.json         # Main config — defines all resource paths
├── haru.moc3                # Model binary (mesh, bones, deformer data)
├── haru.physics3.json       # Physics simulation (hair, ribbons, etc.)
├── haru.userdata3.json      # User-defined data such as hit areas
├── textures/
│   ├── texture_00.png       # Base texture
│   └── texture_01.png       # Additional texture (costume layer, etc.)
├── motions/
│   ├── idle_01.motion3.json # Idle motion 1
│   ├── idle_02.motion3.json # Idle motion 2
│   └── ...
└── expressions/
    ├── happy.exp3.json      # Happy expression
    ├── sad.exp3.json        # Sad expression
    ├── surprised.exp3.json  # Surprised expression
    └── ...

haru.model3.json is the entry point. Parse this file to obtain the paths to all other resources.

SDK Setup

live2d-sdk/
├── Core/
│   ├── live2dcubismcore.js           # Cubism Core (WebAssembly wrapper)
│   └── live2dcubismcore.d.ts
└── Framework/
    └── src/                          # TypeScript source
        ├── cubismframework.ts
        ├── model/
        ├── motion/
        ├── rendering/
        └── ...

Copy Framework/src/ into your project or symlink it. Load Core via a <script> tag or include it through your bundler.

// vite.config.ts — serve Core from public
export default {
  publicDir: 'public',
  // copy Core to public/live2dcubismcore.js
};

<!-- index.html -->
<script src="/live2dcubismcore.js"></script>

---

CubismApp — Initialization and WebGL Context

To use the SDK, you need to initialize the WebGL context and CubismFramework in order.

// src/frontend/live2d/cubism_app.ts
import { CubismFramework, Option, LogLevel } from '../framework/src/live2dcubismframework';
import { HaruModel } from './haru_model';

export class CubismApp {
  private _canvas:   HTMLCanvasElement;
  private _gl:       WebGLRenderingContext;
  private _model:    HaruModel | null = null;
  private _rafId:    number           = 0;

  constructor(canvas: HTMLCanvasElement) {
    this._canvas = canvas;

    const gl = canvas.getContext('webgl', {
      alpha:     true,   // transparent background
      stencil:   true,   // required for masking
      antialias: true,
    });
    if (!gl) throw new Error('Browser does not support WebGL.');
    this._gl = gl;
  }

  initialize(): boolean {
    // viewport setup
    this._gl.viewport(0, 0, this._canvas.width, this._canvas.height);
    this._gl.clearColor(0.0, 0.0, 0.0, 0.0);  // transparent

    // initialize CubismFramework
    const option = new Option();
    option.logFunction   = (msg: string) => console.log('[Live2D]', msg);
    option.loggingLevel  = LogLevel.LogLevel_Verbose;

    CubismFramework.startUp(option);
    CubismFramework.initialize();

    return true;
  }

  async loadModel(modelDir: string, modelFileName: string): Promise<void> {
    this._model = new HaruModel(this._gl);
    await this._model.load(modelDir, modelFileName);
  }

  startRenderLoop(): void {
    const loop = (timestamp: number) => {
      this._gl.clear(this._gl.COLOR_BUFFER_BIT);

      if (this._model) {
        this._model.update(timestamp);
        this._model.draw();
      }

      this._rafId = requestAnimationFrame(loop);
    };
    this._rafId = requestAnimationFrame(loop);
  }

  dispose(): void {
    cancelAnimationFrame(this._rafId);
    if (this._model) this._model.release();
    CubismFramework.dispose();
  }
}

alpha: true is important. Making the background transparent lets you place other UI elements (chat window, status display, etc.) behind Haru.

---

HaruModel — Model Loading and Parameter Control

Extend CubismUserModel to create a Haru-specific model class. This class is the center of all parameter control.

// src/frontend/live2d/haru_model.ts
import {
  CubismUserModel,
  CubismModelSettingJson,
  CubismIdManager,
  CubismMotionManager,
  CubismExpressionMotionManager,
  CubismPhysics,
} from '../framework/src/...'; // actual path depends on SDK structure

export class HaruModel extends CubismUserModel {
  private _gl:          WebGLRenderingContext;
  private _modelSetting: CubismModelSettingJson | null = null;
  private _modelDir:    string = '';

  // Parameter ID cache — avoids per-frame string lookups
  private _idParamMouthOpenY:  CubismId | null = null;
  private _idParamEyeLOpen:    CubismId | null = null;
  private _idParamEyeROpen:    CubismId | null = null;
  private _idParamBreath:      CubismId | null = null;
  private _idParamAngleX:      CubismId | null = null;
  private _idParamAngleY:      CubismId | null = null;
  private _idParamEyeBallX:    CubismId | null = null;
  private _idParamEyeBallY:    CubismId | null = null;

  // Control values
  private _mouthOpenY:    number = 0.0;
  private _breathPhase:   number = 0.0;
  private _blinkTimer:    number = 0.0;
  private _blinkValue:    number = 1.0;
  private _gazeX:         number = 0.0;
  private _gazeY:         number = 0.0;

  constructor(gl: WebGLRenderingContext) {
    super();
    this._gl = gl;
  }

  async load(modelDir: string, modelFileName: string): Promise<void> {
    this._modelDir = modelDir;

    // Load model3.json
    const settingJson = await this._fetchText(`${modelDir}/${modelFileName}`);
    this._modelSetting = new CubismModelSettingJson(
      new ArrayBuffer(0) // in practice, parse from ArrayBuffer
    );

    // Load moc3
    const mocPath  = this._modelSetting.getModelFileName();
    const mocBuffer = await this._fetchBuffer(`${modelDir}/${mocPath}`);
    this.loadModel(mocBuffer);

    // Load textures
    const textureCount = this._modelSetting.getTextureCount();
    for (let i = 0; i < textureCount; i++) {
      const texturePath = `${modelDir}/${this._modelSetting.getTextureFileName(i)}`;
      const texture     = await this._loadTexture(texturePath);
      this.getRenderer().bindTexture(i, texture);
    }

    // Physics simulation
    const physicsPath = this._modelSetting.getPhysicsFileName();
    if (physicsPath) {
      const physicsBuffer = await this._fetchBuffer(`${modelDir}/${physicsPath}`);
      this.loadPhysics(physicsBuffer, physicsBuffer.byteLength);
    }

    // Load expressions
    await this._loadExpressions();

    // Load motions
    await this._loadMotions();

    // Cache parameter IDs
    this._cacheParameterIds();

    // Start default idle motion
    this.startIdleMotion();
  }

  private _cacheParameterIds(): void {
    const idManager = CubismFramework.getIdManager();
    this._idParamMouthOpenY = idManager.getId('ParamMouthOpenY');
    this._idParamEyeLOpen   = idManager.getId('ParamEyeLOpen');
    this._idParamEyeROpen   = idManager.getId('ParamEyeROpen');
    this._idParamBreath     = idManager.getId('ParamBreath');
    this._idParamAngleX     = idManager.getId('ParamAngleX');
    this._idParamAngleY     = idManager.getId('ParamAngleY');
    this._idParamEyeBallX   = idManager.getId('ParamEyeBallX');
    this._idParamEyeBallY   = idManager.getId('ParamEyeBallY');
  }

  /** Inject mouth open value from outside (0.0 – 1.0) */
  setMouthOpenY(value: number): void {
    this._mouthOpenY = Math.max(0, Math.min(1, value));
  }

  /** Inject gaze direction (-1.0 – 1.0) */
  setGaze(x: number, y: number): void {
    this._gazeX = Math.max(-1, Math.min(1, x));
    this._gazeY = Math.max(-1, Math.min(1, y));
  }

  update(timestamp: number): void {
    const deltaTime = this._getDeltaTime(timestamp);

    // Motion update (Live2D internal motion system)
    this._model.saveParameters();
    this._motionManager.updateMotion(this._model, deltaTime);
    this._model.restoreParameters();

    // Breathing animation
    this._updateBreath(deltaTime);

    // Eye blinking
    this._updateBlink(deltaTime);

    // Gaze — apply mouse / configured values
    if (this._idParamAngleX) {
      this._model.addParameterValueById(this._idParamAngleX, this._gazeX * 30);
    }
    if (this._idParamAngleY) {
      this._model.addParameterValueById(this._idParamAngleY, this._gazeY * 30);
    }
    if (this._idParamEyeBallX) {
      this._model.addParameterValueById(this._idParamEyeBallX, this._gazeX);
    }
    if (this._idParamEyeBallY) {
      this._model.addParameterValueById(this._idParamEyeBallY, this._gazeY);
    }

    // Lip sync — apply externally injected value
    if (this._idParamMouthOpenY) {
      this._model.addParameterValueById(this._idParamMouthOpenY, this._mouthOpenY);
    }

    // Physics update
    this._physics?.evaluate(this._model, deltaTime);

    // Vertex update
    this._model.update();
  }

  draw(): void {
    const gl       = this._gl;
    const renderer = this.getRenderer();
    renderer.setRenderState(gl.getParameter(gl.FRAMEBUFFER_BINDING), [0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight]);
    renderer.drawModel();
  }

  // ... (resource cleanup, texture loading, and other helpers)
}

The saveParameters() / restoreParameters() pattern is critical. It saves the current parameter state before the motion system overwrites it, then after the motion update, layers our additional values — lip sync, gaze, and so on — on top.

---

Lip Sync — Making the Mouth Move

The principle behind lip sync is straightforward: compute the volume (RMS) of the audio waveform and map it to ParamMouthOpenY. Full viseme mapping is considerably more complex, but RMS-based lip sync alone produces a natural enough result in practice.

// src/frontend/audio/lipsync.ts

export class LipSyncAnalyzer {
  private _analyser:    AnalyserNode;
  private _dataArray:   Float32Array;
  private _smoothed:    number = 0.0;

  // Smoothing factors — higher values produce smoother but slower movement
  private readonly SMOOTH_UP   = 0.6;  // rise speed
  private readonly SMOOTH_DOWN = 0.85; // fall speed (slower)

  constructor(audioContext: AudioContext, source: AudioBufferSourceNode) {
    this._analyser         = audioContext.createAnalyser();
    this._analyser.fftSize = 2048;
    this._dataArray        = new Float32Array(this._analyser.fftSize);

    source.connect(this._analyser);
    this._analyser.connect(audioContext.destination);
  }

  /** Compute the mouth-open value for the current frame (0.0 – 1.0) */
  getMouthOpenValue(): number {
    this._analyser.getFloatTimeDomainData(this._dataArray);

    // RMS calculation
    let sumSquares = 0;
    for (const v of this._dataArray) {
      sumSquares += v * v;
    }
    const rms = Math.sqrt(sumSquares / this._dataArray.length);

    // Normalize to 0–1 (empirically tuned scaling)
    const normalized = Math.min(1.0, rms * 8.0);

    // Exponential smoothing — different factors for rising vs. falling
    const smooth = normalized > this._smoothed ? this.SMOOTH_UP : this.SMOOTH_DOWN;
    this._smoothed = this._smoothed * smooth + normalized * (1 - smooth);

    return this._smoothed;
  }
}

Using different smoothing factors for rising (SMOOTH_UP) and falling (SMOOTH_DOWN) is intentional. The mouth needs to open quickly so phonemes register visually, but it should close slowly to avoid a choppy, stuttering look.

Wiring Audio Chunk Playback to Lip Sync

In part 5 we received Base64 WAV chunks and played them back via AudioBufferSourceNode. That node needs to be connected to LipSyncAnalyzer.

// src/frontend/audio/player.ts (extended from part 5)
import { LipSyncAnalyzer } from './lipsync';
import { HaruModel } from '../live2d/haru_model';

export class AudioPlayer {
  private _context:    AudioContext;
  private _nextStart:  number          = 0;
  private _lipSync:    LipSyncAnalyzer | null = null;
  private _model:      HaruModel | null = null;
  private _rafId:      number          = 0;

  constructor() {
    this._context = new AudioContext();
  }

  setModel(model: HaruModel): void {
    this._model = model;
  }

  async playChunk(base64wav: string): Promise<void> {
    const bytes   = Uint8Array.from(atob(base64wav), c => c.charCodeAt(0));
    const decoded = await this._context.decodeAudioData(bytes.buffer);

    const source = this._context.createBufferSource();
    source.buffer = decoded;

    // Connect to LipSyncAnalyzer
    this._lipSync = new LipSyncAnalyzer(this._context, source);

    const now     = this._context.currentTime;
    const startAt = Math.max(now, this._nextStart);
    source.start(startAt);
    this._nextStart = startAt + decoded.duration;

    // Start the lip-sync loop when playback begins
    const delay = Math.max(0, (startAt - now) * 1000);
    setTimeout(() => this._startLipSyncLoop(), delay);

    source.onended = () => {
      this._stopLipSyncLoop();
    };
  }

  private _startLipSyncLoop(): void {
    const loop = () => {
      if (!this._lipSync || !this._model) return;
      const value = this._lipSync.getMouthOpenValue();
      this._model.setMouthOpenY(value);
      this._rafId = requestAnimationFrame(loop);
    };
    this._rafId = requestAnimationFrame(loop);
  }

  private _stopLipSyncLoop(): void {
    cancelAnimationFrame(this._rafId);
    // Close the mouth
    if (this._model) this._model.setMouthOpenY(0);
    this._lipSync = null;
  }

  stop(): void {
    this._stopLipSyncLoop();
    this._context.close();
    this._context   = new AudioContext();
    this._nextStart = 0;
  }
}

Routing AudioBufferSourceNode through an AnalyserNode to the destination lets playback and analysis happen simultaneously. No separate thread or Worker is needed — everything runs on the main thread.

---

Emotion → Expression Mapping

In part 2 we designed the LLM response to include [EMOTION:happy] tags. This is where those tags connect to Live2D Expressions.

An Expression file (.exp3.json) is essentially a set of parameter offsets. It moves parameters like ParamEyeLSmile and ParamMouthForm to specific values to produce a facial expression. Which parameters are available varies by model, so you need to open the actual .exp3.json files to check.

// src/frontend/live2d/emotion_mapper.ts
import { EmotionState } from '../../shared/emotion';

// Emotion → Expression filename mapping
// Verify actual filenames against the Expressions array in the model's model3.json
export const EMOTION_EXPRESSION_MAP: Record<EmotionState, string | null> = {
  [EmotionState.CALM]:      null,          // default expression (no Expression applied)
  [EmotionState.HAPPY]:     'happy',
  [EmotionState.CURIOUS]:   'curious',
  [EmotionState.EXCITED]:   'excited',
  [EmotionState.NERVOUS]:   'nervous',
  [EmotionState.PLAYFUL]:   'happy',       // reuse happy expression for playful
  [EmotionState.TIRED]:     'tired',
  [EmotionState.CONCERNED]: 'sad',
};

// Expression transition duration (ms)
export const EXPRESSION_FADE_DURATION_MS = 500;

// Add expression control methods to HaruModel
// src/frontend/live2d/haru_model.ts

private _expressions: Map<string, ACubismMotion> = new Map();

private async _loadExpressions(): Promise<void> {
  const count = this._modelSetting!.getExpressionCount();
  for (let i = 0; i < count; i++) {
    const name = this._modelSetting!.getExpressionName(i);
    const path = this._modelSetting!.getExpressionFileName(i);
    const buf  = await this._fetchBuffer(`${this._modelDir}/${path}`);
    const exp  = CubismExpressionMotion.create(buf, buf.byteLength);
    this._expressions.set(name, exp);
  }
}

/** Switch expression. Pass null to return to the default expression. */
setExpression(name: string | null): void {
  if (name === null) {
    this._expressionManager.stopAllMotions();
    return;
  }
  const exp = this._expressions.get(name);
  if (!exp) {
    console.warn(`[HaruModel] Expression not found: ${name}`);
    return;
  }
  this._expressionManager.startMotionPriority(exp, false, 2);
}

The flow for parsing emotion tags from WebSocket messages and switching expressions:

// src/frontend/voice_client.ts (WebSocket handler extended)
import { EMOTION_EXPRESSION_MAP } from './live2d/emotion_mapper';
import { EmotionState }           from '../shared/emotion';
import { HaruModel }              from './live2d/haru_model';

export class VoiceClient {
  private _ws:    WebSocket;
  private _model: HaruModel;
  private _audio: AudioPlayer;

  handleMessage(event: MessageEvent): void {
    const msg = JSON.parse(event.data);

    switch (msg.type) {
      case 'audio_chunk':
        // Same as part 5 — play audio
        this._audio.playChunk(msg.audio_b64);
        break;

      case 'emotion':
        // Emotion tag → switch expression
        const emotion     = msg.value as EmotionState;
        const expressionName = EMOTION_EXPRESSION_MAP[emotion] ?? null;
        this._model.setExpression(expressionName);
        break;

      case 'tts_end':
        // Speech ended → gradually return to default expression
        setTimeout(() => this._model.setExpression(null), 1000);
        break;

      case 'interrupted':
        this._audio.stop();
        this._model.setExpression(null);
        break;
    }
  }
}

The server also needs to know when to send emotion messages. The simplest approach is to detect [EMOTION:xxx] tags as the LLM response streams in and forward them to the client immediately.

# src/vtuber/api/voice.py — detecting emotion tags during LLM streaming
import re

EMOTION_TAG_PATTERN = re.compile(r'\[EMOTION:(\w+)\]')

async def stream_with_emotion(ws: WebSocket, token_stream, ...):
    accumulated = ''
    async for token in token_stream:
        accumulated += token
        match = EMOTION_TAG_PATTERN.search(accumulated)
        if match:
            emotion = match.group(1).lower()
            # Send to client immediately
            await ws.send_text(json.dumps({
                'type':  'emotion',
                'value': emotion,
            }, ensure_ascii=False))
            # Strip the tag before passing text to TTS
            accumulated = EMOTION_TAG_PATTERN.sub('', accumulated)

Automatic Motion System

When Haru isn't speaking, a frozen screen feels unnatural. Eyes need to blink, breathing needs to be visible, and the gaze should shift slightly — all of that makes the character feel alive.

Eye Blinking

// HaruModel._updateBlink() — called every frame
private _blinkTimer:       number = 0.0;
private _blinkValue:       number = 1.0;
private _blinkState:       'open' | 'closing' | 'closed' | 'opening' = 'open';
private _blinkIntervalSec: number = this._nextBlinkInterval();

private _nextBlinkInterval(): number {
  // Random interval between 2.5–6 seconds — too regular looks robotic
  return 2.5 + Math.random() * 3.5;
}

private _updateBlink(deltaTime: number): void {
  this._blinkTimer += deltaTime;

  const CLOSE_SPEED = 0.1;  // Time to fully close (seconds)
  const OPEN_SPEED  = 0.15; // Time to fully open

  switch (this._blinkState) {
    case 'open':
      if (this._blinkTimer >= this._blinkIntervalSec) {
        this._blinkTimer  = 0;
        this._blinkState  = 'closing';
      }
      this._blinkValue = 1.0;
      break;

    case 'closing':
      this._blinkValue = Math.max(0, this._blinkValue - deltaTime / CLOSE_SPEED);
      if (this._blinkValue <= 0) {
        this._blinkState = 'closed';
        this._blinkTimer = 0;
      }
      break;

    case 'closed':
      this._blinkValue = 0.0;
      if (this._blinkTimer >= 0.05) {  // Hold closed for 50ms
        this._blinkState = 'opening';
      }
      break;

    case 'opening':
      this._blinkValue = Math.min(1, this._blinkValue + deltaTime / OPEN_SPEED);
      if (this._blinkValue >= 1.0) {
        this._blinkState         = 'open';
        this._blinkTimer         = 0;
        this._blinkIntervalSec   = this._nextBlinkInterval();
      }
      break;
  }

  if (this._idParamEyeLOpen) {
    this._model.setParameterValueById(this._idParamEyeLOpen, this._blinkValue);
  }
  if (this._idParamEyeROpen) {
    this._model.setParameterValueById(this._idParamEyeROpen, this._blinkValue);
  }
}

Pay attention to the difference between setParameterValueById and addParameterValueById. set overwrites the value; add adds to the current value. Eye blinking must use set — the intent is to override whatever the motion system has set for the eye parameters.

Breathing

private _breathPhase: number = 0.0;

private _updateBreath(deltaTime: number): void {
  // One breath cycle: ~4 seconds
  const BREATH_CYCLE = 4.0;
  this._breathPhase = (this._breathPhase + deltaTime / BREATH_CYCLE) % 1.0;

  // Sine curve — range 0–1
  const breathValue = (Math.sin(this._breathPhase * Math.PI * 2) + 1.0) * 0.5;

  if (this._idParamBreath) {
    this._model.setParameterValueById(this._idParamBreath, breathValue);
  }
}

Mouse-Based Gaze Tracking

Haru's gaze follows the mouse cursor, giving viewers the feeling that she's looking at them.

// src/frontend/live2d/gaze_tracker.ts

export class GazeTracker {
  private _targetX:   number = 0.0;
  private _targetY:   number = 0.0;
  private _currentX:  number = 0.0;
  private _currentY:  number = 0.0;

  private readonly LERP = 0.05;  // Gaze follow speed (higher = faster)

  constructor(canvas: HTMLCanvasElement) {
    canvas.addEventListener('mousemove', (e: MouseEvent) => {
      const rect = canvas.getBoundingClientRect();
      // Normalize to -1 ~ 1
      this._targetX = ((e.clientX - rect.left) / rect.width  - 0.5) * 2;
      this._targetY = ((e.clientY - rect.top)  / rect.height - 0.5) * -2;
    });
  }

  /** Called every frame — returns current gaze values */
  update(): { x: number; y: number } {
    // Lerp for smooth tracking
    this._currentX += (this._targetX - this._currentX) * this.LERP;
    this._currentY += (this._targetY - this._currentY) * this.LERP;
    return { x: this._currentX, y: this._currentY };
  }

  /** Reset to center when mouse leaves the canvas */
  reset(): void {
    this._targetX = 0;
    this._targetY = 0;
  }
}

// CubismApp.startRenderLoop() — modified
const gazeTracker = new GazeTracker(this._canvas);

const loop = (timestamp: number) => {
  this._gl.clear(this._gl.COLOR_BUFFER_BIT);

  if (this._model) {
    const gaze = gazeTracker.update();
    this._model.setGaze(gaze.x, gaze.y);

    this._model.update(timestamp);
    this._model.draw();
  }

  this._rafId = requestAnimationFrame(loop);
};

---

WebSocket Synchronization — Audio and Animation

Audio and lip sync are handled within the same AudioContext, so they stay in sync automatically. The tricky part is timing emotion expression transitions.

There's a delay between when the server sends an emotion message and when the audio chunk carrying that emotion actually plays back. Audio chunks are queued and played in order, so switching expressions the moment an emotion tag is detected causes a sync mismatch.

The fix is to bundle each emotion message with its corresponding audio chunk using a shared index.

Server: Tying Emotion to Audio Chunks via Index

# src/vtuber/api/voice.py
async def stream_tts_to_client(ws: WebSocket, orchestrator, llm_stream, emotion):
    queue = OrderedAudioQueue()

    # Emotion messages are also sent with index-based coordination
    async def on_sentence(sentence: str, index: int) -> None:
        # Parse emotion tags from the sentence
        match   = EMOTION_TAG_PATTERN.search(sentence)
        emotion_val = match.group(1).lower() if match else None

        if emotion_val:
            # Send emotion_hint before the audio_chunk
            # The client applies it when it plays the corresponding chunk_index
            await ws.send_text(json.dumps({
                'type':        'emotion_hint',
                'chunk_index': index,
                'emotion':     emotion_val,
            }, ensure_ascii=False))

    # ... (rest of the TTS pipeline)

// src/frontend/voice_client.ts — index-based emotion application
private _emotionHints: Map<number, string> = new Map();

handleMessage(event: MessageEvent): void {
  const msg = JSON.parse(event.data);

  switch (msg.type) {
    case 'emotion_hint':
      // Store until the corresponding chunk index is played
      this._emotionHints.set(msg.chunk_index, msg.emotion);
      break;

    case 'audio_chunk':
      // If there's an emotion tied to this chunk, switch the expression first
      const emotion = this._emotionHints.get(msg.index);
      if (emotion) {
        const expName = EMOTION_EXPRESSION_MAP[emotion as EmotionState] ?? null;
        this._model.setExpression(expName);
        this._emotionHints.delete(msg.index);
      }
      this._audio.playChunk(msg.audio_b64);
      break;
  }
}

Perfect synchronization would require using AudioContext's scheduled playback time, but at the sentence level, switching expressions just before a chunk plays is good enough in practice. The human eye can't distinguish an audio-to-expression offset of 200–300 ms.

AnimationController — Full Integration

This is the integration class that ties all components together.

// src/frontend/animation_controller.ts
import { CubismApp }   from './live2d/cubism_app';
import { HaruModel }   from './live2d/haru_model';
import { GazeTracker } from './live2d/gaze_tracker';
import { AudioPlayer } from './audio/player';
import { VoiceClient } from './voice_client';
import { EmotionState } from '../shared/emotion';
import { EMOTION_EXPRESSION_MAP } from './live2d/emotion_mapper';

export class AnimationController {
  private _app:    CubismApp;
  private _model:  HaruModel | null = null;
  private _gaze:   GazeTracker | null = null;
  private _audio:  AudioPlayer;
  private _client: VoiceClient;

  constructor(canvas: HTMLCanvasElement, wsUrl: string) {
    this._app   = new CubismApp(canvas);
    this._audio = new AudioPlayer();
    this._client = new VoiceClient(wsUrl);

    // Route WebSocket events → AnimationController
    this._client.onAudioChunk  = (b64wav, index) => this._onAudioChunk(b64wav, index);
    this._client.onEmotion     = (emotion, idx)  => this._onEmotion(emotion, idx);
    this._client.onTtsEnd      = ()               => this._onTtsEnd();
    this._client.onInterrupted = ()               => this._onInterrupted();
  }

  async initialize(modelDir: string): Promise<void> {
    this._app.initialize();
    await this._app.loadModel(modelDir, 'haru.model3.json');

    this._model = this._app.getModel();
    this._audio.setModel(this._model);

    this._gaze = new GazeTracker(this._app.getCanvas());
    this._app.setGazeTracker(this._gaze);

    this._app.startRenderLoop();
    this._client.connect();
  }

  private _onAudioChunk(b64wav: string, index: number): void {
    this._audio.playChunk(b64wav);
  }

  private _onEmotion(emotion: string, chunkIndex: number): void {
    if (!this._model) return;
    const expName = EMOTION_EXPRESSION_MAP[emotion as EmotionState] ?? null;
    this._model.setExpression(expName);
  }

  private _onTtsEnd(): void {
    // Return to neutral expression after 1 second
    setTimeout(() => this._model?.setExpression(null), 1000);
  }

  private _onInterrupted(): void {
    this._audio.stop();
    this._model?.setExpression(null);
  }

  dispose(): void {
    this._client.disconnect();
    this._audio.stop();
    this._app.dispose();
  }
}

The entry point is straightforward.

// src/frontend/main.ts
import { AnimationController } from './animation_controller';

const canvas = document.getElementById('live2d-canvas') as HTMLCanvasElement;
canvas.width  = 800;
canvas.height = 600;

const controller = new AnimationController(
  canvas,
  'ws://localhost:8000/voice/ws'
);
controller.initialize('/assets/haru').catch(console.error);

// Clean up on page unload
window.addEventListener('beforeunload', () => controller.dispose());

---

HTML Layout

<!-- index.html -->
<!DOCTYPE html>
<html lang="ko">
<head>
  <meta charset="UTF-8" />
  <title>하루 — AI VTuber 비서</title>
  <script src="/live2dcubismcore.js"></script>
  <style>
    body          { margin: 0; background: #1a1a2e; display: flex; }
    #live2d-canvas { position: fixed; right: 0; bottom: 0; width: 480px; height: 640px; }
    #chat-panel   { flex: 1; padding: 20px; color: #eee; font-family: sans-serif; }
  </style>
</head>
<body>
  <div id="chat-panel">
    <!-- 채팅 UI -->
  </div>
  <canvas id="live2d-canvas"></canvas>
  <script type="module" src="/src/frontend/main.ts"></script>
</body>
</html>

Pinning the canvas to the bottom-right so it overlaps the chat panel is the standard VTuber stream layout. The alpha: true WebGL context keeps the canvas background transparent.

---

Performance Optimizations

Parameter ID Lookup Caching

This was handled earlier in _cacheParameterIds(). Calling CubismFramework.getIdManager().getId('ParamMouthOpenY') every frame triggers a string hash lookup hundreds of times per second — 60 fps × multiple parameters. Resolving each ID once at load time and storing it in a member variable is essential.

Rendering Resolution

// Handle device pixel ratio
const dpr = window.devicePixelRatio || 1;
canvas.width  = Math.floor(480 * dpr);
canvas.height = Math.floor(640 * dpr);
canvas.style.width  = '480px';
canvas.style.height = '640px';

Leaving the canvas at its CSS size on a Retina display produces a blurry result. Scale the actual pixel resolution by devicePixelRatio while keeping the CSS size in logical pixels.

AnalyserNode fftSize

For lip-sync purposes, fftSize: 256 is sufficient. fftSize: 2048 is for when you need frequency analysis. RMS calculation only requires time-domain data, so a lower fftSize is more efficient.

this._analyser.fftSize = 256;  // Only using time-domain RMS

---

Updated Project Structure

The complete structure through part 6.

vtuber-assistant/
├── src/
│   ├── vtuber/                    ← backend (Python/FastAPI)
│   │   ├── api/
│   │   │   └── voice.py           ← emotion hint message added
│   │   ├── character/             ← part 2
│   │   ├── llm/                   ← part 2
│   │   ├── memory/                ← part 3
│   │   ├── stt/                   ← part 4
│   │   └── tts/                   ← part 5
│   ├── frontend/                  ← frontend (TypeScript)
│   │   ├── live2d/                ← added in part 6
│   │   │   ├── cubism_app.ts      # WebGL + CubismFramework initialization
│   │   │   ├── haru_model.ts      # model loading + parameter control
│   │   │   ├── gaze_tracker.ts    # mouse gaze tracking
│   │   │   └── emotion_mapper.ts  # emotion → Expression mapping
│   │   ├── audio/
│   │   │   ├── player.ts          ← lip-sync integration added (part 5 extension)
│   │   │   ├── lipsync.ts         ← added in part 6
│   │   │   └── capture.ts         ← part 4
│   │   ├── animation_controller.ts ← added in part 6 (integration)
│   │   ├── voice_client.ts        ← emotion hint handling added
│   │   └── main.ts
│   └── shared/
│       └── emotion.ts             ← EmotionState (shared between frontend/backend)
├── assets/
│   └── haru/
│       ├── haru.model3.json
│       ├── haru.moc3
│       ├── haru.physics3.json
│       ├── textures/
│       ├── motions/
│       └── expressions/
├── public/
│   └── live2dcubismcore.js        ← Live2D Core (placed directly)
└── live2d-sdk/                    ← SDK source (downloaded manually)
    ├── Core/
    └── Framework/

---

Wrap-Up

Here's a summary of everything implemented in part 6:

1. CubismApp: WebGL context + CubismFramework initialization, render loop management
2. HaruModel: Model loading, parameter ID caching, parameter control interface
3. LipSyncAnalyzer: RMS calculation via the Web Audio API AnalyserNode, exponential smoothing for natural mouth movement
4. Eye blinking: State-machine-based, randomized intervals to eliminate robotic regularity
5. Breathing: Sine curve, 4-second cycle
6. GazeTracker: Mouse position → gaze parameters, smoothed with linear interpolation
7. Emotion expressions: Index-based emotion_hint synchronization for tight audio-expression timing
8. AnimationController: Full component integration, WebSocket event routing

Haru now has a body. Her mouth moves when she speaks, her expression changes when she's happy, and she breathes even in silence.

Looking back at the goal defined in part 1 — an AI VTuber assistant capable of voice conversation, with memory, personality, and a living presence on screen — we've substantively achieved it across these six parts.

Part 7 takes this to broadcast-ready quality: OBS integration, multi-viewer conversation, chat parsing, and an autonomous system that lets Haru run a stream unattended, 24 hours a day.

---

Coming Up Next

Part 7: Streaming Integration — Haru Goes Live

2 AM. No one touched anything. Haru's stream started on its own.

Topics covered:

• OBS WebSocket integration: Automating scene transitions and source control from the server
• Twitch/YouTube chat parsing: Reading and reacting to chat messages in real time
• Multi-viewer conversation management: A priority queue system for naturally handling multiple simultaneous viewers
• Autonomous broadcast scheduler: Unattended operation from stream start through segment transitions to sign-off
• Operational monitoring: SSE dashboard, error detection, and automatic restart