Technical

Model Guide

Technical details about the IndexTTS2 model, its architecture, training, and usage for emotional speech synthesis.

Model Overview

The system uses IndexTTS2, an advanced text-to-speech model that combines large language model (LLM) architecture with audio codec technology. It supports controllable emotional speech generation through two distinct mechanisms: emotion vector injection and reference audio-based emotion transfer.

Model Architecture

Text Encoder

Converts input text into semantic token sequences using BPE tokenization (sentencepiece).

Speaker Encoder

Extracts speaker identity features from the reference audio clip (~5 seconds minimum).

GPT-style Autoregressive Decoder

Generates discrete acoustic tokens conditioned on text, speaker, and emotion embeddings.

DVAE (Discrete Variational Autoencoder)

Handles the discrete tokenization of audio for the autoregressive generation process.

BigVGAN Vocoder

Converts generated acoustic tokens back into high-quality waveform audio.

Speech Tokenization

The model uses a two-level tokenization approach:

Semantic Tokens

Capture linguistic content and meaning. Speaker-independent and low bitrate. Similar phonemes map to similar tokens regardless of speaker.

Acoustic Tokens

Capture acoustic details: pitch, timbre, rhythm, and emotional expressiveness. Higher bitrate, used for waveform reconstruction.

Emotion Control

Vector-Based Emotion Control

Uses an 8-dimensional one-hot embedding vector injected during inference. Each dimension corresponds to one emotion. The emo_alpha parameter controls intensity (0.0 = neutral, 1.0 = full emotion).

# Vector-based inference
tts_model.infer(
    audio_prompt="speaker.wav",
    text="This is exciting news!",
    emo_vector=[1, 0, 0, 0, 0, 0, 0, 0],  # happy
    emo_alpha=1.0,
    output_path="output.wav"
)

Reference-Based Emotion Control

Uses an actual audio sample as emotion reference. The model extracts emotional prosody, pitch patterns, and intensity from the reference and applies them to the generated speech while maintaining the target speaker identity.

# Reference-based inference
tts_model.infer(
    audio_prompt="speaker.wav",
    text="This is exciting news!",
    emo_audio_prompt="emotional_ref.wav",
    output_path="output.wav"
)

When to use which method?

Vector-based is faster and gives consistent results for known emotions. Reference-based captures subtle emotional nuances and supports emotion styles not in the predefined set.

Inference Modes

Mode	Latency	Use Case
Streaming	Low (first chunk fast)	Real-time playback, interactive demo
Batch (Vector)	Medium	Generating multiple files with same emotion
Batch (Reference)	Higher	Generating files with emotion transfer

Model Checkpoints

checkpoints/
├── config.yaml           # Model configuration
├── gpt.pth               # Autoregressive decoder weights
├── dvae.pth              # Discrete VAE weights
├── bigvgan_generator.pth # Vocoder generator
├── bigvgan_discriminator.pth
├── bpe.model             # BPE tokenizer
└── ...

Known Limitations

English language only in Phase 1
Reference audio should be at least 5 seconds for good speaker adaptation
Very short texts (<3 words) may produce less stable results
GPU is required for practical inference speed
Some emotion combinations may not be perfectly orthogonal