Emotional TTS
Technical

System Architecture

Technical overview of the Emotional TTS system components, data flow, and deployment architecture.

Architecture Overview

┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Client     │────▶│   FastAPI Server  │────▶│   IndexTTS2     │
│  (Browser/   │     │   (Uvicorn)       │     │   Model         │
│   API call)  │◀────│   Port 8000       │◀────│   (PyTorch)     │
└─────────────┘     └──────────────────┘     └─────────────────┘
                           │                        │
                    ┌──────┴──────┐          ┌──────┴──────┐
                    │   uploads/  │          │  Emotion    │
                    │   outputs/  │          │  Vectors    │
                    └─────────────┘          └─────────────┘

Core Components

FastAPI Application (tts_api.py)

Main entry point. Handles HTTP requests, file uploads, job management, and audio streaming. Runs on Uvicorn ASGI server.

IndexTTS2 Model (index_tts_main/)

The core TTS model. Based on transformer architecture with emotion-controllable generation. Supports both autoregressive and streaming inference.

Emotion Vector System

8-dimensional one-hot vectors for each supported emotion. Injected into the model during inference to control emotional expression.

Job Queue System

In-memory job tracking for batch processing. Jobs run in a ThreadPoolExecutor with configurable max workers.

File Management

Automatic cleanup of temporary upload and output files. Default TTL is 2 hours, configurable via FILE_TTL_SECONDS.

Emotion Control Methods

Vector-Based Control

Uses predefined emotion embedding vectors. Only a neutral speaker reference is needed. The emotion vector is injected into the model to generate speech with the target emotion.

EMOTION_VECTORS = {
    "happy":      [1,0,0,0,0,0,0,0],
    "angry":      [0,1,0,0,0,0,0,0],
    "sad":        [0,0,1,0,0,0,0,0],
    "afraid":     [0,0,0,1,0,0,0,0],
    "disgusted":  [0,0,0,0,1,0,0,0],
    "melancholic":[0,0,0,0,0,1,0,0],
    "surprised":  [0,0,0,0,0,0,1,0],
    "calm":       [0,0,0,0,0,0,0,1],
}

Reference-Based Control

Uses an actual audio sample as emotion reference. The model extracts emotional characteristics from the reference and applies them to the generated speech while maintaining the target speaker identity.

# Reference-based inference
tts_model.infer(
    speaker_audio_path,
    text="Hello world",
    emo_audio_prompt=emotion_ref_path,
    output_path="output.wav"
)

Request Lifecycle

1. Client sends POST /stream_tts_vec
   ├── speaker_file (WAV)
   ├── text (string)
   ├── emotion (string)
   └── emo_alpha (float)

2. Server validates inputs
   ├── Check file format (.wav, .mp3, etc.)
   ├── Validate emotion name
   └── Ensure model is loaded

3. Audio processing
   ├── Save uploaded file to uploads/
   ├── Pad audio to minimum 5 seconds
   └── Load emotion vector

4. Model inference (streaming)
   ├── Generate audio chunks
   ├── Stream WAV chunks to client
   └── Buffer chunks for final save

5. Post-processing
   ├── Concatenate all chunks
   ├── Save final WAV to outputs/
   └── Clean up temporary files

Directory Structure

Docker/
├── tts_api.py              # FastAPI application
├── Dockerfile              # Container build config
├── requirements.txt        # Python dependencies
├── uploads/                # Temporary uploaded files
├── outputs/                # Generated audio files
├── index_tts_main/         # IndexTTS2 model code
│   ├── indextts/
│   │   └── infer_v2.py     # Model inference engine
│   ├── checkpoints/        # Model weights & config
│   └── ...
└── testapi_v*.ipynb        # API test notebooks

Docker Architecture

FROM python:3.10-slim
# PyTorch with CUDA 12.1
RUN pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu121
# Application dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "tts_api:app", "--host", "0.0.0.0", "--port", "8000"]

Configuration

VariableDefaultDescription
FILE_TTL_SECONDS7200Time before auto-deleting output files (seconds)
MAX_WORKERS2Maximum concurrent inference threads
MAX_ITEMS_PER_JOB20Maximum items per batch job