Mistral AI Releases Voxtral 3 Voice Agent SDK and Speech Models

What happened

Mistral AI has released the Voxtral 3 family of speech-to-text and text-to-speech models, along with a new voice agent SDK [1]. This system is designed to handle real-time, multi-step conversations by combining listening, reasoning, tool calling, and speaking into a single streaming pipeline [1].

Key technical details and models include:

• Voxtral Small Transcribe 3: A large speech-to-text model for high-quality transcription of recorded files. It includes features like diarization (identifying different speakers), context-biasing to improve accuracy with technical terms, and end-of-turn detection [1]. It supports over 50 languages, including 24 official EU languages and various Asian and Indic languages [1].
• Voxtral Mini Transcribe 3: A 4B parameter model used for offline transcription of recorded files. It is priced at $0.0035 per minute [1].
• Voxtral Mini Transcribe Realtime 3: A model specifically for real-time transcription, supporting sessions, turn detection, and adjustable streaming delays [2].
• Voxtral Mini TTS 3: A text-to-speech model that supports over 35 languages and 50 dialects. It can adjust emotion, pace, and emphasis based on the conversation [1].
• Pro Voice: An adaptation feature that can reproduce a speaker’s specific character, style, and emotional range using 30 minutes of audio [1].

Users can access these tools through Mistral’s EU-hosted API or by deploying them on their own infrastructure for data sovereignty [1]. The SDK allows agents to interact with business systems and retrieve information during live phone or web conversations [1]. For offline transcription, users can process recorded files to receive text with timestamps and speaker labels [2]. For real-time audio, users can stream audio via a WebSocket [2].

Who is affected

• Contact centers and service desks: Can use the technology for automated agents [1].
• Developers: Can build virtual assistants or hands-free copilots using the SDK [1].
• Organizations requiring data sovereignty: Can deploy the models on their own infrastructure to control audio processing and data retention [1].
• Users of multilingual systems: The models support over 50 languages, including English, French, and Spanish [1].