Whisper Speech Recognition MCP Server: Lightning Fast & Scalable

Whisper Speech Recognition MCP Server

---

中文文档

A high-performance speech recognition MCP server based on Faster Whisper, providing efficient audio transcription capabilities.

Features

• Integrated with Faster Whisper for efficient speech recognition
• Batch processing acceleration for improved transcription speed
• Automatic CUDA acceleration (if available)
• Support for multiple model sizes (tiny to large-v3)
• Output formats include VTT subtitles, SRT, and JSON
• Support for batch transcription of audio files in a folder
• Model instance caching to avoid repeated loading
• Dynamic batch size adjustment based on GPU memory

Installation

Dependencies

• Python 3.10+
• faster-whisper>=0.9.0
• torch==2.6.0+cu126
• torchaudio==2.6.0+cu126
• mcp[cli]>=1.2.0

Installation Steps

1. Clone or download this repository
2. Create and activate a virtual environment (recommended)
3. Install dependencies:

pip install -r requirements.txt

PyTorch Installation Guide

Install the appropriate version of PyTorch based on your CUDA version:

• CUDA 12.6:

  pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126
  

• CUDA 12.1:

  pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
  

• CPU version:

  pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cpu
  

You can check your CUDA version with nvcc --version or nvidia-smi.

Usage

Starting the Server

On Windows, simply run start_server.bat.

On other platforms, run:

python whisper_server.py

Configuring Claude Desktop

1. Open the Claude Desktop configuration file:

* Windows: `%APPDATA%\Claude\claude_desktop_config.json`
* macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`

2. Add the Whisper server configuration:

{
  "mcpServers": {
    "whisper": {
      "command": "python",
      "args": ["D:/path/to/whisper_server.py"],
      "env": {}
    }
  }
}

3. Restart Claude Desktop

Available Tools

The server provides the following tools:

1. get_model_info - Get information about available Whisper models
2. transcribe - Transcribe a single audio file
3. batch_transcribe - Batch transcribe audio files in a folder

Performance Optimization Tips

• Using CUDA acceleration significantly improves transcription speed
• Batch processing mode is more efficient for large numbers of short audio files
• Batch size is automatically adjusted based on GPU memory size
• Using VAD (Voice Activity Detection) filtering improves accuracy for long audio
• Specifying the correct language can improve transcription quality

Local Testing Methods

1. Use MCP Inspector for quick testing:

mcp dev whisper_server.py

2. Use Claude Desktop for integration testing

3. Use command line direct invocation (requires mcp[cli]):

mcp run whisper_server.py

Error Handling

The server implements the following error handling mechanisms:

• Audio file existence check
• Model loading failure handling
• Transcription process exception catching
• GPU memory management
• Batch processing parameter adaptive adjustment

Project Structure

• whisper_server.py: Main server code
• model_manager.py: Whisper model loading and caching
• audio_processor.py: Audio file validation and preprocessing
• formatters.py: Output formatting (VTT, SRT, JSON)
• transcriber.py: Core transcription logic
• start_server.bat: Windows startup script

License

MIT

Acknowledgements

This project was developed with the assistance of these amazing AI tools and models:

• GitHub Copilot - AI pair programmer
• Trae - Agentic AI coding assistant
• Cline - AI-powered terminal
• DeepSeek - Advanced AI model
• Claude-3.7-Sonnet - Anthropic's powerful AI assistant
• Gemini-2.0-Flash - Google's multimodal AI model
• VS Code - Powerful code editor
• Whisper - OpenAI's speech recognition model
• Faster Whisper - Optimized Whisper implementation

Special thanks to these incredible tools and the teams behind them.