WhisperKit Benchmarks are live on Hugging Face! Speech-to-text systems are hard to benchmark holistically given trade-offs across latency, memory, energy efficiency and accuracy. On-device testing makes it doubly challenging. Here is our first version built with Gradio 🧵

The primary goals of this benchmark are: - Accuracy verification with respect to reference implementation from @OpenAI across several key datasets - Performance reports across devices and models for the default WhisperKit config - Continuously publish the results of our regression testing for developers to use as the device support matrix

Multilingual Quality This tab shows the accuracy (WER) results across 77 languages found in the Common Voice 17.0 dataset. We show the results for: - Language Hinted: Suggest WhisperKit to interpret in a specific language to improve accuracy - Language Predicted: Let WhisperKit predict language before transcribing Language classification results are also summarized in an interactive confusion matrix. There is a separate tab for English Quality where we benchmark short-form (librispeech) and long-form (earnings22) quality.

Device Support This tab shows regression testing results for different combinations of device, model version and OS version: - ✅: Combination is tested and verified to clear minimum performance and accuracy bars - ⚠️: Combination is tested and currently does NOT clear the bar. Argmax will review and fix or permanently disable the combination - Not supported: Argmax permanently disabled this combination. This is the case with older device, older OS and larger (not compressed) model variants.

Performance This tab lets you - sort by performance AND quality metrics - filter by device, OS and performance metric ranges The data displayed on the dashboard is also public (Link below), so developers can build presets for WhisperKit to best fit each end-user device. Data link:

Performance (continued) Note that higher performance with WhisperKit is possible. However, the dashboard data represents using the recommended (default) configuration that best balances battery life, thermal sustainability, memory consumption and latency. For example, on M2 Ultra, WhisperKit can run the latest @OpenAl Large V3 Turbo model (v20240930/turbo in WhisperKit) as fast as 72x real-time with a GPU+ANE config. However, the default config (ANE only) is published as 42x real-time on the dashboard:

Gradio Custom Components We're also excited to release a PyPI package featuring Argmax's own custom Gradio components. It powers advanced filters in our WhisperKit benchmarks space and integrates seamlessly.

Acknowledgements @NagaSaiAbhinay for the early contributions that eventually became WhisperKit's regression testing pipeline:

Discord: WhisperKit: WhisperKit Benchmarks:

@huggingface @Gradio Great effort on creating a comprehensive benchmark for WhisperKit. I've struggled with evaluating on-device performance myself, so this is super valuable.