We found a hack to make Whisper transcribe into *any* language (not just English)! ⚡️ This is an EXPERIMENTAL one-step feature: enter your audio as input, and get a transcription in another language. 🔥 Check it out here:

Quick caveat: It's a task that the model isn't explicitly trained on, so test it thoroughly before us 🤗 We'll run more benchmarks this week and share our findings soon! Help us test it better? :)

The audio snippet was taken from "Stupid People" comedy routine by Bill Engvall The video was generated via @mhollemans's brilliant Word-level timestamps space:

It could already transcribe into 96 languages. You meant translate.

Ha! Semantics - in this case we literally force it to transcribe into a target language. So, technically you can say it is translating! Whisper traditionally only translates from language “X” into English.

Cool work! You might also like the work of my colleagues which work on streaming ASR/translation

@altryne have you tried this?

Thats the way Whisper (not the translation api) works. If you specify a language and speak in another it translates to the specified language. Yes, it's undocumented, but what "hack" do you mean?

In my experience those results were quite suboptimal and didn’t quite result in usable transcriptions. With better decoding strategy those issues can be alleviated a bit. So the hack is more so of using contrastive search with Whisper to enable such use cases. Will run more tests soon!

When you selected portuguese and speak spanish the 2022 original versión do a very good translation, undocumented feature?

Yess! Quite possible - although the accuracy of the transcriptions still needs to be checked. I’ve found it to loose context if not used with the right generation strategy! 🤗