I turned Google Gemma EmbeddingGemma 2, an embedding model, into a speech-to-text model, and it turned out to be one of the fastest STT models you can run on device. On a base M3 MacBook Air it transcribes an hour of audio in about 13 seconds: 288× real time.... The idea came in March with Gemini Embedding 2. It puts text, images, audio and video into one shared space, so a spoken word lands right next to the written word. If that's true, you should be able to read a transcript straight out of the embedding model. I couldnt tried it then, but last week EmbeddingGemma 2 came out as an open model, so I tried it.show more

How it works: the model stays untouched. A small reader on top takes each slice of audio, finds the nearest of ~229,000 words in the model's own text space, and strings them together. It trained in under two hours on 8×H100, on openly licensed speech. On the same base M3 Air, it runs about twice as fast as IBM's Granite Speech TurboCTC. The table has more Macs and models. Its word error rate is about 5% right now, against 2–3% for leading STT models. The next iteration is to train a better word reader to improve that. Here is a free Mac app you can download to test it. Here's google's post

You can download the free Mac app here to test it out -

@googlegemma Yo, what did I just saw! this is crazy fast and that too on m3 air

@googlegemma Test on yours and let me know. It should be almost twice as fast

@googlegemma Can imagine the word-error-rate is pretty high because for the unified embedding they will have used tuples of objects/nouns?

@googlegemma About 5% WER on LibriSpeech test-clean: 5.18%. On the harder test-it's about 11.5-11.8%.

@googlegemma Still very good! If you use e.g. SpaCy to extract the nouns and verbs, how does that compare in WER?

@googlegemma Embedding gemma 2 can already identify parts of the speech. I have tested it. on WER, it is about 5 on Librispeech easy set.

@googlegemma Amazing, simply amazing! Well done!

@googlegemma Thank you :D Long-time fan. So, super honored

@googlegemma is this sorcery open source? :D

@googlegemma Soon :D

@googlegemma Awesome, I could imagine we can do all sorts of information transformation if we have a set to choose from using embedding Gemma. Although I expect the WR to be pretty high here?

@googlegemma the WER is currently around 5 for librispeech. But, this is easily finetunable for any domain. I tried it for earnings call recordings. WER moved almost 1.5 hours with a few hours of audio. The reader is just an 87 m model so finetuning happens on the reader.

@explorations_iq @googlegemma "WER moved almost 1.5 hours" What were you actually trying to say here? The WER dropped to 1.5?

@explorations_iq @googlegemma I missed a with there :D WER moved 1.5% with finetuning on a few hours of audio for a specific domain.

@googlegemma Looks great, how good is the accuracy

@googlegemma About 5% WER on LibriSpeech test-clean: 5.18%. On the harder test-it's about 11.5-11.8%.

@googlegemma Resource usage?

Resource use, measured in the app: an hour-long file peaks at about 860-900 MB of memory on a base 8 GB M3 MacBook Air (about 740 MB on an M1 MacBook Pro), and a 3-minute clip at about 630 MB. The models are a 715 MB download at 6-bit. The reading runs on the Neural Engine, with the GPU doing the word lookup.

@googlegemma I mean voz by desert ant labs which is just a compressed version of Parakeet TDT 0.6B v3 is still faster, but this is cool.

@googlegemma has download and tried. please support simplified chinese. thanks a lot

@googlegemma This is huge

@googlegemma Does it produce word timings too? That would ne great.

@googlegemma Yes. It does

@googlegemma Hey, we are both on the frontpage today :)

@googlegemma Damn! I hadn’t known. Thanks for letting me know. Congrats to you too 😀

@googlegemma Fantastic!!

@googlegemma Cool! You plan to share code?

@googlegemma Yes. Soon

@googlegemma Cool! Have you done any contrastive loss training? I wonder if that could further reduce the error rate.

@googlegemma Thanks. Will try that

@googlegemma converting an embedding model into STT is a weird flex and I mean that as a compliment. 288x on a base Air — what's the WER at that speed? latency numbers without quality metrics are just vibes.

@googlegemma Around 5 on Librispeech simple. I have mentioned it. This was just the initial training attempt. The WER can come down a lot.

@googlegemma Hey this is pretty cool. Im assuming this work only for English? How small is the slice of audio ? How did you train it? could u please elaborate on it.

@googlegemma Amazing! These kinds of novel ideas are so fun to follow! Kudos

@ClementDelangue @googlegemma This is pretty incredible tbh. Im gonna bench it to see how it runs on mobile!

@googlegemma Love this ❤️ Such a simple but clever idea 🏆

@googlegemma 288× real-time on a base M3 Air, fully on-device? This is the kind of open-source AI innovation that makes local-first apps genuinely exciting.

@googlegemma nice trick! how does it do on accents?

@googlegemma the WER was a little high on my own speech. But, it can be finetuned with about 1 hour of audio. Takes about 6 minutes to finetune, and it drastically improves the quality

@googlegemma An hour of audio in 13 seconds on a base MacBook Air, from an open embedding model. This is what the edge looks like now: your voice, your notes, your meetings transcribed without ever leaving your machine. Great hack.

@googlegemma This is so clever, I am going to try this on linux now.

@ClementDelangue @googlegemma This is really cool

@googlegemma hey bro! open-source it! have some cooli ideas ...!!

@googlegemma @pHequals7 fyi

@googlegemma Based. Instafollow 👀

@googlegemma Do you check it on non-english language too? For example in Persian we still lack some good ASR and WER is high

@googlegemma I am planning on training it on multilingual datasets. It could work. As of now, it is only English

@googlegemma Ping me if you need any help on Persian and Arabic

@googlegemma Will do, Thank you

@googlegemma An hour in about 13 seconds is the part I would notice. Ran Falcon-ASR on TII’s free demo today, so I'm hoping this one ends up less than 5% WER on my voice. Definitely will give this a try.

@googlegemma The WER might be higher on your own voice becayse 5% is on synthetic voice. But, I will release the model and a method to train it soon. It can improve accuracy on your own voice with just 1 hour of recorded audio and transcript.

@googlegemma Okay, good to know and looking forward to it.

@ClementDelangue @googlegemma This is a smart idea! Congrats! The efficiency seems really amazing. Will give it a try too!

@googlegemma what about decision models which responds in microseconds. have a look

@googlegemma 288× real time on a base Air broke my brain. I tried the same embedding-to-STT trick last month and got garbage from quantizing too aggressively, your writeup just cost me my weekend.

@googlegemma whats the accuracy ? WER?

@googlegemma At the moment, it is around 5% on the librispeech easy set. On my own recordings it is around 10. But, it is easily finetunable since the reader is only 87m. So, around 1-2 hours of your recording can improve the accuracy a lot