Loading video...
Video Failed to Load
Can you use EmbeddingGemma 2 for Speech-to-Text? This experiment uses EmbeddingGemma 2’s unified audio-text vector space to match speech against text via vector similarity. The result? Transcribing an hour of audio in about 13 seconds, locally!
82,812 views • 2 days ago •via X (Twitter)
19 Comments
We love seeing the community find creative new ways to push multimodal embeddings! Check out the demo and breakdown from @TslShahir:
Oh well, time to build a Granola clone lmaooo
Est ce que ca marche dans l’autre sens aussi
matching against text means you need the candidate text up front, so this is closer to keyword spotting than transcribing open speech, right
How about text to speech?
@jundotkim we need this in oMLX mate, setting the embedding Gemma 2 to embedding disabled the ability to transcribe. Can you add the option to pick STT with embedding for it?
Where's Gemma 5 series, we're waiting for it 😭
Transcribing 1 hour audio in seconds. That's interesting.
An hour of audio in about 13 seconds, locally, is a serious speed number. I want to see the error rate on messy real-world recordings next, because a clean audiobook read is the easiest audio there is.
The author's breakdown adds useful context: a trained reader over ~229k words, with ~5% reported word error rate. Speed is only half the comparison.
goodness, locally too
Matching by vector similarity skips decoding, which would explain the 13 seconds. Does it need the candidate text up front?
How much parameter is it? @grok
This could be useful for developers building offline voice tools, especially on devices with limited compute. I'd experiment with a local meeting recorder that identifies action items, a voice-controlled app with a fixed command vocabulary, or a searchable audio archive without sending recordings to the cloud.
matching speech against a fixed list of text works great right up until someone says a word that isn't on the list
Does it have audio diarization?
日本語でも精度出ますか？
@grok how about WER ?
How accurate is it with different accents?
![Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.](https://image.24vids.com/tw-1672048076469698561/ext_tw_video_thumb/1672047839378300928/pu/img/MuoOBsAyUd-HAI7D.jpg)
