Shahir
@TslShahir • 1,210 subscribers
On-device ML engineer and Researcher: fine-tuning, ternary models, and Apple Neural Engine inference. Apple Platforms Research @C_engines
Videos
I turned Google Gemma EmbeddingGemma 2, an embedding model, into a speech-to-text model, and it turned out to be one of the fastest STT models you can run on device. On a base M3 MacBook Air it transcribes an hour of audio in about 13 seconds: 288× real time. The idea came in March with Gemini Embedding 2. It puts text, images, audio and video into one shared space, so a spoken word lands right next to the written word. If that's true, you should be able to read a transcript straight out of the embedding model. I couldnt tried it then, but last week EmbeddingGemma 2 came out as an open model, so I tried it.
Shahir274,697 views • 7 days ago
Can we get Google Gemma's Embedding Gemma 2 to speak? A couple of days ago, I posted about my experiment to get Embedding Gemma 2 to transcribe. Then internet did its internet thing :D and it reached over 238k views on X and over 85K views on LinkedIN, and got it on the front page of HN So, naturally, I wanted to test the reverse. Can we get EG2 to generate speech? Turns out that we can. At least part of it. We used EG2's semantic word embeddings to plan prosody and other details, then a small model generates the audio representation as EG2 would have heard that and a vocos vocoder reconstructs the final voice form And as it happened with STT, this turned out to be ultrafast. 1000 tokens in 3.8 seconds. 79.5x realtime on a base M3 air. Please check out the video. The quality is not close to the best in the industry, but this is the just the first iteration.
Shahir15,643 views • 5 days ago
No more content to load