Are you using OpenAI's Whisper for speech recognition and finding the timestamps are out of sync?
@maxhbain2,1 тыс. подписчиков
78 290 просмотров
11 комментариев
Комментарии из оригинального поста в X (Twitter)

@maxhbain
🧵[2/n] @openAI’s Whisper shows impressive transcription performance, but often the corresponding timestamps are out of sync by several seconds.
10 отметок «Нравится»

@RainmakerTrade
Heightened volatility got you on edge? In my latest free Substack post, discover how a Hidden Markov Model (HMM) can help you navigate market corrections and safeguard your investments.
91 отметка «Нравится»

@maxhbain
🧵[3/n] However, phoneme-based models such as Wav2Vec2.0 produce much more accurate timestamps. WhisperX leverages these models using forced alignment on the whisper transcription to generate word-level timestamps.
17 отметок «Нравится»

@maxhbain
🧵[4/n] The result is word-level timestamp output. See more examples and try it yourself at
16 отметок «Нравится»

@maxhbain
🧵[5/n] Of course, it would be better if a single model did everything. One way would be teacher-student, where whisper is learning to output wav2vec's aligned timestamps. If @OpenAI open-sourced the training data and script, it would be cool to try this :)
10 отметок «Нравится»

@bewarberg
@philipvollet @OpenAI Awesome!
1 отметка «Нравится»

@guskamp
@OpenAI any way to get this working for a musician who struggles aligning vocals to videos ?
1 отметка «Нравится»

@maxhbain
@OpenAI do you mean aligning lyrics to the audio? You can feed the lyrics to the align function in the code, although aligning over such a long sequence could be tricky.
1 отметка «Нравится»

@hencubed
@OpenAI Big bro killing it 🤝
1 отметка «Нравится»

@bulgurbot
@OpenAI @memdotai mem it

@nebsh83
@OpenAI Thanks !
