Пропустить навигацию
Are you using OpenAI's Whisper for speech recognition and finding the timestamps are out of sync?
0:00 / 0:34

Are you using OpenAI's Whisper for speech recognition and finding the timestamps are out of sync?

Max Bain

@maxhbain2,1 тыс. подписчиков

Открыть в X
555 отметок «Нравится»СкачатьОригинальный пост

78 290 просмотров

Are you using OpenAI's Whisper for speech recognition and finding the timestamps are out of sync? Just dropped: WhisperX with word-level timestamp accuracy by force aligning whisper with wav2vec2.0 🧵 [1/n]

11 комментариев

Комментарии из оригинального поста в X (Twitter)

  • Фото профиля Max Bain

    @maxhbain

    🧵[2/n] @openAI’s Whisper shows impressive transcription performance, but often the corresponding timestamps are out of sync by several seconds.

    10 отметок «Нравится»

  • Фото профиля Rainmaker

    @RainmakerTrade

    Heightened volatility got you on edge? In my latest free Substack post, discover how a Hidden Markov Model (HMM) can help you navigate market corrections and safeguard your investments.

    91 отметка «Нравится»

  • Фото профиля Max Bain

    @maxhbain

    🧵[3/n] However, phoneme-based models such as Wav2Vec2.0 produce much more accurate timestamps. WhisperX leverages these models using forced alignment on the whisper transcription to generate word-level timestamps.

    17 отметок «Нравится»

  • Фото профиля Max Bain

    @maxhbain

    🧵[4/n] The result is word-level timestamp output. See more examples and try it yourself at

    16 отметок «Нравится»

  • Фото профиля Max Bain

    @maxhbain

    🧵[5/n] Of course, it would be better if a single model did everything. One way would be teacher-student, where whisper is learning to output wav2vec's aligned timestamps. If @OpenAI open-sourced the training data and script, it would be cool to try this :)

    10 отметок «Нравится»

  • Фото профиля Benjamin Warberg

    @bewarberg

    @philipvollet @OpenAI Awesome!

    1 отметка «Нравится»

  • Фото профиля august kamp

    @guskamp

    @OpenAI any way to get this working for a musician who struggles aligning vocals to videos ?

    1 отметка «Нравится»

  • Фото профиля Max Bain

    @maxhbain

    @OpenAI do you mean aligning lyrics to the audio? You can feed the lyrics to the align function in the code, although aligning over such a long sequence could be tricky.

    1 отметка «Нравится»

  • Фото профиля Hen³

    @hencubed

    @OpenAI Big bro killing it 🤝

    1 отметка «Нравится»

  • Фото профиля hot-pocket.usd

    @bulgurbot

    @OpenAI @memdotai mem it

  • Фото профиля neb

    @nebsh83

    @OpenAI Thanks !