Loading video...
Video Failed to Load
🌟Introducing🎻Violin — an Open-source Video Translation Skill. 📹Video is the dominant medium on the internet, yet most high-quality content (lecture, talk, podcast) is locked behind a single language, leaving global audiences behind. So we built Violin: a video skill that combines speech recognition, LLM translation, and speech synthesis into... show more
139,822 views • 5 months ago •via X (Twitter)
34 Comments

🤗We are grateful to @BarteldsMartijn , @ykwon_0407 , @federicobianchy , and @KaitlynZhou for their thoughtful feedback!! ❤️We also thank the open-source community behind Whisper, DeepSeek, Qwen, and Cartesia, whose contributions form the foundation upon which Violin is built. 😊special thanks go to @nutlope and @percyliang for generously providing videos and valuable feedback during the development of this project!

I think the better technology combination is STT+ translation+speech synthesis+video character mouth generation. Without the last one, the final generated video will actually be very strange.

good suggestion, we have considered this. one factor need to consider is that the mouth gen is more costly as it is video gen, so far our version is lightweight and very fast. but we will love to include this in our next version!

Absolutely brilliant! This tool can be a gamechanger for people learning new languages using content that they are already familiar with or emotionally connected to. If the UI can be tweaked a little to juxtapose the native language alongside the translated language, I'd totally pay for this like today.

Thanks for your good words! @vinayprabhu

Wow, I just gave this a try and it works incredibly well! The translations are spot on. Huge congratulations to the team! 👏🔥

I built something similar - translating a 3+ hour Chinese podcast to English. CN: EN:

lol it looks amazing!

Thanks keviin

We create interactive video lectures for medical students with multiple audio tracks and captions. We translate the text, normalise the length, adjust the tempo and mux the audio into the video. Hope this works better. Will try this out. Thank you!

wow sound amazing! can you please share me the link for your work?

Sound good. How many generate times in we can use ?

@4Dominam 😊Hi @4Dominam so far we offer 1 times due to rate limit, and we will consider further scale this number soon. btw it is very cheap if use api, just 0.0x usd per video of several minute length!

Are you doing any image understanding or just ASR?

Hi we have vlm for vision understanding for chat with the video.

This is great, well done. It could also as easily work local, and even pushed higher using Voxtral realtime, replacing the STT, LLM, and TTS, making it possible with as little 32GB unified memory on a MacBook Pro.

perfect... seems awesome.. exploring

I had previously developed a version of this project as well, but gpt-translate came out.

I like this because I can source it openly

太需要这个了。

So you do audio translation-> merge new audio to video ? Cos lip sync is still with old lang

@reachmeviz thanks for your nice suggestion! lip sync is relate to video gen and more costly. Yes, we plan to include the lip sync as well!

@KevinQHLin Hi, can it work with Vietnamese ?

Yes it can work for Vietnamese!

This the best video translator l have used with ease. Thank you

Thanks for your recognition!

Great! I also build a custom "Violin Studio", Can I copy a part in my project? 3Q

Yes for sure, thanks for your interests! and welcome contribution!

very useful, thank you , I'm using it now

Hey Kevin, Violin is pretty awesome... Would love to support you with pf github sponsorships if you’re okay with it @KevinQHLin

thank for your interests! will let you know if we need sponsor!

Terminal

如果能把这个功能/SKILL做成插件,或者上传视频链接,不需要上传原视频文件也能翻译的话就更棒了

Unfortunately the demo isn’t working network error
