This new workflow feels like cheating. LTX just dropped a new feature called "Audio-to-Video." Most AI generates random clips that vaguely match the vibe. This is different. It listens to the details of your sound file. Here's how this new workflow works:

Upload Source: Drop in raw dialogue, music, or SFX. Analysis: The AI maps the rhythm and pacing. Generation: It builds visuals that hit every beat. Lip-Sync: It automatically syncs character mouths to the dialogue. Sound ON 🔊

Watch the demo attached. Look closely at the lip movements. Raw output:

Available today on @LTXStudio I want to thank the team at Lightricks for partnering with me and for sponsoring this post. #ad

I break down AI workflows that actually save time. Follow me @maxescu so you don't miss the next one.

In the main video you used some clips from original films and swapped out the Borat face and added new audio? A bit confusing how it’s working to get these results.

You input the audio and an image. And an optional prompt. That's all. No clips from the original films. The initial images were created from a single frame capture of the original videos, with the characters swapped for Borat. It's just the feature says: Audio to Video.

1:17-1:25 is well edited.

Matched with the original video 😛

I like the voice between the cuts, it’s not cut exactly when the clip changes and gives nice feel.

I love it, great workflow Alex

Thanks, Amira! It's a new paradigm in AI video creation and I love it!

If you enjoyed this follow me @maxescu to learn more about generative AI. Thanks!

Nice. Does it handle multiple characters in the same shot?

Yeap!

Thanks - also can you give it character reference images (to preserve identity) or does it only use a single image. Thanks again for answering.

Single image, here's the interface

This is awesome 👏🏻👏🏻👏🏻

You guys cooked with 🔥 Congrats on the release!

The sync is still wildly off for actual production, but the pace of change is still stunning.

If you use clean audio, without any background noise, it's almost perfect.

@jacquesBroquard There are dedicated lip sync tools, like that I used to clean up the small sync issues after the fact. It's really good at that. Even the best video models have minor lip sync issues so this is really helpful.

@jacquesBroquard I tried with sync, but it has a very hard time with side angles or fast motion; it breaks very fast. It shines for static dialogue.

Amazing

LTX comes with banger after banger these days!

Really cool! But do you have the ability to prompt it as well? because I believe there can be some ambiguity when it comes to which character should be talking or even what a specific sound is visually.

Here are the options, but you'd be surprised how smart it is when choosing which character talks.

Cool, will definitely play around with it

Have fun!

big question, how many credits is it to use?

From my calculations, it's about $0.08/sec I'm not 100% sure it's correct, but we can ask @LTXStudio

@LTXStudio Is it on their platform? The whole local set up is really complicated

@LTXStudio Yeah, here you go:

What´s the purpose of remixing existing movies like this? It´s a shitty collage. It´s useless. It´s illegal to use another person´s likeness. I hate you all!!! :) :)

And what happens when some of it isn't matching up like in the Shawshank clip? How do you fix it?

Try again!

Awesome! Is this multi-scene?

Nope, but you can do up to 20sec. And then you can do the editing yourself

In the future, every tv will come with a Borat filter.

This Borat takeover is hilariously on point 😭🔥 The lip-sync timing + how it actually reacts to the delivery (not just mouthing words) makes this feel like the first time audio-to-video doesn't look like a bad dub job. @maxescu nailed the demo — which movie scene should get the Borat treatment next? I'm voting for the "I am your father" moment or the Inception hallway fight 😂 #AudioToVideo #AIVideo

"This new workflow feels like cheating." Because it is.

@SachaBaronCohen out of interest, why does nobody sue these businesses using an actors likeness without any licence for their own financial gain?

Doesnt it work for multi actor dialogue?

Yeap

Alex where you use this platform?

@LTXStudio

@LTXStudio Thanks a Lot @maxescu and Amazing tool @LTXStudio

This is step in the right direction but it still screams AI Do you run this locally?

I meant to ask, do you work for/with LTX?

In open source model ?

Yo @cocktailpeanut @deepbeepmeep 👀

@cocktailpeanut @deepbeepmeep 👀