Google just made an incredible AI video breakthrough with its latest diffusion model, Lumiere. 2024 is going to be a massive year for AI video, mark my words. Here's what separates Lumiere from other AI video models:

Before we get into demos, a quick bit on the model's key differences. Lumiere uses a "space-time" neural network to generate clips in a single pass, as opposed to other models that create distant keyframes. Essentially, this avoids the ‘choppiness’ seen in other generations!

FEATURE 1: Video Editing Lumiere can edit any specific portion of a video with a simple mask and prompt. It's always insane to see demos like this — the potential applications are endless.

FEATURE 2: Stylized Generation Using a single reference image, Lumiere can generate videos in the target style by utilizing fine-tuned text-to-image model weights. It can take a single image, learn its style, and then generate consistent videos from that style. This is wild.

FEATURE 3: Image and Text to Video Lumiere can not only generate videos from text prompts, but also directly from a still image! I can finally put my monkey NFT to use 😅

FEATURE 4: Cinemagraphs Lumiere can not only create videos from text prompts but can also edit and animate specific portions of still images.

If you want to read more, here's the full release by Google Research: I'll also be going more in-depth on Lumiere in my newsletter tomorrow. Subscribe to never miss a thing in AI:

That's it for today's news in the world of AI. I do these rundowns daily, so follow me @rowancheung for more. If you found this helpful, spare me a like/retweet to support my content 👇

Excited to be able to create full worlds! If not this year, I think in the next couple of years we might have something like text-to-movie. It’s crazy to see the speed of AI developments.

It’s incredible. We’ll see the beginnings of text to movie this year. People will be able to generate movies from their phones sooner than most think!

The quality of these videos is unreal. Plus the editing capabilities. Insane.