Runway Aleph experiment: using one initial vid clip as a character 'video reference' to generate many more with consistent aesthetics/world Whole video is made from aleph generations based on one initial clip (prompt and clip in the comments)

initial clip was this one I had laying around prompt: "Extrapolate another scene from the same 60 minutes of this footage. Ensure the footage visually matches perfectly (color grading, film stock, visual qualities" (modifications/variations for different activities/settings)

why is this interesting? I mean you can use different images and references for the same thing right? except there's more implicit information about intention within a video clip. I've been wondering for a while if we'll ever get a 'character reference' style functionality that allows using a video of a character in motion, rather than an image. Because far more of the character's intended motion, personality etc. is contained within that footage than could be conveyed in a single or multiple images

I think to some extent in the longer run this may be something needed to more smoothly integrate characters into new scenes and match existing movies in immersive realism. With longer footage/movies having a reference for the personality/acting style associated with a particular character played by a person for instance.

Is that a Pika..shroom?! 😅 Very cute video and great tips! I will be trying this out later tonight. Thanks for sharing!

I'm....not actually sure what he is, he came out of a generation at one point and I've just kept re-using him. Think it'd be fun to see what other people can come up with based on this type of technique, this is essentially a trick that worked with References and also seems to apply to video with Aleph:

Beatiful! 👏 Do certain shots work better as the reference clip? Such as character centered and facing the camera.

trying a lot of variations on this now, the one thing that was an issue here was that where there wasn't visual material (hands, feet) it did its own extrapolation, which mostly worked fine but causes some inconsistency as to how they look in the different shots (there's fingers in the berry shot) makes sense to have the entire character framed within the shot so it's not having to fill in the blanks

Good point. We noticed the same when using Act-Two. If the hands are not in view, there will be some variations in the styling, which makes sense since the model is making its best guess.

yep, what this particular prompt is essentially doing is extrapolating another video based on the video it has, and it appears to be using whatever context/implicit info is available in the video. Gen-4 does very well at environment extrapolation and matching aesthetics so it seems to produce coherent/sensible results. I think there's probably going to be some variation in which footage work well for a task like this, but to me it's exciting that this even worked at all. Seems like it's a capability that will likely increase with future models.

Great use case of Aleph!

Interesting, I didn't know Runway allowed you to use a video clip as a reference for characters/environment. I know that was possible with images, but not video.

Awesome 💯

You’ve unlocked….. VIDEO REFERENCE. I think this calls for a beer.🍺

This came out great

This is so cool!

Cuántos clips usó?

Ignoring the offline experience is how you miss the real customer pain points...and the best solutions.

Cool!

What is the cost / second and WEN API?

I love this little guy 🥹

Awesome job! Thanks for sharing the prompt.

This really helps Ray🔥

Very sweet short and your insights into future use cases, and your examples, were really thought provoking. Thank you.

Incredible world !

good

Aww it’s just too cute! I can’t handle it 😭💖

This needs to be a full film. I love the characters you create.

good job

amazing

Actually insane

Hola, interesante video. Estoy iniciando en el autoestudio sobre videos con IA, y tu contenido me ha gustado. Saludos.

🤓

oh so it's over for actual creativity, innit