We present MotionStream — real-time, long-duration video generation that you can interactively control just by dragging your mouse. All videos here are raw, real-time screen captures without any post-processing. Model runs on a single H100 at 29 FPS and 0.4s latency.

It feels totally different when video models actually become real-time interactive. Drag the mouse, and the cup instantly moves with it, and the water follows. You’re not just watching a video anymore, you’re playing with it.

We achieve this by first fine-tuning a video diffusion model conditioned on point track motions. We then apply Self-Forcing style causal distillation to post-train it into a real-time interactive model. To reduce error accumulation of long video generation, we apply attention sink during the training and inference. For maximum speed in real-time scenario, we incorporate several additional design optimizations: - Efficient encoding of track conditions via simple sinusoidal encoding and input-channel concatenation - Local, sliding-window causal attention - Lightweight VAE decoder

Using point tracks as a unified motion representation allows us to control not only the object motion but also the camera, all in real time. We use monocular depth estimation to lift the input image into 3D and uniformly sample 3D points. Then, as the virtual camera moves, we project each 3D point into the new view to create 2D motion trajectories.

It supports real-time video editing. Given an input video stream, we first edit the initial frame. We can estimates point tracks from the incoming video in real time. MotionStream then generates the output on the fly, conditioned on the edited initial frame and tracked points.

Work led by Joonghyuk Shin (@jhshin2000), and done with wonderful collaborators @zhengqi_li @rzhang88 @junyanz89 Jaesik Park @elishechtman Arxiv: Check out our project page for more details!

is open source, github ?

This is phenomenal. @twominutepapers you've gotta see this if you haven't already

Does attention sink solve the memory issue in long videos? Like if the same object is completely out of frame then come back later does the model can still generate the correct object?

The model does not really have long-term memory because we are using local attention. Objects in the middle will not be remembered. The architecture design is more suitable for manipulating the same object/scene than world exploration.

Got it, thanks for the reply! Attention sink looks really interesting here so I just wonder if dynamic attention sink would work, like discussed in the future work. This work is really cool!

when to release🤨

One issue I can see straight away is the off screen aspects, like hands in this example, are not 'generated' That would be a bit of an issue.

Right I think this is mostly limited by tha capability of the base model - below is another failure case. But we do observe that better base model can fix it, and models are progressing fast!

Found a typo 👍

That's insanely cool!

Awesome ✨

29 FPS on a single H100 is efficient. What’s the resolution limit for real-time playback?

直接控制图像，这不就离用AI玩游戏越来越近了嘛

Could I generate an Avatar out of it that respondieren to Text, and add an lypsinc Model over it?

what the shit

Good stuff

Oh that's fancy!! Nice

lets gooo! this is awesome

very cool!

@TomLikesRobots A link to site would be really useful rather than reading multiple threads and unable to find but interesting reading

Any plan to open-source？

Unbelievable work

Borderline sci‑fi, can't wait to try it

Looks promising!

Congratulations! Great to see this work being published.

This takes interactivity to another level. Instant feedback with real-time control is a game changer for video generation.

这不就离用AI玩游戏越来越近了嘛

Excellent work!

C an you have image input too?

Yes it's already conditioned on the initial frame.

Animation will become a bespoke talent of the old world

Incredible work and awesome presentation!

Wow 🤯