This is a real-time interactive application. The image is the interface. The interface is a live stream of pixels. Everything is simulated as you interact with any part of the screen. There’s no conversion step, there’s no loss. The entire frame becomes the interface.

what applications are you most excited about?

Agents using this + helping run faster experiments and learning

This is crazy and incredible. I really hope you are working on gestural controller input and possibly VR/stereo rendering. Moving things around with your hands would be the ultimate interface.

Reminds me of this

Please bring this demo to the Vision Pro. Will be so incredible to interact with dynamic objects and interfaces like this beyond a normal screen.

would be great to see how this works instead of the current runway app ui and how to extract and transfer all your generated assets from the video stream in decent quality for 3d printing, for example btw seems like you'd still need to prompt/speak somewhere outside the mp4 video streaming ui? in your opinion, does it really replace a generative, lightweight three.js 3d-rich scene, let's say? i see some cases for video streaming too - not praising any specific paradigm, just asking :)

How do you imagine programming these environments would work? There still needs to be rules and data behind things. Like a game needs to know what items are in your inventory.

This is f*ing awesome. Congrats man, my head is 🤯 thinking about the use cases.

Do you think temporal consistency is completely solvable?

Yes

How can you guarantee it without some kind of objective underlying geometry gt?

yeah super fucking cool and clearly the frontier of human computer interaction. Please let me use it myself though

@c_valenzuelab You guys need to get me on the Early Access list as I have Real World use-cases.

How does the video reach the end user? Is it streamed or is it locally rendered?

"No conversion step" is the line that stuck with me. That changes everything downstream.

I still had a screenshot to OCR to DOM step on a prototype last week. The OCR invented a button that was not there. I killed the conversion and drove the pixel buffer. The extra transform was the bug. I wrote that in pipeline.md so I do not add it back.

Very exciting! Are you able to prompt how the controls will work? For example could you hold Ctrl and then drag which would do something different than just drag?

Excited to see what creators build with this.

Es una barbaridad. Me dijiste hace un año que se creaba un nuevo medio, distinto de la tv o el cine. Cada día que pasa es más obvio. Ojalá pronto podamos contarlo en @ZoomNet_tve

Bret Victor lives on this.

pls i have dreams to make