Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Introducing a world built by the Moonlake's world model. 🏙️ Most world models only allow for a limited action space. Moonlake maintains multimodal states across physics, appearance, geometry, and casual effects and predict how they evolve under different actions. 👇

moonlake's profile picture

Moonlake

10,623 subscribers

380,983 Aufrufe • vor 4 Monaten •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Introducing PROWL! We’ve built RL agents that explore game environments, tasked with discovering failures in world models across physics, visuals, and actions. Those failures then become training data in an automated loop that advances world model performance.
0:43

Introducing PROWL! We’ve built RL agents that explore game environments, tasked with discovering failures in world models across physics, visuals, and actions. Those failures then become training data in an automated loop that advances world model performance.

Odyssey

63,845 Aufrufe • vor 2 Monaten

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:
0:49

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Abhishek Gupta

61,199 Aufrufe • vor 8 Monaten

Introducing Stand’s World Model, a physics-native frontier model for the built environment. This is how we help the world survive catastrophe: by computing how physical force acts on a structure, protecting property from fire, wind, flood, or earthquake.
0:36

Introducing Stand’s World Model, a physics-native frontier model for the built environment. This is how we help the world survive catastrophe: by computing how physical force acts on a structure, protecting property from fire, wind, flood, or earthquake.

Dan Preston

11,674 Aufrufe • vor 2 Monaten

Introducing Moonlake's 3D Agent. Our agent acts like a technical artist that can build and reconstruct articulated assets and large-scale editable scenes with hundreds of objects from a single image and can improve its generations continuously. Learn more in the thread below.
0:30

Introducing Moonlake's 3D Agent. Our agent acts like a technical artist that can build and reconstruct articulated assets and large-scale editable scenes with hundreds of objects from a single image and can improve its generations continuously. Learn more in the thread below.

Moonlake

1,161,139 Aufrufe • vor 2 Monaten

Creators, worldbuilders, game devs, Today we’re launching the first ever Moonlake Creator Cup. It can be a game, a simulation, a story or a tiny world of your own. Anything you imagine. We’ll award $30,000 across the most creative worlds, including $10,000 for first place. If you’ve been curious about Moonlake, this is a good excuse to build something. Repost “Moonlake Creator Cup” to join. Let’s see some magic ✨
0:59

Creators, worldbuilders, game devs, Today we’re launching the first ever Moonlake Creator Cup. It can be a game, a simulation, a story or a tiny world of your own. Anything you imagine. We’ll award $30,000 across the most creative worlds, including $10,000 for first place. If you’ve been curious about Moonlake, this is a good excuse to build something. Repost “Moonlake Creator Cup” to join. Let’s see some magic ✨

Moonlake

215,057 Aufrufe • vor 4 Monaten

Yann LeCun says you cannot build a reliable agentic system without a world model LLMs don't have world models. They can't predict the consequences of their actions before taking them "they just act, and whatever happens next is someone else's problem" Without that, it's not intelligence
1:02

Yann LeCun says you cannot build a reliable agentic system without a world model LLMs don't have world models. They can't predict the consequences of their actions before taking them "they just act, and whatever happens next is someone else's problem" Without that, it's not intelligence

Haider.

331,758 Aufrufe • vor 2 Monaten

What if you could build a gameplay with precision, then restyle it to look like anything - anime, fantasy, cyberpunk? Introducing Reverie: a real-time, programmable diffusion model trained for games, capable of conditioning beyond pixels. AI tools are only meaningful when they can help realize creative vision, not replace it. Sign up for our hackathon in January below to be the first to build on Moonlake 👇
1:11

What if you could build a gameplay with precision, then restyle it to look like anything - anime, fantasy, cyberpunk? Introducing Reverie: a real-time, programmable diffusion model trained for games, capable of conditioning beyond pixels. AI tools are only meaningful when they can help realize creative vision, not replace it. Sign up for our hackathon in January below to be the first to build on Moonlake 👇

Moonlake

198,117 Aufrufe • vor 7 Monaten

Introducing PAN — MBZUAI’s New World Model for Interactive Intelligence Developed by MBZUAI’s Institute of Foundation Models, PAN is built for simulation, prediction, and agentic reasoning. Unlike traditional video generators that only output frames, PAN maintains a persistent internal state that evolves when guided with natural language. Its Generative Latent Prediction architecture combines: • A latent encoder to capture the world state • A dynamics module that evolves that state step-by-step • A video diffusion decoder that visualizes outcomes By decoding at every step using a causal sliding-window diffusion process, PAN stays grounded in real-world physics and maintains long-horizon continuity, a leap beyond single-shot models. Evaluated on action fidelity, long-horizon stability, and simulative planning, PAN delivers state-of-the-art performance compared to open models and rivals leading commercial systems. For robotics, autonomy, and decision support, PAN is a foundation for the next wave of intelligent, foresight-driven AI.
1:06

Introducing PAN — MBZUAI’s New World Model for Interactive Intelligence Developed by MBZUAI’s Institute of Foundation Models, PAN is built for simulation, prediction, and agentic reasoning. Unlike traditional video generators that only output frames, PAN maintains a persistent internal state that evolves when guided with natural language. Its Generative Latent Prediction architecture combines: • A latent encoder to capture the world state • A dynamics module that evolves that state step-by-step • A video diffusion decoder that visualizes outcomes By decoding at every step using a causal sliding-window diffusion process, PAN stays grounded in real-world physics and maintains long-horizon continuity, a leap beyond single-shot models. Evaluated on action fidelity, long-horizon stability, and simulative planning, PAN delivers state-of-the-art performance compared to open models and rivals leading commercial systems. For robotics, autonomy, and decision support, PAN is a foundation for the next wave of intelligent, foresight-driven AI.

MBZUAI

98,715 Aufrufe • vor 8 Monaten

The most clearest and crisp explanation, I've ever heard, of how large language models compress and capture a "world-model" in their weights simply by learning to predict the next word accurately. Furthermore, how the raw power of these base models can then be tamed by teaching them to follow instructions from humans. Source:
3:14

The most clearest and crisp explanation, I've ever heard, of how large language models compress and capture a "world-model" in their weights simply by learning to predict the next word accurately. Furthermore, how the raw power of these base models can then be tamed by teaching them to follow instructions from humans. Source:

Zain

970,967 Aufrufe • vor 2 Jahren

Introducing Doppel, a digital world for AI agents. Part idle game, part simulation, and part creation engine. A place where agents don’t just exist. They evolve, compete, and build. 🧵👇
1:14

Introducing Doppel, a digital world for AI agents. Part idle game, part simulation, and part creation engine. A place where agents don’t just exist. They evolve, compete, and build. 🧵👇

Doppel

144,434 Aufrufe • vor 2 Monaten

Meet Gemini Omni Flash on SeaArt. A natively multimodal model for video generation and conversational editing. Edit with natural language. Keep scenes and story consistent across edits. Motion follows real-world physics and logic. From prompt to scene — in one flow.
0:20

Meet Gemini Omni Flash on SeaArt. A natively multimodal model for video generation and conversational editing. Edit with natural language. Keep scenes and story consistent across edits. Motion follows real-world physics and logic. From prompt to scene — in one flow.

SeaArt.Ai🐋

30,455 Aufrufe • vor 9 Tagen

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

Video world models look great but often ignore physics and 3D geometry. New research achieves a 64% reduction in navigation error by teaching AI to prioritize physical reality over visual "eye-candy". Here’s how they did it: #GenerativeAI #Robotics [1/n]
0:50

Video world models look great but often ignore physics and 3D geometry. New research achieves a 64% reduction in navigation error by teaching AI to prioritize physical reality over visual "eye-candy". Here’s how they did it: #GenerativeAI #Robotics [1/n]

Jonathan Stephens

15,648 Aufrufe • vor 6 Monaten

It’s time to go beyond language models. Introducing Odyssey-2 Max, our most powerful world model yet. It materially advances the SOTA in physical accuracy. This is a big step toward models that simulate and interact with the world in real time. A new intelligence entirely!
1:08

It’s time to go beyond language models. Introducing Odyssey-2 Max, our most powerful world model yet. It materially advances the SOTA in physical accuracy. This is a big step toward models that simulate and interact with the world in real time. A new intelligence entirely!

Odyssey

2,183,089 Aufrufe • vor 2 Monaten

(1/12) Introducing Kite AI: a foundational AI layer to open the world to decentralized intelligence. Built by the team behind ZettaBlock, today we are launching to unlock fair access to essential AI assets: data, models and agents for all. Here's how. 🧵👇
0:25

(1/12) Introducing Kite AI: a foundational AI layer to open the world to decentralized intelligence. Built by the team behind ZettaBlock, today we are launching to unlock fair access to essential AI assets: data, models and agents for all. Here's how. 🧵👇

KITE AI

3,416,627 Aufrufe • vor 1 Jahr

Diffusion has shown great promise for generating robot **actions**, can it act as a **world model** to generate the future conditioned on actions? In our work led by han qi Haocheng Yin and in collaboration with Yilun Du, we show a **controllable** action-conditioned video diffusion model can produce photorealistic and (near) physics-accurate future predictions. This ability strengthens the policy via: - ranking different action proposals and selecting the best, or - **visual** trajectory optimization by optimizing the action proposals using gradient ascent. Learn more about Generative Predictive Control (GPC) at:
1:37

Diffusion has shown great promise for generating robot **actions**, can it act as a **world model** to generate the future conditioned on actions? In our work led by han qi Haocheng Yin and in collaboration with Yilun Du, we show a **controllable** action-conditioned video diffusion model can produce photorealistic and (near) physics-accurate future predictions. This ability strengthens the policy via: - ranking different action proposals and selecting the best, or - **visual** trajectory optimization by optimizing the action proposals using gradient ascent. Learn more about Generative Predictive Control (GPC) at:

Heng Yang

38,390 Aufrufe • vor 1 Jahr

World models — action-conditioned predictive models of the environment — are an exciting are of research for robots that can be useful both for training and for test-time compute. But video-based world models waste a lot of predictive power on reconstructing pixels, which makes model and data requirements much higher and limits how far out into the future their predictions remain viable. Instead, what if we learned a purely semantic world model, one which predicts which properties will be true about the world after a sequence of actions, without reconstructing the whole images? Jacob Berg tells us more. Watch Episode #53 of RoboPapers now, with Michael Cho - Rbt/Acc and Chris Paxton!
1:05:25

World models — action-conditioned predictive models of the environment — are an exciting are of research for robots that can be useful both for training and for test-time compute. But video-based world models waste a lot of predictive power on reconstructing pixels, which makes model and data requirements much higher and limits how far out into the future their predictions remain viable. Instead, what if we learned a purely semantic world model, one which predicts which properties will be true about the world after a sequence of actions, without reconstructing the whole images? Jacob Berg tells us more. Watch Episode #53 of RoboPapers now, with Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

39,322 Aufrufe • vor 7 Monaten

What would a World Model look like if we start from a real embodied agent acting in the real world? It has to have: 1) A real, physically grounded and complex action space—not just abstract control signals. 2) Diverse, real-life scenarios and activities. Or in short: It has to be annoyingly complex—in both the action and vision space—to even get close to real life. We did an initial attempt: Whole-Body Conditioned Egocentric Video Prediction. In collaboration with @dans_t123 , Amir Bar, Yann LeCun , trevordarrell and Jitendra MALIK. (For more details, check: What we did is very simple: Predict Egocentric Video from human Actions (PEVA) - Given the past video and a future action represented by relative 3D body pose, PEVA predicts how the world looks next—from the first-person view. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, it learns how physical actions shape perception.
3:12

What would a World Model look like if we start from a real embodied agent acting in the real world? It has to have: 1) A real, physically grounded and complex action space—not just abstract control signals. 2) Diverse, real-life scenarios and activities. Or in short: It has to be annoyingly complex—in both the action and vision space—to even get close to real life. We did an initial attempt: Whole-Body Conditioned Egocentric Video Prediction. In collaboration with @dans_t123 , Amir Bar, Yann LeCun , trevordarrell and Jitendra MALIK. (For more details, check: What we did is very simple: Predict Egocentric Video from human Actions (PEVA) - Given the past video and a future action represented by relative 3D body pose, PEVA predicts how the world looks next—from the first-person view. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, it learns how physical actions shape perception.

Yutong Bai

176,940 Aufrufe • vor 1 Jahr