Загрузка видео...

Не удалось загрузить видео

На главную

ScenarioControl (ECCV 2026) enables end-to-end controllable generative simulation 🚗✨for autonomous driving, allowing users to generate diverse traffic scenarios with fine-grained actor control while preserving realistic scene dynamics. We have released the code, and we’re looking forward to seeing what the community builds with it! 🔗 Check out our project...

_FelixHeide_'s profile picture

Felix Heide

1,998 subscribers

56,417 просмотров • 3 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 1

Фото профиля Conie Lariego
Conie Lariego3 месяцев назад

bruh simulating traffic for AVs without cars acting like broken NPCs is a headache 💀 this actor control is massive. honestly seeing how we process edge-case data during my internship at Lifewood Data Technology shows exactly why this level of simulation is needed...

Похожие видео

ScenarioControl 🚗🛣️ - Scenario Generation from a single Dashcam Image 📸 or Text Prompt 💬!! Excited to introduce a new vision-language control mechanism for learned driving scenario generation. Given a single dashcam image or a scene prompt or an image, we generate a full scene layout 🧩, temporally consistent rollouts, including map 🗺️, agents 🚗, and ego video🛣️ ScenarioControl enables direct, fine-grained control over layout and traffic while preserving realism. It operates in a vectorized latent space with a new cross-global control mechanism to fuse vision-language inputs with scene structure while preserving realism. Interfaces seamlessly with generative video models! Project: Super fun project by Lili Gao, Yanbo Xu , William Koch, Samuele Ruffino, Luke Rowe , Behdad Chalaki, Dmitriy Rivkin, Julian Ost, Roger Girgis, Mario Bijelic.
0:52

ScenarioControl 🚗🛣️ - Scenario Generation from a single Dashcam Image 📸 or Text Prompt 💬!! Excited to introduce a new vision-language control mechanism for learned driving scenario generation. Given a single dashcam image or a scene prompt or an image, we generate a full scene layout 🧩, temporally consistent rollouts, including map 🗺️, agents 🚗, and ego video🛣️ ScenarioControl enables direct, fine-grained control over layout and traffic while preserving realism. It operates in a vectorized latent space with a new cross-global control mechanism to fuse vision-language inputs with scene structure while preserving realism. Interfaces seamlessly with generative video models! Project: Super fun project by Lili Gao, Yanbo Xu , William Koch, Samuele Ruffino, Luke Rowe , Behdad Chalaki, Dmitriy Rivkin, Julian Ost, Roger Girgis, Mario Bijelic.

Felix Heide

22,404 просмотров • 6 месяцев назад

NEWS: Waymo has introduced the Waymo World Model, "a frontier generative model built on Google DeepMind’s Genie 3 that sets a new bar for large-scale, hyper-realistic autonomous driving simulation." "By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. This combination of broad world knowledge, fine-grained controllability, and multi-modal realism enhances Waymo’s ability to safely scale our service across more places and new driving environments. In the following sections we showcase the Waymo World Model in action, featuring simulations of the Waymo Driver navigating diverse rare edge-case scenarios." More info below.
1:03

NEWS: Waymo has introduced the Waymo World Model, "a frontier generative model built on Google DeepMind’s Genie 3 that sets a new bar for large-scale, hyper-realistic autonomous driving simulation." "By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. This combination of broad world knowledge, fine-grained controllability, and multi-modal realism enhances Waymo’s ability to safely scale our service across more places and new driving environments. In the following sections we showcase the Waymo World Model in action, featuring simulations of the Waymo Driver navigating diverse rare edge-case scenarios." More info below.

Sawyer Merritt

133,382 просмотров • 8 месяцев назад

At the Paris Motor Show, XPENG and Tesla brought autonomous driving to real Paris streets as part of the official Autonomous Lab program — the first large-scale public experience of XPENG’s AI driving technology in Europe. I also tried Tesla FSD. Both FSD and NGP have made impressive progress, each with its strengths. At XPENG, we believe our larger models give NGP stronger generalization across diverse driving scenarios. Now, You can still sign up for the test ride and try it out. Competition drives progress. Users win. Let’s keep pushing the tech forward, together.
4:12

At the Paris Motor Show, XPENG and Tesla brought autonomous driving to real Paris streets as part of the official Autonomous Lab program — the first large-scale public experience of XPENG’s AI driving technology in Europe. I also tried Tesla FSD. Both FSD and NGP have made impressive progress, each with its strengths. At XPENG, we believe our larger models give NGP stronger generalization across diverse driving scenarios. Now, You can still sign up for the test ride and try it out. Competition drives progress. Users win. Let’s keep pushing the tech forward, together.

Xiaopeng He

1,321,680 просмотров • 4 дней назад

Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: Paper: Code: Data: Youtube:
2:17

Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: Paper: Code: Data: Youtube:

Gordon (Guocheng) Qian

39,496 просмотров • 7 дней назад

🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous control—combining active vision, high-res touch, and high-DoF robot hands SharpaWave — to complete complex, real-world tasks. Code is fully released; check out our: Homepage: Paper link: Github:
3:55

🤖 What if a humanoid robot could make a hamburger from raw ingredients—all the way to your plate? 🔥 Excited to announce ViTacFormer: our new pipeline for next-level dexterous manipulation with active vision + high-resolution touch. 🎯 For the first time ever, we demonstrate ~2.5 minutes of continuous, autonomous control—combining active vision, high-res touch, and high-DoF robot hands SharpaWave — to complete complex, real-world tasks. Code is fully released; check out our: Homepage: Paper link: Github:

Haoran Geng

96,210 просмотров • 1 год назад

MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual Storytelling via Multi-Layered Semantic-Aware Denoising paper page: Visual storytelling often uses nontypical aspect-ratio images like scroll paintings, comic strips, and panoramas to create an expressive and compelling narrative. While generative AI has achieved great success and shown the potential to reshape the creative industry, it remains a challenge to generate coherent and engaging content with arbitrary size and controllable style, concept, and layout, all of which are essential for visual storytelling. To overcome the shortcomings of previous methods including repetitive content, style inconsistency, and lack of controllability, we propose MagicScroll, a multi-layered, progressive diffusion-based image generation framework with a novel semantic-aware denoising process. The model enables fine-grained control over the generated image on object, scene, and background levels with text, image, and layout conditions. We also establish the first benchmark for nontypical aspect-ratio image generation for visual storytelling including mediums like paintings, comics, and cinematic panoramas, with customized metrics for systematic evaluation. Through comparative and ablation studies, MagicScroll showcases promising results in aligning with the narrative text, improving visual coherence, and engaging the audience. We plan to release the code and benchmark in the hope of a better collaboration between AI researchers and creative practitioners involving visual storytelling.
0:26

MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual Storytelling via Multi-Layered Semantic-Aware Denoising paper page: Visual storytelling often uses nontypical aspect-ratio images like scroll paintings, comic strips, and panoramas to create an expressive and compelling narrative. While generative AI has achieved great success and shown the potential to reshape the creative industry, it remains a challenge to generate coherent and engaging content with arbitrary size and controllable style, concept, and layout, all of which are essential for visual storytelling. To overcome the shortcomings of previous methods including repetitive content, style inconsistency, and lack of controllability, we propose MagicScroll, a multi-layered, progressive diffusion-based image generation framework with a novel semantic-aware denoising process. The model enables fine-grained control over the generated image on object, scene, and background levels with text, image, and layout conditions. We also establish the first benchmark for nontypical aspect-ratio image generation for visual storytelling including mediums like paintings, comics, and cinematic panoramas, with customized metrics for systematic evaluation. Through comparative and ablation studies, MagicScroll showcases promising results in aligning with the narrative text, improving visual coherence, and engaging the audience. We plan to release the code and benchmark in the hope of a better collaboration between AI researchers and creative practitioners involving visual storytelling.

AK

22,379 просмотров • 2 лет назад

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.
6:34

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

Siyuan Huang

11,451 просмотров • 2 лет назад

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.
0:21

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,654 просмотров • 2 лет назад

Some of our air traffic control towers use radars from the 1960s and the operating system we use dates back to 1995 and 2000. We’re dealing with floppy discs and paper strips! The mess we inherited is insane. Technology has far exceeded what we have so U.S. Department of Transportation is upgrading our air traffic control with the most advanced system. Check out all the details in my interview with Miranda Devine on Pod Force One! ⬇️
2:19

Some of our air traffic control towers use radars from the 1960s and the operating system we use dates back to 1995 and 2000. We’re dealing with floppy discs and paper strips! The mess we inherited is insane. Technology has far exceeded what we have so U.S. Department of Transportation is upgrading our air traffic control with the most advanced system. Check out all the details in my interview with Miranda Devine on Pod Force One! ⬇️

Secretary Sean Duffy

223,391 просмотров • 1 год назад

We now run paid ads end-to-end from the Claude Code terminal at our $7M ARR agency. Our head of growth, Ivan, walks through the complete build on camera. > Builds the ICP matrix, sources target companies and contacts, enriches their emails via Prospero > Extracts brand specs from the landing page and assets, matches them against our proven ad template library > Fans out across parallel agents to generate creative variations > Renders the images and auto-uploads the audiences plus ads into Meta and LinkedIn campaigns We ran it live on Lovable just to see what would come out. Watch the entire thing:
1:07:22

We now run paid ads end-to-end from the Claude Code terminal at our $7M ARR agency. Our head of growth, Ivan, walks through the complete build on camera. > Builds the ICP matrix, sources target companies and contacts, enriches their emails via Prospero > Extracts brand specs from the landing page and assets, matches them against our proven ad template library > Fans out across parallel agents to generate creative variations > Renders the images and auto-uploads the audiences plus ads into Meta and LinkedIn campaigns We ran it live on Lovable just to see what would come out. Watch the entire thing:

Michel Lieben

17,477 просмотров • 6 месяцев назад

Today we're thrilled to announce the first end-to-end platform for Reinforcement Fine-Tuning. With just a dozen labeled data points, you can outperform #OpenAI o1 and #DeepSeekR1 on complex tasks. Built on the #GRPO methodology that DeepSeek-R1 popularized, our platform delivers exceptional results. In our real-world PyTorch to Triton transpilation case study, we achieved 3x higher accuracy than OpenAI o1 and DeepSeek-R1 when writing GPU code. Check out the thread below to learn how you can adapt an #opensource #LLM to your use cases with unmatched efficiency. #rft
2:55

Today we're thrilled to announce the first end-to-end platform for Reinforcement Fine-Tuning. With just a dozen labeled data points, you can outperform #OpenAI o1 and #DeepSeekR1 on complex tasks. Built on the #GRPO methodology that DeepSeek-R1 popularized, our platform delivers exceptional results. In our real-world PyTorch to Triton transpilation case study, we achieved 3x higher accuracy than OpenAI o1 and DeepSeek-R1 when writing GPU code. Check out the thread below to learn how you can adapt an #opensource #LLM to your use cases with unmatched efficiency. #rft

Predibase by Rubrik

1,022,546 просмотров • 1 год назад

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:
3:30

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

596,424 просмотров • 3 месяцев назад

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:
3:30

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

113,821 просмотров • 12 дней назад

It turns out code… is the best way… to make videos… ⁉️ So, to take it a step further: we’re excited to introduce Video Arena - watch AIs compete to code the video of your dreams. (We should really be calling it Code 2 Video Arena! 💀) Now live at For $0, you can prompt an idea and then, choose the video which best captures your taste! Build UI walkthroughs, data explainers, launch videos etc.. The arena is currently live with our harness partners HeyGen, Remotion, Hashtagart Mosaic; and is supported by the OpenAI and Paul Jankura models. Please try it out! Looking forward to what y’all build!
0:50

It turns out code… is the best way… to make videos… ⁉️ So, to take it a step further: we’re excited to introduce Video Arena - watch AIs compete to code the video of your dreams. (We should really be calling it Code 2 Video Arena! 💀) Now live at For $0, you can prompt an idea and then, choose the video which best captures your taste! Build UI walkthroughs, data explainers, launch videos etc.. The arena is currently live with our harness partners HeyGen, Remotion, Hashtagart Mosaic; and is supported by the OpenAI and Paul Jankura models. Please try it out! Looking forward to what y’all build!

Arushi Gandhi

27,644 просмотров • 19 дней назад

If you know me, you know I have been thinking about ways to improve video storytelling forever. I am excited to launch Eddie AI - 'ChatGPT for video editing' as our early users say. Eddie is the AI storytelling partner for video pros. Craft rough cuts from your interview footage with a prompt. *In seconds.* Try it: And check out the launch video. And watch it to the end. Like the end end. 😉
2:36

If you know me, you know I have been thinking about ways to improve video storytelling forever. I am excited to launch Eddie AI - 'ChatGPT for video editing' as our early users say. Eddie is the AI storytelling partner for video pros. Craft rough cuts from your interview footage with a prompt. *In seconds.* Try it: And check out the launch video. And watch it to the end. Like the end end. 😉

Shamir Allibhai

30,020 просмотров • 2 лет назад

Check out our #ICRA2024 paper "Actor-Critic Model Predictive Control." Model-free #reinforcementlearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) benefits from robustness and online replanning capabilities. We combine both approaches by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an Actor-Critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in simulation and the real world with a quadcopter across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior. Paper: Full Video with more details: Kudos to Ángel Romero, Yunlong Song IEEE ICRA University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)
1:26

Check out our #ICRA2024 paper "Actor-Critic Model Predictive Control." Model-free #reinforcementlearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) benefits from robustness and online replanning capabilities. We combine both approaches by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an Actor-Critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in simulation and the real world with a quadcopter across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior. Paper: Full Video with more details: Kudos to Ángel Romero, Yunlong Song IEEE ICRA University of Zurich UZH Science UZH Space Hub Aerial Core AUTOASSESS European Research Council (ERC)

Davide Scaramuzza

34,920 просмотров • 2 лет назад

Live Update from Abu Dhabi: A Message from Adam, Head of BD The Pinlink team is excited to share that we’ll be attending the Bitcoin MENA Conference this week. Here’s what we’re up to: Networking with Bitcoin mining businesses to tokenize and fractionalize them on the Pinlink platform. Meeting with some exciting prospective blue chip partners and connecting with existing partners to strategize on big plans we have together. Sharing the first photo and video footage of miner hosting facility. This is our first conference, and we look forward to attending many more to connect with the #DePIN community. Stay tuned for updates as we continue driving Pinlink forward.
0:30

Live Update from Abu Dhabi: A Message from Adam, Head of BD The Pinlink team is excited to share that we’ll be attending the Bitcoin MENA Conference this week. Here’s what we’re up to: Networking with Bitcoin mining businesses to tokenize and fractionalize them on the Pinlink platform. Meeting with some exciting prospective blue chip partners and connecting with existing partners to strategize on big plans we have together. Sharing the first photo and video footage of miner hosting facility. This is our first conference, and we look forward to attending many more to connect with the #DePIN community. Stay tuned for updates as we continue driving Pinlink forward.

PinLink

98,519 просмотров • 1 год назад