Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Diffusion and flow matching-based robot planners are slow and generate noisy and jerky trajectories. Delighted to share our ICRA 2026 paper, which leverages IMLE to improve planning frequency 19-fold from 4.3 Hz to 83 Hz and reduces jerk by 38% relative to flow matching. Joint work w/ Grayson Lee,... show more

Ke Li 🍁

7,395 subscribers

102,877 Aufrufe • vor 4 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

16 Kommentare

Profilbild von Ke Li 🍁 @ ECCV 2026

Ke Li 🍁 @ ECCV 2026vor 4 Monaten

Generative models, such as diffusion models and flow matching, are widely used for robot planning. The goal is to model the probability distribution over plausible future trajectories conditioned on the initial state. (2/7)

Profilbild von Ke Li 🍁 @ ECCV 2026

Ke Li 🍁 @ ECCV 2026vor 4 Monaten

Our method uses Implicit Maximum Likelihood Estimation (IMLE) to train a one-step generative model on trajectories of varying quality. IMLE works by having each trajectory in the training dataset pull the closest generated trajectory towards it. (3/7)

Profilbild von Ke Li 🍁 @ ECCV 2026

Ke Li 🍁 @ ECCV 2026vor 4 Monaten

Vanilla IMLE treats all trajectories in the training dataset equally despite their varying quality. Since our goal is to generate trajectories of high quality, we propose reward-weighted IMLE, which emphasizes trajectories with high reward more than those with low reward. (4/7)

Profilbild von Ke Li 🍁 @ ECCV 2026

Ke Li 🍁 @ ECCV 2026vor 4 Monaten

We demonstrate on offline reinforcement learning tasks that our IMLE model runs in real time on both the CPU and GPU and achieves an order-of-magnitude speedup relative to diffusion models. (5/7)

Profilbild von Ke Li 🍁 @ ECCV 2026

Ke Li 🍁 @ ECCV 2026vor 4 Monaten

We deploy our planner onboard a mobile robot and demonstrate real-time navigation in dynamic multi-agent environments. (6/7)

Profilbild von Ke Li 🍁 @ ECCV 2026

Ke Li 🍁 @ ECCV 2026vor 4 Monaten

For more details, see: - Project website: - Paper: - Code: If you are at ICRA, come check out Poster 301 on Tuesday afternoon! (7/7)

Profilbild von Mr V

Mr Vvor 4 Monaten

Bruh diffusion models are really bad at generalizing in real-time , learnt a lot from your paper and I think I will find myself always going back to this. Who would have thought that the denoising process is the root cause. My initial thoughts were that without the de-noisers generalizations wouldn’t work or be optimal. This is great works and it is real time.

Profilbild von Emeka

Emekavor 4 Monaten

This is so nice ! 👏👏👏

Profilbild von oriel haim

oriel haimvor 4 Monaten

This is really cool! Are you looking for employees? Not for the salary I just like to play with things like this 🙃

Profilbild von Julley Thai

Julley Thaivor 4 Monaten

Lil bro playing crossing road.

Profilbild von Priyav K Kaneria

Priyav K Kaneriavor 4 Monaten

really cool

Profilbild von Mooncast Productions

Mooncast Productionsvor 4 Monaten

boids demo?

Profilbild von Alper Ahmetoglu

Alper Ahmetogluvor 4 Monaten

Felt nostalgic. 2018, I was trying something like IMLE in the other KL direction, and was trying to find a way to prevent collapse. Then I stumbled upon IMLE. Cool idea! Will check this out.

Profilbild von LSMO

LSMOvor 4 Monaten

Cool

Profilbild von Saleh Aldwais

Saleh Aldwaisvor 4 Monaten

This is good it didn’t stop at all butterfly find a way while moving

Profilbild von Wendy Carlosa

Wendy Carlosavor 4 Monaten

looks like how the military would operate if the soldiers were cats

Ähnliche Videos

Diffusion/flow matching is a go-to generative AI paradigm, but it's slow, sample inefficient and complicated. Do we really need all this complexity? We design a small and simple one-step model, ROMS-IMLE, which attains an FID of 2.56 on ImageNet. Joint w/ Chirag Vashist 1/6

Diffusion/flow matching is a go-to generative AI paradigm, but it's slow, sample inefficient and complicated. Do we really need all this complexity? We design a small and simple one-step model, ROMS-IMLE, which attains an FID of 2.56 on ImageNet. Joint w/ Chirag Vashist 1/6

Ke Li 🍁

37,065 Aufrufe • vor 1 Monat

$Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.$

Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.

AK

93,765 Aufrufe • vor 3 Jahren

Meta just announced FlowVid Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis paper page: Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

Meta just announced FlowVid Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis paper page: Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

AK

123,729 Aufrufe • vor 2 Jahren

The holy grail of robotics is to be able to perform previously-unseen, out-of-distribution manipulation tasks “zero shot” in a new environment. NovaFlow proposes an approach which (1) generates a video, (2) computes predicted flow — how points move through the scene — and (3) uses this flow as an objective to generate a motion. Using this procedure, NovaFlow generates motions in unseen scenes, for unseen tasks, and can transfer across embodiments. To learn more, we are joined by Hongyu Li and Jiahui Fu from RAI. Watch Episode #63 of RoboPapers with Chris Paxton and Michael Cho - Rbt/Acc now to learn more!

The holy grail of robotics is to be able to perform previously-unseen, out-of-distribution manipulation tasks “zero shot” in a new environment. NovaFlow proposes an approach which (1) generates a video, (2) computes predicted flow — how points move through the scene — and (3) uses this flow as an objective to generate a motion. Using this procedure, NovaFlow generates motions in unseen scenes, for unseen tasks, and can transfer across embodiments. To learn more, we are joined by Hongyu Li and Jiahui Fu from RAI. Watch Episode #63 of RoboPapers with Chris Paxton and Michael Cho - Rbt/Acc now to learn more!

RoboPapers

21,434 Aufrufe • vor 7 Monaten

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,596 Aufrufe • vor 1 Jahr

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 Aufrufe • vor 1 Jahr

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

AK

46,151 Aufrufe • vor 3 Jahren

This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen*, Chen Wang*, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"

This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen, Chen Wang, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"

Jim Fan

69,064 Aufrufe • vor 3 Jahren

We’ve seen lots of incredible videos of humanoid robots dancing, doing martial arts, running up walls — but these extreme behaviors are usually from individual, highly specialized policies. But now OmniXtreme shows us how to achieve incredible behaviors that push the limits of humanoid motion, by (1) training a flow-based motion generative model, and (2) doing residual RL post-training to handle complex real-world dynamics. Yunshen Wang and Shaohang Zhu join us to talk about their work towards general-purpose high performance humanoid robot control. Watch Episode #76 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

We’ve seen lots of incredible videos of humanoid robots dancing, doing martial arts, running up walls — but these extreme behaviors are usually from individual, highly specialized policies. But now OmniXtreme shows us how to achieve incredible behaviors that push the limits of humanoid motion, by (1) training a flow-based motion generative model, and (2) doing residual RL post-training to handle complex real-world dynamics. Yunshen Wang and Shaohang Zhu join us to talk about their work towards general-purpose high performance humanoid robot control. Watch Episode #76 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

RoboPapers

11,743 Aufrufe • vor 5 Monaten

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:

Heng Yang

19,228 Aufrufe • vor 6 Monaten

We are excited to share that our paper “Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe” published in the IEEE Transactions on Robotics! One year in the making! PDF: Video: We focus on autonomous quadrotor flight in narrow pipes, where self-induced, unsteady airflow can significantly affect stability and control. Our approach closes the loop using real-time flow field measurements: an #EventCamera-based smoke velocimetry method provides low-latency airflow estimates, which are used by a learning-based disturbance estimator and integrated into a reinforcement-learning controller. The results are improved hovering and lateral translation performance, helping make flight in confined spaces more stable and safer! Kudos to Leonard Bauersfeld Leonard Bauersfeld! Reference: Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe, IEEE Transactions on Robotics (T-RO), 2026 European Research Council (ERC) UZH IfI UZH Space Hub University of Zurich Swiss Robotics UZH Science UZHai AUTOASSESS IEEE Transactions on Robotics (T-RO) #DVS Prophesee

We are excited to share that our paper “Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe” published in the IEEE Transactions on Robotics! One year in the making! PDF: Video: We focus on autonomous quadrotor flight in narrow pipes, where self-induced, unsteady airflow can significantly affect stability and control. Our approach closes the loop using real-time flow field measurements: an #EventCamera-based smoke velocimetry method provides low-latency airflow estimates, which are used by a learning-based disturbance estimator and integrated into a reinforcement-learning controller. The results are improved hovering and lateral translation performance, helping make flight in confined spaces more stable and safer! Kudos to Leonard Bauersfeld Leonard Bauersfeld! Reference: Low-Latency Event-Based Velocimetry for Quadrotor Control in a Narrow Pipe, IEEE Transactions on Robotics (T-RO), 2026 European Research Council (ERC) UZH IfI UZH Space Hub University of Zurich Swiss Robotics UZH Science UZHai AUTOASSESS IEEE Transactions on Robotics (T-RO) #DVS Prophesee

Davide Scaramuzza

50,360 Aufrufe • vor 7 Monaten

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:

Léo

15,777 Aufrufe • vor 2 Monaten

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

🔥 JUST IN: Open-source robotics dataset from 100% real-world scenarios! 🤯 Chinese robotics company AGIBOT just released AGIBOT WORLD 2026, an open-source dataset systematically covering key embodied AI research directions. Built entirely from real-world environments: commercial spaces, and homes. Collected using AGIBOT G2 robots in free-form collection mode, providing structured, accurately annotated, high-quality data. Digital twin technology creates 1:1 scale replicas in simulation matching the real environments. Both real-world and simulation data are open-sourced. The AGIBOT G2 platform collects multiple data types simultaneously: RGB(D) cameras, tactile sensors, force sensors, LiDAR, IMU, and full-body joint states. Whole-body control coordinates arms, waist, and hands for complex tasks. First-person teleoperation lets operators control the robot from its perspective. The tasks covered are fine-grained manipulation, ultra-long-horizon tasks, spatial navigation, dual-arm coordination, and multi-agent/human-robot collaboration. The dataset includes error-recovery trajectories with annotations. Most datasets only show successful demonstrations. AGIBOT includes failures and how the robot recovers, teaching models how to handle mistakes. After collection, data is tested through policy training and real-robot deployment to ensure quality. Then processed through industrial quality control with multiple screening and cleaning rounds. Making it open-source accelerates embodied AI research by giving researchers access to high-quality real-world robot data at scale. 🇨🇳 Learn more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

40,583 Aufrufe • vor 6 Monaten

Long horizon robotics is hard. Robots drift, forget the goal, and fall apart when a task takes hundreds of steps. This small independent team hit rank 1 on Stanford’s Behavior1K benchmark by Fei Fei Li’s lab… Ahead of teams from NVIDIA, CMU, and several big labs. What they found useful: ✅ Split each task into stages so the policy always knows where it is Keeps the robot on track for long sequences ✅ Predict a chunk of actions instead of step by step Reduces jitter and creates smoother motion ✅ Add correlated noise during training Matches real robot behavior better and makes the policy more stable ✅ Use many samples per forward pass Lowers variance and helps when trajectories are long ✅ Keep high level context isolated from noisy signals Prevents the model from getting confused mid task These ideas transfer well to real robots: Stage awareness, chunked execution, realistic noise, and clean attention flow all reduce drift and make long tasks possible. All built by a small independent group: Ilia Akash, and Gleb Zarin. Thanks for pointing out, Abhishek Parmar. I have no scientific verdict on the full method, but the ideas are interesting and worth reading. —- Weekly robotics and AI insights. Subscribe free:

Long horizon robotics is hard. Robots drift, forget the goal, and fall apart when a task takes hundreds of steps. This small independent team hit rank 1 on Stanford’s Behavior1K benchmark by Fei Fei Li’s lab… Ahead of teams from NVIDIA, CMU, and several big labs. What they found useful: ✅ Split each task into stages so the policy always knows where it is Keeps the robot on track for long sequences ✅ Predict a chunk of actions instead of step by step Reduces jitter and creates smoother motion ✅ Add correlated noise during training Matches real robot behavior better and makes the policy more stable ✅ Use many samples per forward pass Lowers variance and helps when trajectories are long ✅ Keep high level context isolated from noisy signals Prevents the model from getting confused mid task These ideas transfer well to real robots: Stage awareness, chunked execution, realistic noise, and clean attention flow all reduce drift and make long tasks possible. All built by a small independent group: Ilia Akash, and Gleb Zarin. Thanks for pointing out, Abhishek Parmar. I have no scientific verdict on the full method, but the ideas are interesting and worth reading. —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

12,954 Aufrufe • vor 10 Monaten

M.S. Barbano et al. (2010) investigated "the size, shape, position, pre-transport setting, and long-axis orientation of 175 boulders found along the south-eastern Ionian coast of Sicily. These megaclasts were deposited by the sea or moved from the same rocky coast to the shore platform and distributed within 60 m of the shoreline. Most boulders are rectangular to ellipsoidal, with sharp, broken edges. They are calcarenite fragments up to 27 t in mass."[1] Sicily detail: The ECDO State 1 to State 2 simulation detail is modelled over nine hours. The locations of the boulders are at (d) along the southeastern coast of Sicily. Orientation of the long axes of the boulders is visualised in the rose chart, closely matching the direction of flow seen in this region. Evidence of submarine cutting of the shelf can be seen southeast of the saddles in Calabria (a and b) and east of Sicily (c), where heavy flow may have been an agent of bench cutting along the flow axes (d) which aligns very closely with the ECDO moment of inertia for this location (12-13° west of north). Orange/red markers are marine fossil deposits. State 2 water levels are expected to have been higher than seen here as some returning water is lost from the simulation domain. [1] [2]

M.S. Barbano et al. (2010) investigated "the size, shape, position, pre-transport setting, and long-axis orientation of 175 boulders found along the south-eastern Ionian coast of Sicily. These megaclasts were deposited by the sea or moved from the same rocky coast to the shore platform and distributed within 60 m of the shoreline. Most boulders are rectangular to ellipsoidal, with sharp, broken edges. They are calcarenite fragments up to 27 t in mass."[1] Sicily detail: The ECDO State 1 to State 2 simulation detail is modelled over nine hours. The locations of the boulders are at (d) along the southeastern coast of Sicily. Orientation of the long axes of the boulders is visualised in the rose chart, closely matching the direction of flow seen in this region. Evidence of submarine cutting of the shelf can be seen southeast of the saddles in Calabria (a and b) and east of Sicily (c), where heavy flow may have been an agent of bench cutting along the flow axes (d) which aligns very closely with the ECDO moment of inertia for this location (12-13° west of north). Orange/red markers are marine fossil deposits. State 2 water levels are expected to have been higher than seen here as some returning water is lost from the simulation domain. [1] [2]

Craig Stone

28,110 Aufrufe • vor 11 Monaten

PhD Students – How to automatically identify 90% of the issues in your research paper before you submit it to a journal? This is possible through manual or automated paper review. First, let’s understand the following. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐚 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰? Paper review is a process in which subject matter experts evaluate your paper based on the following criteria: 1. Significance – Is this research important? 2. Novelty – Is this research new? 3. Methodology – Is this research carried out in the correct way? 4. Verifiability – Can other researchers verify this research? 5. Presentation – Is the research presented in the right way? 𝐖𝐡𝐲 𝐭𝐨 𝐡𝐚𝐯𝐞 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰𝐞𝐝 𝐛𝐞𝐟𝐨𝐫𝐞 𝐬𝐮𝐛𝐦𝐢𝐬𝐬𝐢𝐨𝐧? ➟ Identify the critical issues in your paper ➟ Fix those issues to increase the chances of your paper acceptance 𝐇𝐨𝐰 𝐭𝐨 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐞 “𝐬𝐞𝐥𝐟-𝐫𝐞𝐯𝐢𝐞𝐰” 𝐨𝐟 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫? Paperpal just launched an amazing feature – AI Review. Using this feature, you can get instant self-feedback. This feature will help you in the following ways. ➝ Check for gaps in your logic ➝ Get feedback on the structure and flow of your writing ➝ Review your research questions ➝ Identify opportunities to strengthen your paper ➝ Increase the chances of your paper acceptance Here is a step-by-step process for using AI Review feature. Step 1: Go to and login. Step 2: Open an existing document or make a new document Step 3: Go to the right-side bar and click on checks | AI Review. Step 4: For this feature to work there should be more than 150 words. Step 5: Copy and paste your paper. Step 6: Now go to the right side and check the prompts Step 7: With these prompts, you will evaluate your paper. Step 8: You will find various prompts e.g., suggest writing feedback, check flow and structure etc. Step 9: You can select a prompt from the existing prompts or write your custom prompt and execute Step 10: Paperpal will generate feedback as per the prompt. Step 11: Read through the feedback and save it for further use. Use other specific prompts for tailored feedback. Step 12: This way you can evaluate various aspects of your paper yourself. This is a very customized and efficient way of automatically reviewing your paper. You can also go one step further to work on the feedback and improve your paper based on suggestions. Please note that AI Review feature does not replace human or expert reviewers in any way. This feature only aims to provide you with quick self-feedback. Try the AI Review feature of Paperpal. Paperpal link:

PhD Students – How to automatically identify 90% of the issues in your research paper before you submit it to a journal? This is possible through manual or automated paper review. First, let’s understand the following. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐚 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰? Paper review is a process in which subject matter experts evaluate your paper based on the following criteria: 1. Significance – Is this research important? 2. Novelty – Is this research new? 3. Methodology – Is this research carried out in the correct way? 4. Verifiability – Can other researchers verify this research? 5. Presentation – Is the research presented in the right way? 𝐖𝐡𝐲 𝐭𝐨 𝐡𝐚𝐯𝐞 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫 𝐫𝐞𝐯𝐢𝐞𝐰𝐞𝐝 𝐛𝐞𝐟𝐨𝐫𝐞 𝐬𝐮𝐛𝐦𝐢𝐬𝐬𝐢𝐨𝐧? ➟ Identify the critical issues in your paper ➟ Fix those issues to increase the chances of your paper acceptance 𝐇𝐨𝐰 𝐭𝐨 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐞 “𝐬𝐞𝐥𝐟-𝐫𝐞𝐯𝐢𝐞𝐰” 𝐨𝐟 𝐲𝐨𝐮𝐫 𝐩𝐚𝐩𝐞𝐫? Paperpal just launched an amazing feature – AI Review. Using this feature, you can get instant self-feedback. This feature will help you in the following ways. ➝ Check for gaps in your logic ➝ Get feedback on the structure and flow of your writing ➝ Review your research questions ➝ Identify opportunities to strengthen your paper ➝ Increase the chances of your paper acceptance Here is a step-by-step process for using AI Review feature. Step 1: Go to and login. Step 2: Open an existing document or make a new document Step 3: Go to the right-side bar and click on checks | AI Review. Step 4: For this feature to work there should be more than 150 words. Step 5: Copy and paste your paper. Step 6: Now go to the right side and check the prompts Step 7: With these prompts, you will evaluate your paper. Step 8: You will find various prompts e.g., suggest writing feedback, check flow and structure etc. Step 9: You can select a prompt from the existing prompts or write your custom prompt and execute Step 10: Paperpal will generate feedback as per the prompt. Step 11: Read through the feedback and save it for further use. Use other specific prompts for tailored feedback. Step 12: This way you can evaluate various aspects of your paper yourself. This is a very customized and efficient way of automatically reviewing your paper. You can also go one step further to work on the feedback and improve your paper based on suggestions. Please note that AI Review feature does not replace human or expert reviewers in any way. This feature only aims to provide you with quick self-feedback. Try the AI Review feature of Paperpal. Paperpal link:

Faheem Ullah

15,270 Aufrufe • vor 1 Jahr

You might have seen the WuBOT performing at the 2026 Spring Festival Gala; however, most high-dynamic extreme motions you see are executed by overfitted tracking policies. Until now, training a unified policy capable of performing various extreme motions with a high success rate remained an unsolved challenge. We spent an entire year digging into the barrier between general tracking and extreme physical behaviors. After burning through dozens of G1 robots, we finally identified the bottleneck of learning and physical executability. With these discoveries, we developed OmniXtreme: the first general policy that can execute diverse extreme motions, including consecutive flips, extreme balancing, and even breakdancing with rapid contact switches! This capability is achieved by pre-training a flow-based generative control policy and then post-training with actuation-aware residual RL for complex physical dynamics—a step we found critical for successful real-world transfer. This work is a joint collaboration with Unitree. Together, we are pushing the physical limits of humanoid robots. It is incredibly exciting to see a general "robot gymnast" and "robot breakdancer" come to life! It was also our first time publishing a paper with XingXing, which was an enlightening experience. The model checkpoints are now released—we welcome you to play with them! 📦 📄 Paper: 🌐 Project: 💻 Code:

You might have seen the WuBOT performing at the 2026 Spring Festival Gala; however, most high-dynamic extreme motions you see are executed by overfitted tracking policies. Until now, training a unified policy capable of performing various extreme motions with a high success rate remained an unsolved challenge. We spent an entire year digging into the barrier between general tracking and extreme physical behaviors. After burning through dozens of G1 robots, we finally identified the bottleneck of learning and physical executability. With these discoveries, we developed OmniXtreme: the first general policy that can execute diverse extreme motions, including consecutive flips, extreme balancing, and even breakdancing with rapid contact switches! This capability is achieved by pre-training a flow-based generative control policy and then post-training with actuation-aware residual RL for complex physical dynamics—a step we found critical for successful real-world transfer. This work is a joint collaboration with Unitree. Together, we are pushing the physical limits of humanoid robots. It is incredibly exciting to see a general "robot gymnast" and "robot breakdancer" come to life! It was also our first time publishing a paper with XingXing, which was an enlightening experience. The model checkpoints are now released—we welcome you to play with them! 📦 📄 Paper: 🌐 Project: 💻 Code:

Siyuan Huang

108,684 Aufrufe • vor 7 Monaten

We are excited to share our work “Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones” published in IEEE Transactions on Robotics IEEE Transactions on Robotics (T-RO), which tackles sharp radiance field reconstruction under agile drone motion, where RGB frames are heavily motion-blurred and pose priors become unreliable! 4 years in the making! Code & dataset released! PDF: Code & Dataset: Full Narrated Video: High-speed flight is essential for time- and battery-constrained missions (e.g., inspection, exploration, search & rescue). However, fast motion corrupts visual data with severe motion blur and introduces drift/noise in visual-inertial odometry, making NeRF-based 3D reconstruction particularly brittle. We propose a unified framework that leverages asynchronous #EventCamera streams together with motion-blurred frames to reconstruct high-fidelity radiance fields from agile drone flights. Our key idea is to embed event-image fusion directly into radiance field optimization while jointly refining a shared, continuous-time camera trajectory initialized from event-based VIO. This enables us to recover sharp radiance fields and accurate trajectories without ground-truth supervision during training. We validate our method on synthetic data and on real sequences captured by a drone flying up to 2 m/s. Despite severe blur and noisy pose priors, our method preserves fine scene details and achieves a performance gain of over 50% on real-world data compared to state-of-the-art methods. Kudos to Rong Zou and Marco Cannici! Marco Cannici Reference: Rong Zou*, Marco Cannici*, Davide Scaramuzza Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones IEEE Transactions on Robotics (T-RO), 2026 NCCR Robotics European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub

We are excited to share our work “Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones” published in IEEE Transactions on Robotics IEEE Transactions on Robotics (T-RO), which tackles sharp radiance field reconstruction under agile drone motion, where RGB frames are heavily motion-blurred and pose priors become unreliable! 4 years in the making! Code & dataset released! PDF: Code & Dataset: Full Narrated Video: High-speed flight is essential for time- and battery-constrained missions (e.g., inspection, exploration, search & rescue). However, fast motion corrupts visual data with severe motion blur and introduces drift/noise in visual-inertial odometry, making NeRF-based 3D reconstruction particularly brittle. We propose a unified framework that leverages asynchronous #EventCamera streams together with motion-blurred frames to reconstruct high-fidelity radiance fields from agile drone flights. Our key idea is to embed event-image fusion directly into radiance field optimization while jointly refining a shared, continuous-time camera trajectory initialized from event-based VIO. This enables us to recover sharp radiance fields and accurate trajectories without ground-truth supervision during training. We validate our method on synthetic data and on real sequences captured by a drone flying up to 2 m/s. Despite severe blur and noisy pose priors, our method preserves fine scene details and achieves a performance gain of over 50% on real-world data compared to state-of-the-art methods. Kudos to Rong Zou and Marco Cannici! Marco Cannici Reference: Rong Zou, Marco Cannici, Davide Scaramuzza Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones IEEE Transactions on Robotics (T-RO), 2026 NCCR Robotics European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub

Davide Scaramuzza

12,028 Aufrufe • vor 7 Monaten

We are excited to share our #ICRA2026 paper "Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight"! Paper: Video: Can we use Model-Based #ReinforcementLearning (MBRL) to fly a drone from pixels to commands? In this work, we train quadrotor navigation policies from scratch using #WorldModels, mapping raw onboard camera pixels directly to control commands, much like a human pilot! While model-free methods like PPO are sample-inefficient and struggle in this setting, we leverage #MBRL to train visuomotor policies capable of agile flight through a racetrack using only raw pixel observations, no explicit state estimation needed. A key finding: because our policies are trained end-to-end directly from pixels, we no longer need the perception-aware reward term used in previous methods. Instead, this behavior emerges naturally! The policies learn to guide the camera toward feature-rich areas of the observation space on their own. Kudos to Ángel Romero Ashwin Shenai Ismail Geles Elie Aljalbout Reference: "Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight" Angel Romero*, Ashwin Shenai*, Ismail Geles, Elie Aljalbout, Davide Scaramuzza IEEE International Conference on Robotics and Automation (ICRA), Vienna, 2026. European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub Swiss Robotics NCCR Robotics

We are excited to share our #ICRA2026 paper "Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight"! Paper: Video: Can we use Model-Based #ReinforcementLearning (MBRL) to fly a drone from pixels to commands? In this work, we train quadrotor navigation policies from scratch using #WorldModels, mapping raw onboard camera pixels directly to control commands, much like a human pilot! While model-free methods like PPO are sample-inefficient and struggle in this setting, we leverage #MBRL to train visuomotor policies capable of agile flight through a racetrack using only raw pixel observations, no explicit state estimation needed. A key finding: because our policies are trained end-to-end directly from pixels, we no longer need the perception-aware reward term used in previous methods. Instead, this behavior emerges naturally! The policies learn to guide the camera toward feature-rich areas of the observation space on their own. Kudos to Ángel Romero Ashwin Shenai Ismail Geles Elie Aljalbout Reference: "Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight" Angel Romero, Ashwin Shenai, Ismail Geles, Elie Aljalbout, Davide Scaramuzza IEEE International Conference on Robotics and Automation (ICRA), Vienna, 2026. European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub Swiss Robotics NCCR Robotics

Davide Scaramuzza

16,089 Aufrufe • vor 5 Monaten

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model with Gradio demo local demo: This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model with Gradio demo local demo: This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.

AK

810,731 Aufrufe • vor 2 Jahren