Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

⚡ Video diffusion is slow. PDD changes that. 🚀 Introducing Parallel Decoding Distillation, our latest work on accelerating video diffusion models. Paper: Page: By Neta Shaul, Julius Berner, and Chao Liu. Below PDD on LTX-2.3 🔊

Arash Vahdat

11,364 subscribers

19,965 Aufrufe • vor 8 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

🚀 Sol Video Inference Engine is here! An agent-native, training-free full-stack accelerator for video diffusion. It auto-tunes cache + sparse attn + token pruning + quant + kernel fusion for any model/hardware/config. >2× end-to-end speedup on 64B Cosmos3-Super, 22B LTX-2.3 and 2B SANA-Video — near-lossless VBench quality, minimal human effort. Practical acceleration for real video gen deployment. 📄 Paper: 🌐 Project: 💻 Code: Proud of the team! 🎉

🚀 Sol Video Inference Engine is here! An agent-native, training-free full-stack accelerator for video diffusion. It auto-tunes cache + sparse attn + token pruning + quant + kernel fusion for any model/hardware/config. >2× end-to-end speedup on 64B Cosmos3-Super, 22B LTX-2.3 and 2B SANA-Video — near-lossless VBench quality, minimal human effort. Practical acceleration for real video gen deployment. 📄 Paper: 🌐 Project: 💻 Code: Proud of the team! 🎉

Enze Xie

35,993 Aufrufe • vor 1 Monat

Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation paper page: Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.

Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation paper page: Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.

AK

375,160 Aufrufe • vor 3 Jahren

Chop the gradients ✂️! We found that truncating decoder gradients in latent video diffusion to a fixed window allows us to finetune on videos with pixel-wise perceptual losses without running out of memory. Pixel losses have been essential for image generation and reconstruction, but until now, they haven't scaled to long-duration, high-resolution video diffusion due to recursive activation accumulation in causal decoders, leading to OOM during training 💥📉. Project: Video diffusion models can do a lot more 🚀 when you can backprop the decoder! Post-process neural rendered scenes, super-resolve videos, harmonize lighting in controlled synthetic driving scenes, and inpaint videos — all in a single step ⚡ with a quick finetune from a standard diffusion model.

Chop the gradients ✂️! We found that truncating decoder gradients in latent video diffusion to a fixed window allows us to finetune on videos with pixel-wise perceptual losses without running out of memory. Pixel losses have been essential for image generation and reconstruction, but until now, they haven't scaled to long-duration, high-resolution video diffusion due to recursive activation accumulation in causal decoders, leading to OOM during training 💥📉. Project: Video diffusion models can do a lot more 🚀 when you can backprop the decoder! Post-process neural rendered scenes, super-resolve videos, harmonize lighting in controlled synthetic driving scenes, and inpaint videos — all in a single step ⚡ with a quick finetune from a standard diffusion model.

Felix Heide

28,399 Aufrufe • vor 3 Monaten

World modeling and imitation learning have largely been considered two disparate worlds. In our recent work, Unified World Models, just accepted to #RSS2025, Chuning Zhu provides a dead-simple unifying solution: just train a joint diffusion model over actions and future states, but with *decoupled* diffusion time steps across these modalities. Manipulating these decoupled time steps then allows for marginalization or conditioning on actions or states; a single model can serve as a policy, forward dynamics model, video prediction model, or inverse dynamics model by simply setting diffusion timesteps carefully. The resulting model can leverage video datasets along with robot training data much more effectively, and shows improved robustness, generalization, and flexibility. This is exciting because it is frustratingly simple, scalable, and shows strong improvement on real-world robotics problems. Please refer to Chuning Zhu 's excellent thread for more details! More details/code can be found on our website and in the paper -

World modeling and imitation learning have largely been considered two disparate worlds. In our recent work, Unified World Models, just accepted to #RSS2025, Chuning Zhu provides a dead-simple unifying solution: just train a joint diffusion model over actions and future states, but with decoupled diffusion time steps across these modalities. Manipulating these decoupled time steps then allows for marginalization or conditioning on actions or states; a single model can serve as a policy, forward dynamics model, video prediction model, or inverse dynamics model by simply setting diffusion timesteps carefully. The resulting model can leverage video datasets along with robot training data much more effectively, and shows improved robustness, generalization, and flexibility. This is exciting because it is frustratingly simple, scalable, and shows strong improvement on real-world robotics problems. Please refer to Chuning Zhu 's excellent thread for more details! More details/code can be found on our website and in the paper -

Abhishek Gupta

11,430 Aufrufe • vor 1 Jahr

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses

Hao AI Lab

78,660 Aufrufe • vor 1 Jahr

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

AK

25,449 Aufrufe • vor 2 Jahren

🚀 Introducing Sparse VideoGen2 (SVG2) — Pareto-frontier video generation acceleration with semantic-aware sparse attention! 🏆Spotlight paper accepted by #NeurIPS2025 ✅ Training-free & plug-and-play ✅ Up to 2.5× faster on HunyuanVideo, 1.9× faster on Wan 2.1 ✅ SOTA quality preserved (no artifacts, no flickering, no distortion) 📚Paper: 💻Code: 🌐Website: ⚡Attention Kernel: Joint work with: @randwalk0 Yilong Zhao Muyang Li Jintao Zhang Han Cai Yujun Lin Xiuyu Li Chenfeng_X Kelly Peng Jianfei Chen Song Han Kurt Keutzer Ion Stoica See details below ⬇️ (1/6) NeurIPS Conference #NeurIPS #VideoGeneration #GenerativeAI #MachineLearning #EfficientAI #OpenSource

🚀 Introducing Sparse VideoGen2 (SVG2) — Pareto-frontier video generation acceleration with semantic-aware sparse attention! 🏆Spotlight paper accepted by #NeurIPS2025 ✅ Training-free & plug-and-play ✅ Up to 2.5× faster on HunyuanVideo, 1.9× faster on Wan 2.1 ✅ SOTA quality preserved (no artifacts, no flickering, no distortion) 📚Paper: 💻Code: 🌐Website: ⚡Attention Kernel: Joint work with: @randwalk0 Yilong Zhao Muyang Li Jintao Zhang Han Cai Yujun Lin Xiuyu Li Chenfeng_X Kelly Peng Jianfei Chen Song Han Kurt Keutzer Ion Stoica See details below ⬇️ (1/6) NeurIPS Conference #NeurIPS #VideoGeneration #GenerativeAI #MachineLearning #EfficientAI #OpenSource

Haocheng Xi

43,470 Aufrufe • vor 10 Monaten

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

Jonathan Stephens

17,712 Aufrufe • vor 7 Monaten

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google actually decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:

Bilawal Sidhu

44,822 Aufrufe • vor 2 Jahren

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 Aufrufe • vor 1 Jahr

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

MrNeRF

27,428 Aufrufe • vor 1 Jahr

introducing the media synthesis museum an active and interactive entity created to preserve generative cultural objects it starts as a Hugging Face organization that contains modern code for old techniques: VQGAN+CLIP, DALL-E Mini, ModelScope Video, Stable Diffusion 1.5 you can use old models/technique directly on Spaces or locally on modern hardware/software, without the old "colab notebook" dependency rot the idea is to really preserve and make accessible those artifacts and aesthetics - both open source. In the future, we hope to also have also historically relevant closed source like DALL-E 1 and DALL-E 2 from OpenAI, older Midjourney models, older Runway apps/techniques/models (cc Cristóbal Valenzuela David Sam Altman)

introducing the media synthesis museum an active and interactive entity created to preserve generative cultural objects it starts as a Hugging Face organization that contains modern code for old techniques: VQGAN+CLIP, DALL-E Mini, ModelScope Video, Stable Diffusion 1.5 you can use old models/technique directly on Spaces or locally on modern hardware/software, without the old "colab notebook" dependency rot the idea is to really preserve and make accessible those artifacts and aesthetics - both open source. In the future, we hope to also have also historically relevant closed source like DALL-E 1 and DALL-E 2 from OpenAI, older Midjourney models, older Runway apps/techniques/models (cc Cristóbal Valenzuela David Sam Altman)

apolinario (poli)

11,143 Aufrufe • vor 1 Monat

Plagued by perished DTF? 👇🏻👇🏻 When #DTF sponsors and namesets on #footballshirts start disintegrating - the paint smears that result can be hard to fix. See below a classic Birmingham City FC 🟦⬜shirt that we fixed up. The results are fab and the process is explained in our latest video. #DTFPrinting #FootballShirts #ShirtRestoration #KitCollectors #RetroKits #DTFFail #KitRepair #CustomKits #FootyKits #BCFC #wefixkits

Plagued by perished DTF? 👇🏻👇🏻 When #DTF sponsors and namesets on #footballshirts start disintegrating - the paint smears that result can be hard to fix. See below a classic Birmingham City FC 🟦⬜shirt that we fixed up. The results are fab and the process is explained in our latest video. #DTFPrinting #FootballShirts #ShirtRestoration #KitCollectors #RetroKits #DTFFail #KitRepair #CustomKits #FootyKits #BCFC #wefixkits

We Fix Kits

13,434 Aufrufe • vor 1 Jahr

🚀 Excited to share our #ICLR2025 work on planning with neural dynamics models! While our lab has developed diverse neural dynamics models for manipulating rigid, deformable, and granular objects, having the model alone doesn’t solve the problem—planning with it remains a challenge. 💡 Enter BaB-ND, led by Keyi and Jiangwei! We propose a scalable, GPU-accelerated branch-and-bound algorithm, inspired by neural network verification, to enable effective planning for diverse objects modeled with neural dynamics. 🔗 Project page (open-source + detailed docs!): 🎥 Watch the video to see T being pushed around obstacles, and check out Keyi’s thread for more details!

🚀 Excited to share our #ICLR2025 work on planning with neural dynamics models! While our lab has developed diverse neural dynamics models for manipulating rigid, deformable, and granular objects, having the model alone doesn’t solve the problem—planning with it remains a challenge. 💡 Enter BaB-ND, led by Keyi and Jiangwei! We propose a scalable, GPU-accelerated branch-and-bound algorithm, inspired by neural network verification, to enable effective planning for diverse objects modeled with neural dynamics. 🔗 Project page (open-source + detailed docs!): 🎥 Watch the video to see T being pushed around obstacles, and check out Keyi’s thread for more details!

Yunzhu Li

10,561 Aufrufe • vor 1 Jahr

Excited that our paper StreamdiffusionV2 received the Best Research Paper Award at #MLSys26! 🚀Video generation is quickly moving from demos to production-facing workloads. It is no longer a turn-based pipeline but should be a streaming pipeline to interact with users. 📖Our project page: and paper: 👂Come join the talk if you are interested in streaming video generation. Our talk will be at the Research Track Oral Presentation: Best Paper Session on Tue 8:45AM at #MLSys26 , I will talk about how we attacked the efficiency and quality challenges. Hope to see you there! ❤️Huge thanks to all authors! This work would not have been possible without the incredible effort from the entire team. Big shout out to Tianrui Feng, Zhi Li, Shuo Yang , Haocheng Xi, Muyang Li , lllyasviel , Xiuyu Li , Keting Yang, Kelly Peng, Song Han , Maneesh Agrawala, Kurt Keutzer , and あき先生(Aki)

Excited that our paper StreamdiffusionV2 received the Best Research Paper Award at #MLSys26! 🚀Video generation is quickly moving from demos to production-facing workloads. It is no longer a turn-based pipeline but should be a streaming pipeline to interact with users. 📖Our project page: and paper: 👂Come join the talk if you are interested in streaming video generation. Our talk will be at the Research Track Oral Presentation: Best Paper Session on Tue 8:45AM at #MLSys26 , I will talk about how we attacked the efficiency and quality challenges. Hope to see you there! ❤️Huge thanks to all authors! This work would not have been possible without the incredible effort from the entire team. Big shout out to Tianrui Feng, Zhi Li, Shuo Yang , Haocheng Xi, Muyang Li , lllyasviel , Xiuyu Li , Keting Yang, Kelly Peng, Song Han , Maneesh Agrawala, Kurt Keutzer , and あき先生(Aki)

Chenfeng_X

59,315 Aufrufe • vor 2 Monaten

🇨🇳 Another great Chinese Model, OmniHuman-1.5 from ByteDance Turns 1 image plus a voice track into expressive avatar video by pairing a System 1 and System 2 inspired planner with a Diffusion Transformer, Produces coherent motion for over 1 minute with moving camera and multi character scenes. Most avatar models move to the beat of the audio but miss meaning, so gestures feel generic and emotions feel shallow. The fix here is a Multimodal LLM planner that listens to the speech and drafts a structured plan describing intent, emotions, beats, and high level actions, which gives the motion engine clear semantic targets instead of only rhythm. The motion engine is a Multimodal Diffusion Transformer that fuses the plan with audio, the single reference image, and optional text prompts, then synthesizes continuous body, face, and head motion that matches both words and tone. A key trick is a Pseudo Last Frame, a synthetic target that summarizes the next expected state, which stabilizes fusion across modalities and keeps motion consistent over long spans. From just 1 image and speech, the system outputs speaking avatars with synchronized lips, context aware gestures, and continuous camera movement, and it also supports multi character interactions without manual choreography. Reported results show strong lip sync accuracy, high video quality, natural motion, and close match to text prompts, and the same setup works on nonhuman characters too.

Rohan Paul

63,859 Aufrufe • vor 11 Monaten

🚀 Sol-Attn is here! We present a training-free sparse attention method that accelerates video generation while better preserving quality. Sol-Attn unifies dynamic routing, sparse computation, and approximate correction in a single online-softmax pass: • On-the-fly block thresholding for dynamic yet controllable budgets • Proxy-score reuse to approximate unselected blocks Results (vs dense FlashAttention-3): • Wan 2.1-14B: 2.02× end-to-end • HunyuanVideo-13B: 2.12× end-to-end • LTX 2.3: up to 2.4× end-to-end When integrated into Sol-Engine (with kernel fusion + caching): • Wan 2.1-14B: 3.48× end-to-end • HunyuanVideo-13B: 5.08× end-to-end Already available in Sol-Engine. The B200 kernel is still under further optimization. 🎬 Project: 📄 Paper: 🔗 Code:

🚀 Sol-Attn is here! We present a training-free sparse attention method that accelerates video generation while better preserving quality. Sol-Attn unifies dynamic routing, sparse computation, and approximate correction in a single online-softmax pass: • On-the-fly block thresholding for dynamic yet controllable budgets • Proxy-score reuse to approximate unselected blocks Results (vs dense FlashAttention-3): • Wan 2.1-14B: 2.02× end-to-end • HunyuanVideo-13B: 2.12× end-to-end • LTX 2.3: up to 2.4× end-to-end When integrated into Sol-Engine (with kernel fusion + caching): • Wan 2.1-14B: 3.48× end-to-end • HunyuanVideo-13B: 5.08× end-to-end Already available in Sol-Engine. The B200 kernel is still under further optimization. 🎬 Project: 📄 Paper: 🔗 Code:

Enze Xie

20,575 Aufrufe • vor 10 Tagen

Strikes on the rear of the Armed Forces of Ukraine (and again, Dnipropetrovsk) - the consequences - what the "objective control" footage shows... The enemy is much better at concealing the results of their strikes than we are, and therefore there is relatively little footage of their effectiveness. Usually, such footage appears from moving vehicles (which is almost impossible for the Security Service of Ukraine to track). And it is precisely such footage that shows the scale of the consequences of today's strikes on Dnipropetrovsk and its surroundings. As we can see from the video, there are THREE major fire outbreaks in the city (they say the main oil depots have been hit). Not to mention smaller ones (like gas stations). By the way, about the gas stations. Judging by the fact that not only footage of burning gas stations is appearing online, but also footage of strikes on them (the latest video), we can see that this is indeed very precise work by our "heroes". And it's almost impossible to cover all the gas stations. And if this becomes a practice for the enemy, they will start having major logistical problems. Subscribe

Strikes on the rear of the Armed Forces of Ukraine (and again, Dnipropetrovsk) - the consequences - what the "objective control" footage shows... The enemy is much better at concealing the results of their strikes than we are, and therefore there is relatively little footage of their effectiveness. Usually, such footage appears from moving vehicles (which is almost impossible for the Security Service of Ukraine to track). And it is precisely such footage that shows the scale of the consequences of today's strikes on Dnipropetrovsk and its surroundings. As we can see from the video, there are THREE major fire outbreaks in the city (they say the main oil depots have been hit). Not to mention smaller ones (like gas stations). By the way, about the gas stations. Judging by the fact that not only footage of burning gas stations is appearing online, but also footage of strikes on them (the latest video), we can see that this is indeed very precise work by our "heroes". And it's almost impossible to cover all the gas stations. And if this becomes a practice for the enemy, they will start having major logistical problems. Subscribe

🇷🇺 Yuri Podolyaka

15,108 Aufrufe • vor 3 Monaten

Furniture assembly is the task everyone name-drops and nobody actually attempts at real scale. Every demo I have seen is a scaled down IKEA leg or a single arm on a toy chair. This paper does it properly, real scale, bimanual, up to 7 subtasks and 1,550 control steps per episode, and it is validated on a real Kinova Gen3, not just in sim. That real-robot number is the one that matters: only a 16 percent drop on the hardest task going from simulation to hardware. That is a small enough gap to take seriously, and it did not happen by accident. They built a VR teleoperation rig specifically for coordinated dual-arm collection, because generic single-arm teleop setups do not capture the coordination real assembly needs, and the model predicts a continuous progress signal alongside the action chunk rather than a discrete subtask label, letting it auto-transition and catch drift before it compounds into total failure. The simulation ablation is what got them there, 48 to 80 percent over baselines, with another 21 points from their perception and control design study alone, but that is groundwork, not the headline. Watch the video, there is a clip of the robot misgrasping the seat panel, reopening the gripper, and regrasping on its own. That is not scripted recovery behaviour, it emerged from training, and it emerged on hardware. Excellent work from the team from Mitsubishi Electric Research Laboratories, with Oxford and UNC Chapel Hill Clinical Laboratory Science. Video and project page in comments. #Robotics #Manipulation #VLA

Furniture assembly is the task everyone name-drops and nobody actually attempts at real scale. Every demo I have seen is a scaled down IKEA leg or a single arm on a toy chair. This paper does it properly, real scale, bimanual, up to 7 subtasks and 1,550 control steps per episode, and it is validated on a real Kinova Gen3, not just in sim. That real-robot number is the one that matters: only a 16 percent drop on the hardest task going from simulation to hardware. That is a small enough gap to take seriously, and it did not happen by accident. They built a VR teleoperation rig specifically for coordinated dual-arm collection, because generic single-arm teleop setups do not capture the coordination real assembly needs, and the model predicts a continuous progress signal alongside the action chunk rather than a discrete subtask label, letting it auto-transition and catch drift before it compounds into total failure. The simulation ablation is what got them there, 48 to 80 percent over baselines, with another 21 points from their perception and control design study alone, but that is groundwork, not the headline. Watch the video, there is a clip of the robot misgrasping the seat panel, reopening the gripper, and regrasping on its own. That is not scripted recovery behaviour, it emerged from training, and it emerged on hardware. Excellent work from the team from Mitsubishi Electric Research Laboratories, with Oxford and UNC Chapel Hill Clinical Laboratory Science. Video and project page in comments. #Robotics #Manipulation #VLA

Stephen James

14,952 Aufrufe • vor 29 Tagen

The recent Massachusetts Institute of Technology (MIT) CSAIL paper published on Recursive Language Models is a fascinating look into how AI systems reason in 2026. You can check it out here ➡️ → The paper notes that even frontier models suffer from “context rot” as inputs grow longer. More tokens don’t mean more understanding. Instead of compressing or summarising, RLMs “treat long prompts as part of an external environment” …and let the model programmatically inspect, decompose, and recursively requery itself over precise sections. This matters in Web3. Why? Smart contracts are long, stateful, and brittle. One missed assumption = unnecessary hassle. In the video below, we applied RLM principles to improve the prompt shown. The old prompt would try to force multi-step pauses. Therefore breaking the AI's workflow. Now it decomposes requirements, scores confidence for each component, verifies the logic, and then synthesises. Result = Production-ready contracts with flagged risks, not blind single-pass outputs. Try it out with our Smart Contract Generator today!

The recent Massachusetts Institute of Technology (MIT) CSAIL paper published on Recursive Language Models is a fascinating look into how AI systems reason in 2026. You can check it out here ➡️ → The paper notes that even frontier models suffer from “context rot” as inputs grow longer. More tokens don’t mean more understanding. Instead of compressing or summarising, RLMs “treat long prompts as part of an external environment” …and let the model programmatically inspect, decompose, and recursively requery itself over precise sections. This matters in Web3. Why? Smart contracts are long, stateful, and brittle. One missed assumption = unnecessary hassle. In the video below, we applied RLM principles to improve the prompt shown. The old prompt would try to force multi-step pauses. Therefore breaking the AI's workflow. Now it decomposes requirements, scores confidence for each component, verifies the logic, and then synthesises. Result = Production-ready contracts with flagged risks, not blind single-pass outputs. Try it out with our Smart Contract Generator today!

ChainGPT

82,385 Aufrufe • vor 6 Monaten