Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Controllability matters for video generation. We thus propose SparseCtrl that enables keyframe (sketch, depth, img) animation/transition/prediction/interpolation by feeding TEMPORALLY SPARSE condition maps. It’s also compatible with #AnimateDiff. Page:

Ceyuan Yang

1,915 subscribers

39,542 views • 2 years ago •via X (Twitter)

Science & Technology Education #AnimateDiff

Anya Rossi• Live Now

Private livecam show

5 Comments

歸藏(guizang.ai)2 years ago

@camenduru I'm not entirely sure if I got this right, but it sounds like this research turns the Controlnet, which used to be active in every frame, into something that's only active in a few key frames. This enhances video control while reducing resource consumption, right?

k_nearest2 years ago

When do you think this will be up on Arxiv? I’d like to read it in my study group when it’s available.

Yinghao Xu2 years ago

Amazing!

Jay Guthrie2 years ago

@camenduru I WISH THE CODE WAS OUT 😭😭😭😭

k_nearest2 years ago

Looks cool!

Related Videos

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 views • 1 year ago

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

AK

294,442 views • 2 years ago

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

AK

367,000 views • 1 year ago

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

AK

161,530 views • 2 years ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,992 views • 2 years ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 views • 3 years ago

Welcome to the Farcana Playtest The gaming world is shifting. We’ve seen trends come and go. We’ve seen games simplify, casualize, and streamline. But at Farcana, we’ve stayed true to our mission: to build a competitive, ability-driven shooter with depth, precision, and strategy. Since 2022, we’ve been designing a game where every match matters, where teamwork, mastery, and mechanics come together to create something more than just a shooter. Look, we know you are tired of shooters that sacrifice skill for accessibility. Farcana fills that competitive gap between Valorant and CS, where your abilities matter, but your aim matters more. From March 20 to April 1, our early beta opens its doors. And this is what's waiting inside. We've packed this playtest with: • Twin Battlegrounds – Maps with purpose, not just pretty backdrops • Nine Stars – Heroes we've rebuilt from the ground up • Training Room – You asked, we built it • Refreshed Visuals – Thousands of iterations later • Founder Packs – First taste of our player economy, plus an AI companion Farcana: A Shooter Built for Depth The difference is in the details: • Abilities with actual tactical depth - blending TPS precision with hack-and-slash elements • Shooting that responds - our prediction system makes high-ping players competitive • Melee that hurts - not an afterthought like in Overwatch • Team economy over rigid roles - no forced tank-healer-damage trinity • Maps built for competition - with real thought about sightlines and engagement zones The Road Ahead This playtest lays groundwork. Coming soon: • Team ability combos - like Teacher's tornado becoming a firestorm with Sherwood's help • Character customization through our Aspect System • Multi-objective game modes that reward strategy, not just reflexes • Boost System - turning good plays into game-changing moments Why Now? With Marvel Rivals going casual, somebody needs to hold the line for competitive players. That's us. That's Farcana. We built Dizzaract because we believe gamers should shape the worlds they play in. Now, it's time to play your part. From March 20 to April 1, experience the future of TPS gaming, built on your feedback, your passion, and your skill. Drop by the Epic Games Store Store. Wishlist us. Let's show them what competitive TPS can be. -The Farcana Team 💫

Welcome to the Farcana Playtest The gaming world is shifting. We’ve seen trends come and go. We’ve seen games simplify, casualize, and streamline. But at Farcana, we’ve stayed true to our mission: to build a competitive, ability-driven shooter with depth, precision, and strategy. Since 2022, we’ve been designing a game where every match matters, where teamwork, mastery, and mechanics come together to create something more than just a shooter. Look, we know you are tired of shooters that sacrifice skill for accessibility. Farcana fills that competitive gap between Valorant and CS, where your abilities matter, but your aim matters more. From March 20 to April 1, our early beta opens its doors. And this is what's waiting inside. We've packed this playtest with: • Twin Battlegrounds – Maps with purpose, not just pretty backdrops • Nine Stars – Heroes we've rebuilt from the ground up • Training Room – You asked, we built it • Refreshed Visuals – Thousands of iterations later • Founder Packs – First taste of our player economy, plus an AI companion Farcana: A Shooter Built for Depth The difference is in the details: • Abilities with actual tactical depth - blending TPS precision with hack-and-slash elements • Shooting that responds - our prediction system makes high-ping players competitive • Melee that hurts - not an afterthought like in Overwatch • Team economy over rigid roles - no forced tank-healer-damage trinity • Maps built for competition - with real thought about sightlines and engagement zones The Road Ahead This playtest lays groundwork. Coming soon: • Team ability combos - like Teacher's tornado becoming a firestorm with Sherwood's help • Character customization through our Aspect System • Multi-objective game modes that reward strategy, not just reflexes • Boost System - turning good plays into game-changing moments Why Now? With Marvel Rivals going casual, somebody needs to hold the line for competitive players. That's us. That's Farcana. We built Dizzaract because we believe gamers should shape the worlds they play in. Now, it's time to play your part. From March 20 to April 1, experience the future of TPS gaming, built on your feedback, your passion, and your skill. Drop by the Epic Games Store Store. Wishlist us. Let's show them what competitive TPS can be. -The Farcana Team 💫

FAR Labs

680,476 views • 1 year ago

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

Matthias Niessner

18,855 views • 8 months ago

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,585 views • 2 years ago

We are excited to share our #CORL2024 paper on learning quadrotor obstacle avoidance from the visual stream of a single #eventcamera! Trained entirely in simulation! We demonstrate obstacle avoidance both in the dark and in a forest up to 5m/s. PDF: Video: Project page: Event cameras are sensors that output per-pixel-level intensity changes at microsecond latency resolution; they feature nearly zero motion blur and high dynamic range but produce a very large volume of events under significant ego-motion and further lack a high-fidelity continuous-time sensor model in simulation, making direct #sim2real transfer not possible. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance policy with “approximated” simulated events and then fine-tune the perception component with limited events-and-depth real-world data. This technique bridges the sim2real gap for #eventcameras! As at the current state, there is no continuous-time sensor model for event cameras, we hope that this work can finally spur future research leveraging simulation for training event-vision-based policies to create faster, agile robots! Kudos to Anish Bhattacharya, @marcocannic, Vijay Kumar Nikolai Matni UZH Science University of Zurich UZH Space Hub UZH IfI European Research Council (ERC) GRASP Laboratory Penn Engineering

We are excited to share our #CORL2024 paper on learning quadrotor obstacle avoidance from the visual stream of a single #eventcamera! Trained entirely in simulation! We demonstrate obstacle avoidance both in the dark and in a forest up to 5m/s. PDF: Video: Project page: Event cameras are sensors that output per-pixel-level intensity changes at microsecond latency resolution; they feature nearly zero motion blur and high dynamic range but produce a very large volume of events under significant ego-motion and further lack a high-fidelity continuous-time sensor model in simulation, making direct #sim2real transfer not possible. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance policy with “approximated” simulated events and then fine-tune the perception component with limited events-and-depth real-world data. This technique bridges the sim2real gap for #eventcameras! As at the current state, there is no continuous-time sensor model for event cameras, we hope that this work can finally spur future research leveraging simulation for training event-vision-based policies to create faster, agile robots! Kudos to Anish Bhattacharya, @marcocannic, Vijay Kumar Nikolai Matni UZH Science University of Zurich UZH Space Hub UZH IfI European Research Council (ERC) GRASP Laboratory Penn Engineering

Davide Scaramuzza

17,219 views • 1 year ago

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

148,998 views • 1 year ago

VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams paper page: Neural Radiance Fields (NeRFs) excel in photorealistically rendering static scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous devices remains challenging, due to data storage and computational constraints. In this paper, we introduce VideoRF, the first approach to enable real-time streaming and rendering of dynamic radiance fields on mobile platforms. At the core is a serialized 2D feature image stream representing the 4D radiance field all in one. We introduce a tailored training scheme directly applied to this 2D domain to impose the temporal and spatial redundancy of the feature image stream. By leveraging the redundancy, we show that the feature image stream can be efficiently compressed by 2D video codecs, which allows us to exploit video hardware accelerators to achieve real-time decoding. On the other hand, based on the feature image stream, we propose a novel rendering pipeline for VideoRF, which has specialized space mappings to query radiance properties efficiently. Paired with a deferred shading model, VideoRF has the capability of real-time rendering on mobile devices thanks to its efficiency. We have developed a real-time interactive player that enables online streaming and rendering of dynamic scenes, offering a seamless and immersive free-viewpoint experience across a range of devices, from desktops to mobile phones.

VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams paper page: Neural Radiance Fields (NeRFs) excel in photorealistically rendering static scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous devices remains challenging, due to data storage and computational constraints. In this paper, we introduce VideoRF, the first approach to enable real-time streaming and rendering of dynamic radiance fields on mobile platforms. At the core is a serialized 2D feature image stream representing the 4D radiance field all in one. We introduce a tailored training scheme directly applied to this 2D domain to impose the temporal and spatial redundancy of the feature image stream. By leveraging the redundancy, we show that the feature image stream can be efficiently compressed by 2D video codecs, which allows us to exploit video hardware accelerators to achieve real-time decoding. On the other hand, based on the feature image stream, we propose a novel rendering pipeline for VideoRF, which has specialized space mappings to query radiance properties efficiently. Paired with a deferred shading model, VideoRF has the capability of real-time rendering on mobile devices thanks to its efficiency. We have developed a real-time interactive player that enables online streaming and rendering of dynamic scenes, offering a seamless and immersive free-viewpoint experience across a range of devices, from desktops to mobile phones.

AK

38,686 views • 2 years ago

⚡️📣👇Tremendously excited to share our new Cell article, where we develop TriPath, a method for analyzing 3D pathology samples using weakly supervised AI. Article: TriPath enables 3D computational pathology via 3D multiple instance learning allowing AI models to capture intricate morphological details from pathology volumes. Code: Blog post: Tested on two different imaging modalities, and patient cohorts from two institutions. Our superstar Andrew H. Song put in a monumental effort of leading the study, in a fantastic collaboration with Jonathan Liu at University of Washington . Interesting aspects: - Utilizing the whole tissue volume and leveraging 3D deep learning enable superior risk prediction performance compared to 2D deep learning baselines based on a few sampled tissue sections that emulate standard clinical practice. This indicates TriPath can harness additional information provided by 3D tissue morphology. - The performance is also superior to clinical baselines from a reader study that involved six expert pathologists. - The morphologically heterogeneous tissue volume could lead to opposing patient-level outcome predictions, dependent on which portion of the tissue volume is used. This concurs with current clinical literature warning that tissue sampling bias can lead to misdiagnosis. Some limitations: - While the 3D pathology cohort size is unprecedented, it is smaller than typical 2D pathology cohorts. Further large-scale studies will be required for validation. Nevertheless, we believe that this study will initiate a positive cycle, encouraging academic institutions and pharmaceutical companies to contribute large banks of human tissue blocks with paired clinical outcomes, thus speeding up advancements in 3D computational pathology. Concluding insights: We believe that 3D pathology is just around the corner - It has the huge potential to not only augment/improve the current clinical practice centered around 2D examination of human tissue, but also help reveal novel biomarkers for prognosis and therapeutic response.. Harvard Medical School Harvard Data Science Initiative Mass General Brigham Broad Institute

⚡️📣👇Tremendously excited to share our new Cell article, where we develop TriPath, a method for analyzing 3D pathology samples using weakly supervised AI. Article: TriPath enables 3D computational pathology via 3D multiple instance learning allowing AI models to capture intricate morphological details from pathology volumes. Code: Blog post: Tested on two different imaging modalities, and patient cohorts from two institutions. Our superstar Andrew H. Song put in a monumental effort of leading the study, in a fantastic collaboration with Jonathan Liu at University of Washington . Interesting aspects: - Utilizing the whole tissue volume and leveraging 3D deep learning enable superior risk prediction performance compared to 2D deep learning baselines based on a few sampled tissue sections that emulate standard clinical practice. This indicates TriPath can harness additional information provided by 3D tissue morphology. - The performance is also superior to clinical baselines from a reader study that involved six expert pathologists. - The morphologically heterogeneous tissue volume could lead to opposing patient-level outcome predictions, dependent on which portion of the tissue volume is used. This concurs with current clinical literature warning that tissue sampling bias can lead to misdiagnosis. Some limitations: - While the 3D pathology cohort size is unprecedented, it is smaller than typical 2D pathology cohorts. Further large-scale studies will be required for validation. Nevertheless, we believe that this study will initiate a positive cycle, encouraging academic institutions and pharmaceutical companies to contribute large banks of human tissue blocks with paired clinical outcomes, thus speeding up advancements in 3D computational pathology. Concluding insights: We believe that 3D pathology is just around the corner - It has the huge potential to not only augment/improve the current clinical practice centered around 2D examination of human tissue, but also help reveal novel biomarkers for prognosis and therapeutic response.. Harvard Medical School Harvard Data Science Initiative Mass General Brigham Broad Institute

Faisal Mahmood

65,520 views • 2 years ago

Great infrastructure is what underpins the technological trends that consumers/users end up experiencing in their daily lives, it will be the same for Web3. Web2 applications like Uber are a great example of how infrastructure has shaped Web2. This breakthrough ridesharing application wouldn’t have initially been possible without the infrastructure provided by Twilio APIs to message users, Google Maps to see location data, and payments APIs to pay drivers. The same story is now unfolding for Web3 applications enabled to come into existence by Chainlink. When Chainlink launched, DeFi had less value than what one application in DeFi holds today. With access to secure and reliable price data via Chainlink, DeFi grew to over $200 billion in just a few years; There is a clear pattern between the launch of oracle networks and the growth of advanced applications on a blockchain: Developers can’t build truly “smart” contract applications without the oracle infrastructure that Chainlink provides. As the Chainlink platform has expanded, I’ve been excited by the sheer amount of advanced features that developers are now able to put into their smart contracts by using Chainlink. It’s clear to me that giving Web3 developers reliable and extensive infrastructure is the best way to unlock the next leap forward for our industry and for the creation of a cryptographic truth-powered society. API and cross-chain connectivity through oracle advancements such as Chainlink Functions and CCIP represent additional new building blocks for the next wave of Web3 apps. CCIP is also something we are actively working with the capital markets/banking industry on adopting for enabling their systems to efficiently interact with hundreds of blockchains via a single integration. When Chainlink enables all the world’s systems to efficiently connect to multiple chains through a single, cryptographically-secured interface is when we will all see a large acceleration in the value that flows into our industry and the widespread adoption which that leads to. Thank you to Jacquelyn Melinek for hosting me on TechCrunch’s Chain reaction podcast, where I explore these concepts in more depth:

Great infrastructure is what underpins the technological trends that consumers/users end up experiencing in their daily lives, it will be the same for Web3. Web2 applications like Uber are a great example of how infrastructure has shaped Web2. This breakthrough ridesharing application wouldn’t have initially been possible without the infrastructure provided by Twilio APIs to message users, Google Maps to see location data, and payments APIs to pay drivers. The same story is now unfolding for Web3 applications enabled to come into existence by Chainlink. When Chainlink launched, DeFi had less value than what one application in DeFi holds today. With access to secure and reliable price data via Chainlink, DeFi grew to over $200 billion in just a few years; There is a clear pattern between the launch of oracle networks and the growth of advanced applications on a blockchain: Developers can’t build truly “smart” contract applications without the oracle infrastructure that Chainlink provides. As the Chainlink platform has expanded, I’ve been excited by the sheer amount of advanced features that developers are now able to put into their smart contracts by using Chainlink. It’s clear to me that giving Web3 developers reliable and extensive infrastructure is the best way to unlock the next leap forward for our industry and for the creation of a cryptographic truth-powered society. API and cross-chain connectivity through oracle advancements such as Chainlink Functions and CCIP represent additional new building blocks for the next wave of Web3 apps. CCIP is also something we are actively working with the capital markets/banking industry on adopting for enabling their systems to efficiently interact with hundreds of blockchains via a single integration. When Chainlink enables all the world’s systems to efficiently connect to multiple chains through a single, cryptographically-secured interface is when we will all see a large acceleration in the value that flows into our industry and the widespread adoption which that leads to. Thank you to Jacquelyn Melinek for hosting me on TechCrunch’s Chain reaction podcast, where I explore these concepts in more depth:

Sergey Nazarov

316,490 views • 3 years ago

I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by Jerry Liu, CEO of LlamaIndex 🦙. This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning. In detail, you'll learn about: - Routing: Where your agent will use decision-making to route requests to multiple tools. - Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments. - Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process. You’ll also learn how to step through what your agent is doing to debug and improve it iteratively. It’s an exciting time to build agents. Sign up and get started here!

I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by Jerry Liu, CEO of LlamaIndex 🦙. This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning. In detail, you'll learn about: - Routing: Where your agent will use decision-making to route requests to multiple tools. - Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments. - Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process. You’ll also learn how to step through what your agent is doing to debug and improve it iteratively. It’s an exciting time to build agents. Sign up and get started here!

Andrew Ng

297,131 views • 2 years ago

I MADE THE BEST 2D SPRITE AI PIPELINE EVER! After testing so much, doing tons of math, trying various different methods I finally got truly the cleanest sprite result I could ask for! You should bookmark this post! ✅ I think this is the happiest 'aha' moment I've had since starting this journey. Day 48 of making a custom webGL game engine using Cursor 🔽🔽🔽 I've been trying to solve sprites since I started this project. I came close a couple of times but it was always something - either dynamic lights wouldn't work, or the genAI results would make them grainy, or I would have to have incorrect alpha etc. After so much struggling, wanting to give up, thinking it's impossible - it was finally done! Perfect matte interpolation of black & white backgrounds resulting in a clean outline of the sprite, half transparent effects keeping their full quality. But this was not enough for me. I wanted more. I wanted those dynamic lights for the most badass anime effects that I'll be creating for the next many weeks! And somehow, I managed to come up with the perfect balance that is nice to the eye with almost no visible imperfections or artifacts when using any kind of dynamic lights. See for yourself! 🔽🔽🔽 How does this magic happen? 1⃣ First, the animation is created using Kling AI. I found it the best to keep the quality anime 2D style I wanted, but I guess others may work as well. 2⃣ Then, the frames are carefully extracted at lossless quality and processed via automation through Nano Banana pass to create an opposite background (white->black or black->white). 3⃣ Once we have a proper spritesheet, I then used MiDaS to compute the depth maps, and then algorithmically create PBR maps for lighting. The remaining depth maps can also be used for various effects, although haven't explored it much yet. 4⃣ Finally, all elements are cut into single frames and overlapped with their respective normal maps. If done right, they're pretty much ready for runtime. I like my heroes to be 512 to truly stand out like fighting game characters, but all of this works with far smaller or bigger sprites. Mind you, it's important to note that Nano Banana has a max output of 4k. Therefore you don't want to make your atlases larger than 4k - which is still quite a lot since it allows you 8x8 512 frames. 💡💡💡 I believe the output quality is beyond anything currently available, and I'll be looking into packaging the whole thing into a web app so people can use it. If you find this useful, please repost it! I really wanted to share this method to the world. Thank you for reading! Look forward to day 49 update!

I MADE THE BEST 2D SPRITE AI PIPELINE EVER! After testing so much, doing tons of math, trying various different methods I finally got truly the cleanest sprite result I could ask for! You should bookmark this post! ✅ I think this is the happiest 'aha' moment I've had since starting this journey. Day 48 of making a custom webGL game engine using Cursor 🔽🔽🔽 I've been trying to solve sprites since I started this project. I came close a couple of times but it was always something - either dynamic lights wouldn't work, or the genAI results would make them grainy, or I would have to have incorrect alpha etc. After so much struggling, wanting to give up, thinking it's impossible - it was finally done! Perfect matte interpolation of black & white backgrounds resulting in a clean outline of the sprite, half transparent effects keeping their full quality. But this was not enough for me. I wanted more. I wanted those dynamic lights for the most badass anime effects that I'll be creating for the next many weeks! And somehow, I managed to come up with the perfect balance that is nice to the eye with almost no visible imperfections or artifacts when using any kind of dynamic lights. See for yourself! 🔽🔽🔽 How does this magic happen? 1⃣ First, the animation is created using Kling AI. I found it the best to keep the quality anime 2D style I wanted, but I guess others may work as well. 2⃣ Then, the frames are carefully extracted at lossless quality and processed via automation through Nano Banana pass to create an opposite background (white->black or black->white). 3⃣ Once we have a proper spritesheet, I then used MiDaS to compute the depth maps, and then algorithmically create PBR maps for lighting. The remaining depth maps can also be used for various effects, although haven't explored it much yet. 4⃣ Finally, all elements are cut into single frames and overlapped with their respective normal maps. If done right, they're pretty much ready for runtime. I like my heroes to be 512 to truly stand out like fighting game characters, but all of this works with far smaller or bigger sprites. Mind you, it's important to note that Nano Banana has a max output of 4k. Therefore you don't want to make your atlases larger than 4k - which is still quite a lot since it allows you 8x8 512 frames. 💡💡💡 I believe the output quality is beyond anything currently available, and I'll be looking into packaging the whole thing into a web app so people can use it. If you find this useful, please repost it! I really wanted to share this method to the world. Thank you for reading! Look forward to day 49 update!

Startracker 🔺

145,925 views • 4 months ago

I am posting this on behalf of ex British Policeman, Mark Sexton M.Sexton for many reasons, not least of which I think this route he is taking against tyranny, whilst very different to mine, is a route that needed to be taken. If it resonates with you, please share it on X, and download the video, copy and paste, and repost everywhere you can. Here's Mark's message: It’s been 40 days exactly and we’ve had no acknowledgment from King Charles III to the petitions sent by many thousands of us. Please watch the video, then click on the links and read in order 1, 2, and then 3. Link 4 is the original eight page petition we sent to the king, attached here for information and reference, also for those who have never seen or read it before. Share this far and wide every household across the nation needs to be made aware giving them the choice of actioning this or not. 1, 2, 3, 4,

I am posting this on behalf of ex British Policeman, Mark Sexton M.Sexton for many reasons, not least of which I think this route he is taking against tyranny, whilst very different to mine, is a route that needed to be taken. If it resonates with you, please share it on X, and download the video, copy and paste, and repost everywhere you can. Here's Mark's message: It’s been 40 days exactly and we’ve had no acknowledgment from King Charles III to the petitions sent by many thousands of us. Please watch the video, then click on the links and read in order 1, 2, and then 3. Link 4 is the original eight page petition we sent to the king, attached here for information and reference, also for those who have never seen or read it before. Share this far and wide every household across the nation needs to be made aware giving them the choice of actioning this or not. 1, 2, 3, 4,

Mark Attwood

341,545 views • 1 year ago

It’s time to expose Toyota. Toyota USA has been one of the most trusted brands in America but they’ve gone totally woke. Here’s some of what we found: • Toyota sponsored a drag queen program at a summer camp for kids identifying as LGBTQ+. • Toyota opposes laws that ban sex changes for kids and funds groups who work to make sex changes legal for children and they worked with the Human Rights Campaign Rapid Response to oppose these laws. • Toyota openly supports “the equality act" which would allow men into girl’s bathrooms, sports and locker-rooms. • Toyota funded the Human Rights Campaign Rapid Response’s Time To Thrive Summit where they worked with the largest teachers union to push gender ideology into elementary schools. • Donated to the HRC, the Trevor Project, Dallas Resource Center and Los Angeles LGBT Center and the Workplace Equality Summit — All supporters of child transitions. The Trevor Project features chat rooms where adults have been caught talking to kids about sexual kinks, how to transition, masturbation and more. They also have a quick exit feature to wipe the browser and hide the website from parents. • Funded many "all ages" pride events. • Woke DEI trainings. • LGBT Chamber of Commerce member. • Preferential treatment for "diverse" suppliers. • Hosted LGBTQ+ events at corporate. • Created custom cars with a trans flag. • ERG groups divided by race + sexual orientation. • A total commitment to DEI policies. • 100/100 CEI score from the HRC for 16 straight years which means that they pay for not only employee transitions but for children of employee’s to transition in states where it’s legal. To put it mildly, Toyota seems to have forgotten who their core customers are. They depend on American families and Japanese families to buy their cars. It’s time to remind them who their customers are. I don’t think the values at corporate reflect the values many Toyota/Lexus owners have (with the exception of maybe Prius owners who probably like the woke stuff). Do Toyota/Lexus owners want the money they spend with Toyota to be used later by corporate to push an ideology that’s diametrically opposed to their own values? If you think Toyota’s values don’t align with yours and you’re a customer or potential customer who wants to speak out, their contact info is ⬇️ Remember to ALWAYS BE KIND. Many in customer service agree with you. These are all publicly listed contacts: Toyota customer care: +1-800-331-4331 Specific Emails: Ed Hellwig 469-292-1165 edward.hellwig@toyota.com Marissa Borjon 469-292-6395 marissa.borjon@toyota.com Victor Vanov 469-292-1318 victor.vanov@toyota.com Contact them with one click: Our goal with reporting is never destruction, it’s restoration of sanity. We do this by informing consumers about the policies companies are adopting so they can make choices about what they’re willing to support as a consumer. That’s not cancel culture, it’s capitalism. When we use our voices and wallets to vote our values, we can change the world and we can restore great companies to a culture of sanity, meritocracy and political neutrality OR we can inspire competitors to step up to fight for our business. The customer is king and most of us just want companies we shop at to stop virtue signaling about divisive social, cultural and political issues. Companies need to stop dividing us by injecting wokeness! If you agree then share this video and tag other accounts so they share it too. If you support our movement to expose and remove woke policies in corporate America then you can help fund our growing team by subscribing to my X page for $5 a month or choose any amount once at You can become a whistleblower against a woke company at by emailing EliminateDEI@protonmail.com or by going to P.S. What’s sad is there’s sooooo much more we couldn’t fit in this video. Maybe we’ll do a part 2 and/or drop key nuggets of information over the next few weeks… 🤔

It’s time to expose Toyota. Toyota USA has been one of the most trusted brands in America but they’ve gone totally woke. Here’s some of what we found: • Toyota sponsored a drag queen program at a summer camp for kids identifying as LGBTQ+. • Toyota opposes laws that ban sex changes for kids and funds groups who work to make sex changes legal for children and they worked with the Human Rights Campaign Rapid Response to oppose these laws. • Toyota openly supports “the equality act" which would allow men into girl’s bathrooms, sports and locker-rooms. • Toyota funded the Human Rights Campaign Rapid Response’s Time To Thrive Summit where they worked with the largest teachers union to push gender ideology into elementary schools. • Donated to the HRC, the Trevor Project, Dallas Resource Center and Los Angeles LGBT Center and the Workplace Equality Summit — All supporters of child transitions. The Trevor Project features chat rooms where adults have been caught talking to kids about sexual kinks, how to transition, masturbation and more. They also have a quick exit feature to wipe the browser and hide the website from parents. • Funded many "all ages" pride events. • Woke DEI trainings. • LGBT Chamber of Commerce member. • Preferential treatment for "diverse" suppliers. • Hosted LGBTQ+ events at corporate. • Created custom cars with a trans flag. • ERG groups divided by race + sexual orientation. • A total commitment to DEI policies. • 100/100 CEI score from the HRC for 16 straight years which means that they pay for not only employee transitions but for children of employee’s to transition in states where it’s legal. To put it mildly, Toyota seems to have forgotten who their core customers are. They depend on American families and Japanese families to buy their cars. It’s time to remind them who their customers are. I don’t think the values at corporate reflect the values many Toyota/Lexus owners have (with the exception of maybe Prius owners who probably like the woke stuff). Do Toyota/Lexus owners want the money they spend with Toyota to be used later by corporate to push an ideology that’s diametrically opposed to their own values? If you think Toyota’s values don’t align with yours and you’re a customer or potential customer who wants to speak out, their contact info is ⬇️ Remember to ALWAYS BE KIND. Many in customer service agree with you. These are all publicly listed contacts: Toyota customer care: +1-800-331-4331 Specific Emails: Ed Hellwig 469-292-1165 [email protected] Marissa Borjon 469-292-6395 [email protected] Victor Vanov 469-292-1318 [email protected] Contact them with one click: Our goal with reporting is never destruction, it’s restoration of sanity. We do this by informing consumers about the policies companies are adopting so they can make choices about what they’re willing to support as a consumer. That’s not cancel culture, it’s capitalism. When we use our voices and wallets to vote our values, we can change the world and we can restore great companies to a culture of sanity, meritocracy and political neutrality OR we can inspire competitors to step up to fight for our business. The customer is king and most of us just want companies we shop at to stop virtue signaling about divisive social, cultural and political issues. Companies need to stop dividing us by injecting wokeness! If you agree then share this video and tag other accounts so they share it too. If you support our movement to expose and remove woke policies in corporate America then you can help fund our growing team by subscribing to my X page for $5 a month or choose any amount once at You can become a whistleblower against a woke company at by emailing [email protected] or by going to P.S. What’s sad is there’s sooooo much more we couldn’t fit in this video. Maybe we’ll do a part 2 and/or drop key nuggets of information over the next few weeks… 🤔

Robby Starbuck

5,514,561 views • 1 year ago

Over the past two years, AI video models have been competing on realism, resolution, and duration. But no matter how impressive the results look, we remain passive viewers: we press play, watch the clip, and it ends. AlayaWorld Alaya Lab is attempting something fundamentally different. Instead of generating a fixed video, it generates a world that continues to unfold as you move through it. These three demos show the same journey toward a green village rendered in three distinct styles: photorealistic, oil painting, and line art. As the camera moves forward, the model continues generating the road, fences, trees, and distant village. This is not simply an existing video with different filters applied. The environment is generated continuously along the camera trajectory, allowing the scene to develop as the user explores it. AlayaWorld streams video at 720p and 24 FPS while supporting camera movement and viewpoint control. The real breakthrough is not just image quality. Once generation becomes fast enough to respond within an interactive loop, the user is no longer merely watching a video. They become a participant inside the generated world. The world can also respond to new instructions. During generation, users can introduce prompts that trigger spells, summon characters, create explosions, or transform the environment. Most video models follow an initial prompt and produce a predetermined clip. AlayaWorld can respond to changing intent while the world is still running, allowing subsequent events to evolve according to the user’s commands. Generating an attractive frame is relatively easy. Maintaining a coherent world over time is much harder. As a video model repeatedly predicts the next frame, small errors can accumulate until roads, buildings, and objects begin to distort or disappear. AlayaWorld combines spatial memory with compressed historical context, helping the model remember both where things are and what has already happened. This enables stable generation lasting more than one minute while improving consistency when the camera leaves an area and later returns. This may be the next step for AI video: not simply generating a longer movie, but generating a world that can be explored, changed, and interacted with. AlayaWorld is developed by Alaya Lab. The team is progressively releasing its inference code, training code, and datasets, with an online experience expected to launch near the end of the month. Project page:

Over the past two years, AI video models have been competing on realism, resolution, and duration. But no matter how impressive the results look, we remain passive viewers: we press play, watch the clip, and it ends. AlayaWorld Alaya Lab is attempting something fundamentally different. Instead of generating a fixed video, it generates a world that continues to unfold as you move through it. These three demos show the same journey toward a green village rendered in three distinct styles: photorealistic, oil painting, and line art. As the camera moves forward, the model continues generating the road, fences, trees, and distant village. This is not simply an existing video with different filters applied. The environment is generated continuously along the camera trajectory, allowing the scene to develop as the user explores it. AlayaWorld streams video at 720p and 24 FPS while supporting camera movement and viewpoint control. The real breakthrough is not just image quality. Once generation becomes fast enough to respond within an interactive loop, the user is no longer merely watching a video. They become a participant inside the generated world. The world can also respond to new instructions. During generation, users can introduce prompts that trigger spells, summon characters, create explosions, or transform the environment. Most video models follow an initial prompt and produce a predetermined clip. AlayaWorld can respond to changing intent while the world is still running, allowing subsequent events to evolve according to the user’s commands. Generating an attractive frame is relatively easy. Maintaining a coherent world over time is much harder. As a video model repeatedly predicts the next frame, small errors can accumulate until roads, buildings, and objects begin to distort or disappear. AlayaWorld combines spatial memory with compressed historical context, helping the model remember both where things are and what has already happened. This enables stable generation lasting more than one minute while improving consistency when the camera leaves an area and later returns. This may be the next step for AI video: not simply generating a longer movie, but generating a world that can be explored, changed, and interacted with. AlayaWorld is developed by Alaya Lab. The team is progressively releasing its inference code, training code, and datasets, with an online experience expected to launch near the end of the month. Project page:

Rachel🥥

18,218 views • 23 hours ago