Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Introducing KaleidošŸ’® from AI at Meta ā€” a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a...

liu_shikun's profile picture

Shikun Liu

1,273 subscribers

22,464 Aufrufe ā€¢ vor 11 Monaten ā€¢via X (Twitter)

LIVE

Anya Rossiā€¢ Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfĆ¼gbar

Kommentare vom Original-Post werden hier angezeigt

Ƅhnliche Videos

3D Gaussian Splatting for Real-Time Radiance Field Rendering paper page: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
0:18

3D Gaussian Splatting for Real-Time Radiance Field Rendering paper page: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

AK

634,067 Aufrufe ā€¢ vor 3 Jahren

Is Google taking initial steps to enhance Street View? For some reason, Street View seems stuck in technology that feels outdated. I wonder if we'll see such improvements on the product side. Also, note how much better it performs in all aspects compared to Zip-NeRF in their presented material. It offers more details and fewer artifacts. Great work! "LODGE: Level-of-Detail Large-Scale Gaussian Splatting with Efficient Rendering" Contributions: ā€¢ We propose a novel LOD representation for 3DGS which, unlike previous methods [27, 28, 17], does not recompute the list of used Gaussians at each frame. This allows for acceleration and compaction, enabling the rendering of large-scale scenes even on mobile devices. ā€¢ We design a strategy to automatically select optimal hyperparameters for splitting LODs, whereas most other methods require manual tuning of hyperparameters for each 3D scene. ā€¢ To further accelerate rendering, we split the scene into chunks and pre-compute sets of active Gaussians per chunk. ā€¢ Finally, we introduce a novel opacity interpolation scheme to produce visually pleasing rendering and eliminate artifacts when transitioning between chunks.
0:19

Is Google taking initial steps to enhance Street View? For some reason, Street View seems stuck in technology that feels outdated. I wonder if we'll see such improvements on the product side. Also, note how much better it performs in all aspects compared to Zip-NeRF in their presented material. It offers more details and fewer artifacts. Great work! "LODGE: Level-of-Detail Large-Scale Gaussian Splatting with Efficient Rendering" Contributions: ā€¢ We propose a novel LOD representation for 3DGS which, unlike previous methods [27, 28, 17], does not recompute the list of used Gaussians at each frame. This allows for acceleration and compaction, enabling the rendering of large-scale scenes even on mobile devices. ā€¢ We design a strategy to automatically select optimal hyperparameters for splitting LODs, whereas most other methods require manual tuning of hyperparameters for each 3D scene. ā€¢ To further accelerate rendering, we split the scene into chunks and pre-compute sets of active Gaussians per chunk. ā€¢ Finally, we introduce a novel opacity interpolation scheme to produce visually pleasing rendering and eliminate artifacts when transitioning between chunks.

MrNeRF

62,564 Aufrufe ā€¢ vor 1 Jahr

If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all by some denoising and gradient maths. I won't be surprised if Sora is trained on lots of synthetic data using Unreal Engine 5. It has to be! Let's breakdown the following video. Prompt: "Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee." - The simulator instantiates two exquisite 3D assets: pirate ships with different decorations. Sora has to solve text-to-3D implicitly in its latent space. - The 3D objects are consistently animated as they sail and avoid each other's paths. - Fluid dynamics of the coffee, even the foams that form around the ships. Fluid simulation is an entire sub-field of computer graphics, which traditionally requires very complex algorithms and equations. - Photorealism, almost like rendering with raytracing. - The simulator takes into account the small size of the cup compared to oceans, and applies tilt-shift photography to give a "minuscule" vibe. - The semantics of the scene does not exist in the real world, but the engine still implements the correct physical rules that we expect. Next up: add more modalities and conditioning, then we have a full data-driven UE that will replace all the hand-engineered graphics pipelines.
0:15

If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all by some denoising and gradient maths. I won't be surprised if Sora is trained on lots of synthetic data using Unreal Engine 5. It has to be! Let's breakdown the following video. Prompt: "Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee." - The simulator instantiates two exquisite 3D assets: pirate ships with different decorations. Sora has to solve text-to-3D implicitly in its latent space. - The 3D objects are consistently animated as they sail and avoid each other's paths. - Fluid dynamics of the coffee, even the foams that form around the ships. Fluid simulation is an entire sub-field of computer graphics, which traditionally requires very complex algorithms and equations. - Photorealism, almost like rendering with raytracing. - The simulator takes into account the small size of the cup compared to oceans, and applies tilt-shift photography to give a "minuscule" vibe. - The semantics of the scene does not exist in the real world, but the engine still implements the correct physical rules that we expect. Next up: add more modalities and conditioning, then we have a full data-driven UE that will replace all the hand-engineered graphics pipelines.

Jim Fan

6,183,399 Aufrufe ā€¢ vor 2 Jahren

The term "continual learning" has become overloaded if you see it as an ML problem. One classic thread is about memorization: regularization-based continual learning methods, such as EWC, MAS, and SI, estimate which parameters mattered for previous tasks and resist changing them too much. One modern thread is about adaptation: test-time training and inference-time learning methods, such as TTT, adapt part of the model on the incoming test stream before making predictions. These are sometimes discussed as separate threads. But in modern scalable architectures, I think they are better seen as complementary constraints: a model that learns quickly at test time also benefits from a mechanism for deciding what not to forget. In our #ECCV2026 paper, we study this in large-scale 4D reconstruction: how to build fast spatial memory that can adapt over long observation streams while reducing collapse and forgetting. Instead of using fully plastic test-time updates, we stabilize fast-weight adaptation with an elastic prior that balances adaptation and memory. Key ideas: - Elastic Test-Time Training: Fisher-weighted consolidation for fast-weight updates - EMA anchor weights that provide a moving reference for stability - Chunk-by-chunk inference for long 3D/4D observation streams We show that this scales across large 3D/4D pretraining settings, including both LRM-style and LVSM-style models, and improves reconstruction across benchmarks including Stereo4D, NVIDIA, and DL3DV-140. We release model checkpoints across different design choices: resolution, post-training curriculum, and whether the model uses an explicit 4DGS intermediate representation. - Homepage: - Paper: - Code: - Models: This work is co-led with Xueyang Yu, contributed by Haoyu Zhen Yuncong Yang, and advised by Michigan SLED Lab Chuang Gan.
0:14

The term "continual learning" has become overloaded if you see it as an ML problem. One classic thread is about memorization: regularization-based continual learning methods, such as EWC, MAS, and SI, estimate which parameters mattered for previous tasks and resist changing them too much. One modern thread is about adaptation: test-time training and inference-time learning methods, such as TTT, adapt part of the model on the incoming test stream before making predictions. These are sometimes discussed as separate threads. But in modern scalable architectures, I think they are better seen as complementary constraints: a model that learns quickly at test time also benefits from a mechanism for deciding what not to forget. In our #ECCV2026 paper, we study this in large-scale 4D reconstruction: how to build fast spatial memory that can adapt over long observation streams while reducing collapse and forgetting. Instead of using fully plastic test-time updates, we stabilize fast-weight adaptation with an elastic prior that balances adaptation and memory. Key ideas: - Elastic Test-Time Training: Fisher-weighted consolidation for fast-weight updates - EMA anchor weights that provide a moving reference for stability - Chunk-by-chunk inference for long 3D/4D observation streams We show that this scales across large 3D/4D pretraining settings, including both LRM-style and LVSM-style models, and improves reconstruction across benchmarks including Stereo4D, NVIDIA, and DL3DV-140. We release model checkpoints across different design choices: resolution, post-training curriculum, and whether the model uses an explicit 4DGS intermediate representation. - Homepage: - Paper: - Code: - Models: This work is co-led with Xueyang Yu, contributed by Haoyu Zhen Yuncong Yang, and advised by Michigan SLED Lab Chuang Gan.

Martin Ziqiao Ma

33,940 Aufrufe ā€¢ vor 2 Monaten

One of the most underrated ideas in AI filmmaking might be previsualization. Traditional film crews donā€™t wait until the final shoot to figure out where the camera should move or how actors should travel through a space. You plan those things first. A similar idea is starting to make sense for AI video. GPT-6 Astra can understand and map the 3D environment, while the Viewport provides a way to establish camera movement and character paths before rendering. Then manually bring the scene into CapCut PC and let Seedance 2.5 turn that structure into video. From there, CapCut PC can take the footage through the editing process. Maybe 3D isnā€™t just another way to generate AI video. Maybe it becomes the pre-production layer AI filmmaking has been missing. CapCut PC is an interesting place to experiment with that workflow. #CapCutPC #GPT6Astra #AIVideo #Seedance25
0:15

One of the most underrated ideas in AI filmmaking might be previsualization. Traditional film crews donā€™t wait until the final shoot to figure out where the camera should move or how actors should travel through a space. You plan those things first. A similar idea is starting to make sense for AI video. GPT-6 Astra can understand and map the 3D environment, while the Viewport provides a way to establish camera movement and character paths before rendering. Then manually bring the scene into CapCut PC and let Seedance 2.5 turn that structure into video. From there, CapCut PC can take the footage through the editing process. Maybe 3D isnā€™t just another way to generate AI video. Maybe it becomes the pre-production layer AI filmmaking has been missing. CapCut PC is an interesting place to experiment with that workflow. #CapCutPC #GPT6Astra #AIVideo #Seedance25

Alok Kumar

30,501 Aufrufe ā€¢ vor 9 Tagen

Trained on zero real-world data. Learned to walk, pick up boxes, and follow multi-step instructions... in the REAL world. ( šŸ“Œ Paper below) Researchers from Amazon FAR, Berkeley, Stanford, and CMU scanned real rooms with an iPhone, rebuilt them as 3D Gaussian Splatting scenes, then generated 48,000 synthetic trajectories of a Unitree G1 walking, grasping, and placing objects inside those virtual replicas. They rendered the robot's first-person camera view from each run and paired it with the matching language instruction and motion data. That's the dataset every humanoid team needs and nobody has: synced egocentric video + language + kinematics, at scale. Instead of collecting it in the real world, they manufactured it. They trained a vision-language-kinematics policy on that synthetic data alone, then deployed it on the physical G1 across five task types: navigation to a named object, lifting boxes of three different sizes with no per-size tuning, chained multi-step tasks, robustness to mid-task layout changes and flickering lights, and multi-minute long-horizon runs. No real-world fine-tuning at any point. Real-world interaction data has been the hard limit on humanoid learning... slow, expensive, and small. If scanning a room once and synthesizing thousands of labeled interactions holds up as a general recipe, that limit moves. Data stops being the bottleneck robotics teams have to solve for. šŸ“Œ Paper: Project: ā€”ā€”- Weekly robotics and AI insights. Subscribe free:
0:15

Trained on zero real-world data. Learned to walk, pick up boxes, and follow multi-step instructions... in the REAL world. ( šŸ“Œ Paper below) Researchers from Amazon FAR, Berkeley, Stanford, and CMU scanned real rooms with an iPhone, rebuilt them as 3D Gaussian Splatting scenes, then generated 48,000 synthetic trajectories of a Unitree G1 walking, grasping, and placing objects inside those virtual replicas. They rendered the robot's first-person camera view from each run and paired it with the matching language instruction and motion data. That's the dataset every humanoid team needs and nobody has: synced egocentric video + language + kinematics, at scale. Instead of collecting it in the real world, they manufactured it. They trained a vision-language-kinematics policy on that synthetic data alone, then deployed it on the physical G1 across five task types: navigation to a named object, lifting boxes of three different sizes with no per-size tuning, chained multi-step tasks, robustness to mid-task layout changes and flickering lights, and multi-minute long-horizon runs. No real-world fine-tuning at any point. Real-world interaction data has been the hard limit on humanoid learning... slow, expensive, and small. If scanning a room once and synthesizing thousands of labeled interactions holds up as a general recipe, that limit moves. Data stops being the bottleneck robotics teams have to solve for. šŸ“Œ Paper: Project: ā€”ā€”- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

12,950 Aufrufe ā€¢ vor 1 Monat

Check out this Stereo4D paper from Google DeepMind. It's a pretty clever approach to a persistent problem in computer vision -- getting good training data for how things move in 3D. The key insight is using VR180 videos -- those stereo fisheye videos we launched back in 2017 for YouTubeVR. It was always clear that structured stereo datasets would be valuable for computer vision -- and we launched some powerful VR tools with it back in 2017 (link below). But what's the game changer now in 2024 is the scale -- they're providing 110K high quality clips :-) That's the kind of massive, real-world AI dataset that was just a dream back then! They're using it to train this model called DynaDUSt3R that can predict both 3D structure and motion from video frames. Which means it tracks how objects move between frames while simultaneously reconstructing their 3D shape. And given we're dealing with real stereoscopic content, results are notably better than synthetic data, giving you a faithful rendition of the real-world with a diverse set of subject matter. It's one of those through lines when tackling a timeless mission like mapping the world or spatial computing -- VR content created for immersion becoming the foundation for teaching machines to understand how the world moves. Sometimes innovation chains together in unexpected ways! Links to projects belowā›“ļø
0:12

Check out this Stereo4D paper from Google DeepMind. It's a pretty clever approach to a persistent problem in computer vision -- getting good training data for how things move in 3D. The key insight is using VR180 videos -- those stereo fisheye videos we launched back in 2017 for YouTubeVR. It was always clear that structured stereo datasets would be valuable for computer vision -- and we launched some powerful VR tools with it back in 2017 (link below). But what's the game changer now in 2024 is the scale -- they're providing 110K high quality clips :-) That's the kind of massive, real-world AI dataset that was just a dream back then! They're using it to train this model called DynaDUSt3R that can predict both 3D structure and motion from video frames. Which means it tracks how objects move between frames while simultaneously reconstructing their 3D shape. And given we're dealing with real stereoscopic content, results are notably better than synthetic data, giving you a faithful rendition of the real-world with a diverse set of subject matter. It's one of those through lines when tackling a timeless mission like mapping the world or spatial computing -- VR content created for immersion becoming the foundation for teaching machines to understand how the world moves. Sometimes innovation chains together in unexpected ways! Links to projects belowā›“ļø

Bilawal Sidhu

68,515 Aufrufe ā€¢ vor 1 Jahr

A Closer Look at Alchemist AI v3.5šŸ”Ž In v1, one agent handled everything. Youā€™d enter a prompt like ā€œmake a snake gameā€ and get a single block of codeā€”one input, one output. v2 introduced multiple agents with specialized roles: prompt refinement, frontend, backend, and review. It brought more structure, but the process was still linear. Each step produced a single output. With v3, a single agent could handle multiple tasks. It generated HTML, CSS, JavaScript, searched for assets, and moreā€”all in one go, while maintaining full context. Now in v3.5, we combine both models. Multiple agents work in parallel, each capable of producing multiple outputs at once. The prompt compiler adapts based on what you're building. A 3D game? It prioritizes game engines, rendering, and mechanics. A website or 2D app? It shifts focus to relevant frameworks and tools. The tool handler can call several services simultaneously, feeding context-aware data into the code generator for real-time execution. From single-output generation to parallel, adaptive workflows. In the next breakdown, weā€™ll dive into the architecture behind our proprietary engine in v3.5.
0:11

A Closer Look at Alchemist AI v3.5šŸ”Ž In v1, one agent handled everything. Youā€™d enter a prompt like ā€œmake a snake gameā€ and get a single block of codeā€”one input, one output. v2 introduced multiple agents with specialized roles: prompt refinement, frontend, backend, and review. It brought more structure, but the process was still linear. Each step produced a single output. With v3, a single agent could handle multiple tasks. It generated HTML, CSS, JavaScript, searched for assets, and moreā€”all in one go, while maintaining full context. Now in v3.5, we combine both models. Multiple agents work in parallel, each capable of producing multiple outputs at once. The prompt compiler adapts based on what you're building. A 3D game? It prioritizes game engines, rendering, and mechanics. A website or 2D app? It shifts focus to relevant frameworks and tools. The tool handler can call several services simultaneously, feeding context-aware data into the code generator for real-time execution. From single-output generation to parallel, adaptive workflows. In the next breakdown, weā€™ll dive into the architecture behind our proprietary engine in v3.5.

ALCHEMIST AI šŸ”®

12,689 Aufrufe ā€¢ vor 1 Jahr

Two weeks ago I fixed one of my teeth with algorithms I wrote a couple of years ago! I got hooked by 3D scanning when I started to work for a software shop in Zurich that was programming 3D computational geometry algorithms for denture scanning to produce crowns (and more). Back then, a typical reconstruction pipeline was like: scan the patientā€™s teeth using an intraoral scanner, reconstruct the surface mesh, design the restoration digitally, and finally mill the crown out of ceramic. We were working mostly with point clouds and meshes, but it wasnā€™t just math, it was craftsmanship translated into a digital process. Every micron mattered. You could literally see how a good algorithm meant a better fit in someoneā€™s mouth. Gaussian Splatting isnā€™t about surface reconstruction, itā€™s about appearance reconstruction. It doesnā€™t care about explicit topology, it captures how light interacts with the scene. In a sense, itā€™s the opposite philosophy of the dental world: instead of modeling what the object is, it models how the object looks. 3D Gaussian Splatting enables applications like training self driving cars, teaching robots to understand their environment, creating virtual worlds, or monitoring real sites. It represents scenes as millions of small Gaussians rendered in real time without the need for meshes or textures. Coming from a world where precision geometry was everything, this shift felt natural. Itā€™s still about reconstruction, but with a different goal: not manufacturing a perfect object, but reproducing how the world actually looks. Two weeks ago I got my first dental crown, made with the same software, reconstruction algorithms, and Swiss precision I once helped develop. I havenā€™t worked there in two years, but sitting in that chair and seeing the process from the other side was a proud moment. It reminded me why I love this field.
0:18

Two weeks ago I fixed one of my teeth with algorithms I wrote a couple of years ago! I got hooked by 3D scanning when I started to work for a software shop in Zurich that was programming 3D computational geometry algorithms for denture scanning to produce crowns (and more). Back then, a typical reconstruction pipeline was like: scan the patientā€™s teeth using an intraoral scanner, reconstruct the surface mesh, design the restoration digitally, and finally mill the crown out of ceramic. We were working mostly with point clouds and meshes, but it wasnā€™t just math, it was craftsmanship translated into a digital process. Every micron mattered. You could literally see how a good algorithm meant a better fit in someoneā€™s mouth. Gaussian Splatting isnā€™t about surface reconstruction, itā€™s about appearance reconstruction. It doesnā€™t care about explicit topology, it captures how light interacts with the scene. In a sense, itā€™s the opposite philosophy of the dental world: instead of modeling what the object is, it models how the object looks. 3D Gaussian Splatting enables applications like training self driving cars, teaching robots to understand their environment, creating virtual worlds, or monitoring real sites. It represents scenes as millions of small Gaussians rendered in real time without the need for meshes or textures. Coming from a world where precision geometry was everything, this shift felt natural. Itā€™s still about reconstruction, but with a different goal: not manufacturing a perfect object, but reproducing how the world actually looks. Two weeks ago I got my first dental crown, made with the same software, reconstruction algorithms, and Swiss precision I once helped develop. I havenā€™t worked there in two years, but sitting in that chair and seeing the process from the other side was a proud moment. It reminded me why I love this field.

MrNeRF

290,257 Aufrufe ā€¢ vor 10 Monaten

Robots can now reconstruct 3D scenes in real time from a single RGB camera. [šŸ“ Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like Ļ€Ā³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: ā€¢ Up to 30 FPS ā€¢ 10 to 15x speedup ā€¢ Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes ā€¢ Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperialā€™s Dyson Robotics Lab and many others who contributed to this! šŸ“ Save projects page + paper for later: Video: ā€”ā€”- if it matters in AI or Robotics you'll read it here first:
0:18

Robots can now reconstruct 3D scenes in real time from a single RGB camera. [šŸ“ Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like Ļ€Ā³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: ā€¢ Up to 30 FPS ā€¢ 10 to 15x speedup ā€¢ Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes ā€¢ Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperialā€™s Dyson Robotics Lab and many others who contributed to this! šŸ“ Save projects page + paper for later: Video: ā€”ā€”- if it matters in AI or Robotics you'll read it here first:

Ilir Aliu

53,992 Aufrufe ā€¢ vor 5 Monaten

As a graphics engine coder I think when you look at a flickering bug like this one in the video below itā€™s not immediately obvious what is going on. The key here is observation - to study this flickering/bugged render carefully - what do we see? Firstly for me it was very obvious that nearly all of the scene shadows were flashing on and off - but (but!) there was a secondary issue where some buildings and parts of the sky were also flashing purple. Hmmmm. Interesting. I initially thought then this might be two separate bugs - but because the sky purple element could only based on full screen post fx and not 3D rendering I looked at this first with a few GPU captures to step through all our post processing to find the rendering stage which made these pixels turn purple: When I did this I found the colour 3D texture LUT grading that makes our different biomes have unique colour palettes was going very wrong - colours near 0 or 1 were wrapping and making the purple elements that we see in the said sky and base parts. The only way this could happen was if the texture was corrupt (which it was not) or if the 3D texture sampling was wrapping and not clamped as intended. That was the Eureka moment - because if the post fx had the wrong texture sampler then the disappearing shadows which also require an exact texture sampler for comparing depth might be also wrong because of the same kind of texture sampling issue! So with this idea that the engine was using the wrong texture samplers, but only in very high draw call scenes like the big base here I the looked at some engine limits and found the bug very quickly - a circular dx12 descriptor buffer for samplers running out over multiple frames, reusing the wrong data for new scenes inflight. Hence the flickering, as the GPU randomly got wrong samplers for some post textures or shadow depth. Easy to fix with triple limits for future expansion and also adding an assert/debug spam in case this limit is ever reached again - QA testers would see this message and report if they ever saw a flicker with this style of bug. My bug and my bad from 2017 porting NMS to DX12 without foreseeing how massively complex bases and our game would grow.
0:14

As a graphics engine coder I think when you look at a flickering bug like this one in the video below itā€™s not immediately obvious what is going on. The key here is observation - to study this flickering/bugged render carefully - what do we see? Firstly for me it was very obvious that nearly all of the scene shadows were flashing on and off - but (but!) there was a secondary issue where some buildings and parts of the sky were also flashing purple. Hmmmm. Interesting. I initially thought then this might be two separate bugs - but because the sky purple element could only based on full screen post fx and not 3D rendering I looked at this first with a few GPU captures to step through all our post processing to find the rendering stage which made these pixels turn purple: When I did this I found the colour 3D texture LUT grading that makes our different biomes have unique colour palettes was going very wrong - colours near 0 or 1 were wrapping and making the purple elements that we see in the said sky and base parts. The only way this could happen was if the texture was corrupt (which it was not) or if the 3D texture sampling was wrapping and not clamped as intended. That was the Eureka moment - because if the post fx had the wrong texture sampler then the disappearing shadows which also require an exact texture sampler for comparing depth might be also wrong because of the same kind of texture sampling issue! So with this idea that the engine was using the wrong texture samplers, but only in very high draw call scenes like the big base here I the looked at some engine limits and found the bug very quickly - a circular dx12 descriptor buffer for samplers running out over multiple frames, reusing the wrong data for new scenes inflight. Hence the flickering, as the GPU randomly got wrong samplers for some post textures or shadow depth. Easy to fix with triple limits for future expansion and also adding an assert/debug spam in case this limit is ever reached again - QA testers would see this message and report if they ever saw a flicker with this style of bug. My bug and my bad from 2017 porting NMS to DX12 without foreseeing how massively complex bases and our game would grow.

Martin Griffiths

72,855 Aufrufe ā€¢ vor 1 Jahr

This week is already so hot. šŸ”„ Massive release from Decart : Lucy 2.0 a World Editing Model running at 1080p, 30FPS in realtime. This is truly exciting, the era of real-time generative reality is here. We are moving from watching AI video to living inside AI video. A breakthrough model capable of transforming the visual world in real-time. Moving beyond offline rendering, Lucy 2.0 delivers high-fidelity 1080p video generation with near-zero latency. Lucy 2.0 literally "redraws" the entire world pixel-by-pixel, while you are watching it. e.g. If you want to be an anime character, it doesn't just put a mask on you. It turns your skin into anime skin, your hair into anime hair, and the lighting in your room into anime lighting. Lucy 2.0 is also trained to stop the generated video from slowly falling apart over time, so the same stream can run much longer without faces and details drifting. So why is this a "Massive Deal"? Traditional AI video-generation model takes a prompt, you wait 10ā€“20 minutes, and the computer "bakes" a video for you. You couldn't touch it or change it while it was happening. But Lucy 2.0 works like a mirror. It happens in real-time (30 frames per second). There is no waiting. You move your hand, the AI character moves its hand instantly. The craziest part isn't the visuals; it's the physics. Usually, AI hallucinations are glitchyā€”hands merge into faces, walls melt. Lucy 2.0 understands how the world works without being told. It knows that if you take off a helmet, there is hair underneath. It knows that if you splash water, droplets fly. It learned "physics" just by watching millions of videos. The physical behavior you see emerges from learned visual dynamics, not from engineered geometry or explicit physics engines. Their official technical report explicitly states that the model does not use traditional 3D engines, depth maps, or wireframes. It is a "pure diffusion model."
0:14

This week is already so hot. šŸ”„ Massive release from Decart : Lucy 2.0 a World Editing Model running at 1080p, 30FPS in realtime. This is truly exciting, the era of real-time generative reality is here. We are moving from watching AI video to living inside AI video. A breakthrough model capable of transforming the visual world in real-time. Moving beyond offline rendering, Lucy 2.0 delivers high-fidelity 1080p video generation with near-zero latency. Lucy 2.0 literally "redraws" the entire world pixel-by-pixel, while you are watching it. e.g. If you want to be an anime character, it doesn't just put a mask on you. It turns your skin into anime skin, your hair into anime hair, and the lighting in your room into anime lighting. Lucy 2.0 is also trained to stop the generated video from slowly falling apart over time, so the same stream can run much longer without faces and details drifting. So why is this a "Massive Deal"? Traditional AI video-generation model takes a prompt, you wait 10ā€“20 minutes, and the computer "bakes" a video for you. You couldn't touch it or change it while it was happening. But Lucy 2.0 works like a mirror. It happens in real-time (30 frames per second). There is no waiting. You move your hand, the AI character moves its hand instantly. The craziest part isn't the visuals; it's the physics. Usually, AI hallucinations are glitchyā€”hands merge into faces, walls melt. Lucy 2.0 understands how the world works without being told. It knows that if you take off a helmet, there is hair underneath. It knows that if you splash water, droplets fly. It learned "physics" just by watching millions of videos. The physical behavior you see emerges from learned visual dynamics, not from engineered geometry or explicit physics engines. Their official technical report explicitly states that the model does not use traditional 3D engines, depth maps, or wireframes. It is a "pure diffusion model."

Rohan Paul

12,761 Aufrufe ā€¢ vor 7 Monaten

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to ā€œspace-time diffusionā€ for video generation! Now what the heck does that mean exactly?! šŸŒā³ ā†’ TL;DR it utilizes a ā€œSpace-Time UNetā€ architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this ā€œcompressed space-time representationā€ to generate the full clip at once, it's far more temporally consistent. ā†’ Another benefit of generating the full video at once is that you can ā€œdirectā€ the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities ā€“ from video inpainting to image-to-video and beyond. ā†’ New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. ā†’ Could this end up inside YouTube? Yā€™all know iā€™m obsessed with blending reality and imagination ā€“ so itā€™s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! šŸ¤žšŸ½ Links: šŸ”—Paper: šŸ”—Project:
0:13

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to ā€œspace-time diffusionā€ for video generation! Now what the heck does that mean exactly?! šŸŒā³ ā†’ TL;DR it utilizes a ā€œSpace-Time UNetā€ architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this ā€œcompressed space-time representationā€ to generate the full clip at once, it's far more temporally consistent. ā†’ Another benefit of generating the full video at once is that you can ā€œdirectā€ the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities ā€“ from video inpainting to image-to-video and beyond. ā†’ New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. ā†’ Could this end up inside YouTube? Yā€™all know iā€™m obsessed with blending reality and imagination ā€“ so itā€™s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! šŸ¤žšŸ½ Links: šŸ”—Paper: šŸ”—Project:

Bilawal Sidhu

44,822 Aufrufe ā€¢ vor 2 Jahren

What if you could choreograph an entire action scene before generating a single frame? šŸ„‹šŸŽ¬ Thatā€™s what I wanted to test with Pippitofficialā€™s 3D Director Studio + Seedance 2.5. I created a scene with two girls demonstrating a series of karate moves and used the 3D workspace to figure out the sequence first. The interesting part is that I wasnā€™t relying on a giant prompt to explain every movement. I could place both characters in the scene, set up the environment, choose props from 150+ available options, and then map out how the performance should unfold. One move at a time. Step in ā†’ stance ā†’ punch ā†’ block ā†’ kick ā†’ reset ā†’ next move. The timeline makes it possible to arrange actions and control their timing, while movement trajectories help define where the characters should go. And then thereā€™s the camera. Instead of hoping the generated video finds the right angle, I could plan the camera movement around the choreography and decide how the sequence should be captured. Thatā€™s the part I found most useful. You can think through the scene visually before asking AI to generate the final result. Characters. Movement. Timing. Camera. All planned first. Then Seedance 2.5 turns that setup into the actual video. For me, this feels like a shift from simply prompting AI to actually directing the shot. Try Pippit 3D Director Studio: #3DDirectorStudio #PippitAI #Seedance25 #PippitPartner
0:10

What if you could choreograph an entire action scene before generating a single frame? šŸ„‹šŸŽ¬ Thatā€™s what I wanted to test with Pippitofficialā€™s 3D Director Studio + Seedance 2.5. I created a scene with two girls demonstrating a series of karate moves and used the 3D workspace to figure out the sequence first. The interesting part is that I wasnā€™t relying on a giant prompt to explain every movement. I could place both characters in the scene, set up the environment, choose props from 150+ available options, and then map out how the performance should unfold. One move at a time. Step in ā†’ stance ā†’ punch ā†’ block ā†’ kick ā†’ reset ā†’ next move. The timeline makes it possible to arrange actions and control their timing, while movement trajectories help define where the characters should go. And then thereā€™s the camera. Instead of hoping the generated video finds the right angle, I could plan the camera movement around the choreography and decide how the sequence should be captured. Thatā€™s the part I found most useful. You can think through the scene visually before asking AI to generate the final result. Characters. Movement. Timing. Camera. All planned first. Then Seedance 2.5 turns that setup into the actual video. For me, this feels like a shift from simply prompting AI to actually directing the shot. Try Pippit 3D Director Studio: #3DDirectorStudio #PippitAI #Seedance25 #PippitPartner

Shabnam Parveen

51,713 Aufrufe ā€¢ vor 9 Tagen

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft paper page: Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces an instruction-tuned Video Pretraining (VPT) model for Minecraft called STEVE-1, demonstrating that the unCLIP approach, utilized in DALL-E 2, is also effective for creating instruction-following sequential decision-making agents. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, bypassing the need for costly human text annotations. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 costs just $60 to train and can follow a wide range of short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.
0:14

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft paper page: Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces an instruction-tuned Video Pretraining (VPT) model for Minecraft called STEVE-1, demonstrating that the unCLIP approach, utilized in DALL-E 2, is also effective for creating instruction-following sequential decision-making agents. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, bypassing the need for costly human text annotations. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 costs just $60 to train and can follow a wide range of short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

AK

144,811 Aufrufe ā€¢ vor 3 Jahren

"Pros wonā€™t use generative AI, and when the bubble pops, nobody will ever talk about it again." No. Thatā€™s delusional. 1/ Generative AI is already being used professionally at the level of big studios like Disney ($1B to OpenAI), and thereā€™s zero doubt that studios like Industrial Light & Magic, Netflix, Hollywood VFX experts, etc. are already experimenting with it too. Or do you think theyā€™re idiots? Theyā€™re not idiots at all. They have the experience and, more importantly, the DISTRIBUTION POWER. The point is: someone with taste, judgment, and storytelling experience, basically from their living room, will have access to (almost, or not even almost) the same capability as the big guys, because the pure "making stuff" skills have been commoditized, and the new way to create is just NATURAL LANGUAGE. What hasnā€™t been commoditized is good taste, the ability to create great stories that move people, and the ability to get them in front of people. So in the end, what wins is story quality and distribution. Having good taste, making a name for yourself, and owning strong IP (Marvel, etc.) will still matter. Thatā€™ll be true right up until AI is genuinely opinionated and can create by itself: if it comes to that, with zero human direction, stuff as good as (or better than) the very best human experts today, and on top of that, interactive in real time... Because yeah: thereā€™s nothing in this universe that actually prevents that from happening. BUT WEā€™RE NOT THERE. For now, generative AI is a tool that needs direction and taste to make anything decent. And I hope it stays that way for a long time, because otherwise thatā€™s going to be a brutal hit to humanityā€™s ego. 2/ On the "bubble": you have to distinguish between a stock valuation bubble (possible, I actually believe it) vs a bubble like some people imagine where it "pops" and we never hear about AI again. That obviously makes no sense given how insanely useful it is. It can only grow, and itā€™s going to grow fast, regardless of any stock market drawdowns (the internet kept growing even when valuations got nuked in 2000). Either way, the near future is going to be extremely interesting.
0:15

"Pros wonā€™t use generative AI, and when the bubble pops, nobody will ever talk about it again." No. Thatā€™s delusional. 1/ Generative AI is already being used professionally at the level of big studios like Disney ($1B to OpenAI), and thereā€™s zero doubt that studios like Industrial Light & Magic, Netflix, Hollywood VFX experts, etc. are already experimenting with it too. Or do you think theyā€™re idiots? Theyā€™re not idiots at all. They have the experience and, more importantly, the DISTRIBUTION POWER. The point is: someone with taste, judgment, and storytelling experience, basically from their living room, will have access to (almost, or not even almost) the same capability as the big guys, because the pure "making stuff" skills have been commoditized, and the new way to create is just NATURAL LANGUAGE. What hasnā€™t been commoditized is good taste, the ability to create great stories that move people, and the ability to get them in front of people. So in the end, what wins is story quality and distribution. Having good taste, making a name for yourself, and owning strong IP (Marvel, etc.) will still matter. Thatā€™ll be true right up until AI is genuinely opinionated and can create by itself: if it comes to that, with zero human direction, stuff as good as (or better than) the very best human experts today, and on top of that, interactive in real time... Because yeah: thereā€™s nothing in this universe that actually prevents that from happening. BUT WEā€™RE NOT THERE. For now, generative AI is a tool that needs direction and taste to make anything decent. And I hope it stays that way for a long time, because otherwise thatā€™s going to be a brutal hit to humanityā€™s ego. 2/ On the "bubble": you have to distinguish between a stock valuation bubble (possible, I actually believe it) vs a bubble like some people imagine where it "pops" and we never hear about AI again. That obviously makes no sense given how insanely useful it is. It can only grow, and itā€™s going to grow fast, regardless of any stock market drawdowns (the internet kept growing even when valuations got nuked in 2000). Either way, the near future is going to be extremely interesting.

Javi Lopez ā›©ļø

75,277 Aufrufe ā€¢ vor 7 Monaten