正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Genesis supports simulating various types of physical phenomena. We developed from scratch a unified physics engine that integrates various SOTA physics solvers (MPM, SPH, FEM, Rigid Body, PBD, etc.), supporting simulation of a wide range of materials: rigid body, articulated body, Cloth, Liquid, Smoke, Deformables, Thin-shell materials, Elastic/Plastic Body,... show more

Zhou Xian

15,868 subscribers

101,451 次观看 • 1 年前 •via X (Twitter)

教育艺术科学技术

Anya Rossi• Live Now

Private livecam show

12 条评论

Zhou Xian 的头像

Zhou Xian1 年前

Everything you love about generative models — now powered by real physics! Announcing the Genesis project — after a 24-month large-scale research collaboration involving over 20 research labs — a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications. Genesis's physics engine is developed in pure Python, while being 10-80x faster than existing GPU-accelerated stacks like Isaac Gym and MJX. It delivers a simulation speed ~430,000 faster than in real-time, and takes only 26 seconds to train a robotic locomotion policy transferrable to the real world on a single RTX4090 (see tutorial: The Genesis physics engine and simulation platform is fully open source at We'll gradually roll out access to our generative framework in the near future. Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism. We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds, together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI and other applications. Open Source Code: Project webpage: Documentation: 1/n

Zhou Xian 的头像

Zhou Xian1 年前

Nvidia brought GPU acceleration to robotic simulation, speeding up simulation speed by more than one order of magnitude compared to CPU-based simulation. This brought numerous amazing robotic skills to life by leveraging large-scale GPU-parallelized simulation. Genesis pushes up this speed by another order of magnitude. Note that the speed improvement is achieved with no compromise in simulation accuracy. 2/n

Zhou Xian 的头像

Zhou Xian1 年前

Genesis is the first-ever platform providing comprehensive support for soft muscles and soft robot and their interaction with rigid robots. Genesis also ships with a URDF-like soft-robot configuration system. 4/n

Zhou Xian 的头像

Zhou Xian1 年前

Genesis's generative framework supports generating 3D and fully interactive scenes for training robotic skills 5/n

Zhou Xian 的头像

Zhou Xian1 年前

Our generative agent autonomously proposes robotic tasks, design environments, write reward functions, and ultimately leading to automated generation of robotic policies. 6/n

Zhou Xian 的头像

Zhou Xian1 年前

Genesis's generative framework supports data generation beyond robotics, such as character motion: 7/n

Zhou Xian 的头像

Zhou Xian1 年前

Genesis's GPU parallellized IK solver is able to solve IK for 10,000 Franka arms simultaneously, under 2ms: 8/n

Zhou Xian 的头像

Zhou Xian1 年前

We support native non-convex collision handling: 9/n

Zhou Xian 的头像

Zhou Xian1 年前

Genesis supports a physically accurate tactile sensing simulation module: (Will be integrated into the main branch in a future release soon) 10/n

Zhou Xian 的头像

Zhou Xian1 年前

Finally, a cute interactive physical Tetris game made with Genesis :) Thanks to all the amazing collaborators who together made everything possible over the last two years! There's no space here to @ every single one, but a huge kudos to the whole Genesis team! We welcome everyone from the open-source community to come join us and build Genesis with us together! 11/11

TheRuminator 的头像

TheRuminator1 年前

How well does it handle energy conservation in a gas simulation under gravity?

SOAI.world 的头像

SOAI.world1 年前

Time for a Colab with @Blender

相关视频

Everything you love about generative models — now powered by real physics! Announcing the Genesis project — after a 24-month large-scale research collaboration involving over 20 research labs — a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications. Genesis's physics engine is developed in pure Python, while being 10-80x faster than existing GPU-accelerated stacks like Isaac Gym and MJX. It delivers a simulation speed ~430,000 faster than in real-time, and takes only 26 seconds to train a robotic locomotion policy transferrable to the real world on a single RTX4090 (see tutorial: The Genesis physics engine and simulation platform is fully open source at We'll gradually roll out access to our generative framework in the near future. Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism. We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds, together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI and other applications. Open Source Code: Project webpage: Documentation: 1/n

Everything you love about generative models — now powered by real physics! Announcing the Genesis project — after a 24-month large-scale research collaboration involving over 20 research labs — a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications. Genesis's physics engine is developed in pure Python, while being 10-80x faster than existing GPU-accelerated stacks like Isaac Gym and MJX. It delivers a simulation speed ~430,000 faster than in real-time, and takes only 26 seconds to train a robotic locomotion policy transferrable to the real world on a single RTX4090 (see tutorial: The Genesis physics engine and simulation platform is fully open source at We'll gradually roll out access to our generative framework in the near future. Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism. We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds, together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI and other applications. Open Source Code: Project webpage: Documentation: 1/n

Zhou Xian

3,816,945 次观看 • 1 年前

We are back again :) After three weeks of quiet building. Introducing Genesis World 1.0, our latest simulation platform, the second release in our full-stack suite. Open-sourced. Robotics is still bottlenecked by the 1× speed of the physical world. Every model, checkpoint, and data recipe eventually needs to be tested on physical hardware, slowly, expensively, and with limited coverage. One hour in reality can become 100 days in simulation. That is how robotics model iteration moves from a wall-clock bottleneck to a compute problem. To make this work, simulation has to be both fast and trustworthy. Over the past year, we rebuilt the entire stack: a GPU-accelerated cross-platform compiler, penetration-free multi-physics contact solvers, unified rigid and deformable physics, and a photo-realistic renderer purpose-built for physical AI applications. We built Nyx, a high-performance path-traced rendering engine for robotics application. Genesis World 1.0 achieves near realtime performance with our latest development for penetration-free IPC solver, supporting various types of deformables beyond rigid bodies. It supports contact-rich, dexterous manipulation simulation across different embodiments: unitree, sharpa, wuji, genesis hand and various types of grippers. Under the hood is Quadrants, our effort in pushing forward cross-platform GPU-accelerated computation. Quadrants started as a fork of Taichi, and we rebuilt most of the critical parts for optimizing simulation workloads, giving 10x faster launch time and up to 4.6x runtime performance compared to the initial Genesis release. Together, they bring us to an unprecedentedly low sim-to-real gap, enabling zero-shot real-to-sim model evaluation and much faster iteration of GENE. All available today. Genesis World 1.0: Quadrants: Nyx:

We are back again :) After three weeks of quiet building. Introducing Genesis World 1.0, our latest simulation platform, the second release in our full-stack suite. Open-sourced. Robotics is still bottlenecked by the 1× speed of the physical world. Every model, checkpoint, and data recipe eventually needs to be tested on physical hardware, slowly, expensively, and with limited coverage. One hour in reality can become 100 days in simulation. That is how robotics model iteration moves from a wall-clock bottleneck to a compute problem. To make this work, simulation has to be both fast and trustworthy. Over the past year, we rebuilt the entire stack: a GPU-accelerated cross-platform compiler, penetration-free multi-physics contact solvers, unified rigid and deformable physics, and a photo-realistic renderer purpose-built for physical AI applications. We built Nyx, a high-performance path-traced rendering engine for robotics application. Genesis World 1.0 achieves near realtime performance with our latest development for penetration-free IPC solver, supporting various types of deformables beyond rigid bodies. It supports contact-rich, dexterous manipulation simulation across different embodiments: unitree, sharpa, wuji, genesis hand and various types of grippers. Under the hood is Quadrants, our effort in pushing forward cross-platform GPU-accelerated computation. Quadrants started as a fork of Taichi, and we rebuilt most of the critical parts for optimizing simulation workloads, giving 10x faster launch time and up to 4.6x runtime performance compared to the initial Genesis release. Together, they bring us to an unprecedentedly low sim-to-real gap, enabling zero-shot real-to-sim model evaluation and much faster iteration of GENE. All available today. Genesis World 1.0: Quadrants: Nyx:

Genesis AI

306,834 次观看 • 1 个月前

MEET BLUE: NVIDIA’S REAL-LIFE WALL-E Blue is an AI robot that moves, reacts, and talks like something straight out of a Pixar film. Blue runs on 2 NVIDIA computers and uses a physics engine for ultra-realistic tactile feedback, rigid and soft body simulation, all in real time. This isn’t a concept, this is how NVIDIA plans to train robots for the real world. And yes, it casually stole the show. Source: entrepreneurialogy

MEET BLUE: NVIDIA’S REAL-LIFE WALL-E Blue is an AI robot that moves, reacts, and talks like something straight out of a Pixar film. Blue runs on 2 NVIDIA computers and uses a physics engine for ultra-realistic tactile feedback, rigid and soft body simulation, all in real time. This isn’t a concept, this is how NVIDIA plans to train robots for the real world. And yes, it casually stole the show. Source: entrepreneurialogy

Mario Nawfal

38,388 次观看 • 11 个月前

Introducing ✨RigidFormer: Learning Rigid Dynamics with Transformers - our attempt to scale learning-based physical dynamics with Transformers. RigidFormer learns rigid dynamics with Transformers. It is a mesh-free, object-centric Transformer for multi-object rigid-body contact dynamics from point clouds. Learning physics with purely neural simulators, without relying on traditional physics engines, is an important and widely studied problem. Prior SOTA methods often use graph neural networks for accuracy and generalization, but still struggle with efficient, high-fidelity simulation at scale. RigidFormer uses only point inputs, matches or outperforms mesh-based baselines on standard benchmarks, runs much faster, generalizes across point resolutions and datasets, and scales to 200+ objects. We also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. RigidFormer is mesh-free: it does not require mesh connectivity, SDFs, or vertex-level message passing, making it well-suited for point-cloud observations and scalable simulation. This architecture can also be adapted to learn soft-body dynamics by replacing the rigid-body module (differentiable Kabsch alignment). 🎬See our video for more details. Many thanks to my amazing collaborators: Minghao Guo Minghao Guo, Haixu Wu Haixu Wu 吴海旭, Doug Roble, Tuur Stuyck Tuur Stuyck, and Wojciech Matusik Wojciech Matusik. Project page: Paper:

Introducing ✨RigidFormer: Learning Rigid Dynamics with Transformers - our attempt to scale learning-based physical dynamics with Transformers. RigidFormer learns rigid dynamics with Transformers. It is a mesh-free, object-centric Transformer for multi-object rigid-body contact dynamics from point clouds. Learning physics with purely neural simulators, without relying on traditional physics engines, is an important and widely studied problem. Prior SOTA methods often use graph neural networks for accuracy and generalization, but still struggle with efficient, high-fidelity simulation at scale. RigidFormer uses only point inputs, matches or outperforms mesh-based baselines on standard benchmarks, runs much faster, generalizes across point resolutions and datasets, and scales to 200+ objects. We also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. RigidFormer is mesh-free: it does not require mesh connectivity, SDFs, or vertex-level message passing, making it well-suited for point-cloud observations and scalable simulation. This architecture can also be adapted to learn soft-body dynamics by replacing the rigid-body module (differentiable Kabsch alignment). 🎬See our video for more details. Many thanks to my amazing collaborators: Minghao Guo Minghao Guo, Haixu Wu Haixu Wu 吴海旭, Doug Roble, Tuur Stuyck Tuur Stuyck, and Wojciech Matusik Wojciech Matusik. Project page: Paper:

Zhiyang (Frank) Dou

571,726 次观看 • 2 个月前

That's sick! 🤯 Genesis AI simulates robots playing yo-yo! 🪀 Genesis AI just open-sourced Genesis World 1.0, and it might be one of the most important infrastructure releases in robotics this year. Robotics is still bottlenecked by the 1× speed of the physical world. Every model needs to be tested on real hardware, slowly, expensively, with limited coverage. Genesis World 1.0 from Genesis AI flips that equation: One hour in reality becomes 100 days in simulation. That turns a wall-clock bottleneck into a compute problem. And compute problems are solvable. The technical stack they rebuilt from scratch is serious: → GPU-accelerated cross-platform compiler via Quadrants, 10x faster launch time and up to 4.6x runtime vs the initial Genesis release → Penetration-free multi-physics contact solvers, the thing that makes simulation actually trustworthy → Unified rigid AND deformable physics in a single engine → Nyx, a high-performance path-traced rendering engine purpose-built for physical AI The sim-to-real gap has historically been the graveyard of robotics research. Policies that work beautifully in simulation fall apart on real hardware. Genesis World 1.0 is a direct attack on that problem. And it's fully open-source. The companies that master simulation infrastructure will train better robots faster than anyone else. Find it here: Genesis World 1.0: Quadrants: Nyx: Theophile Gervet, Zhou Xian congrats! 👏🏼 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

That's sick! 🤯 Genesis AI simulates robots playing yo-yo! 🪀 Genesis AI just open-sourced Genesis World 1.0, and it might be one of the most important infrastructure releases in robotics this year. Robotics is still bottlenecked by the 1× speed of the physical world. Every model needs to be tested on real hardware, slowly, expensively, with limited coverage. Genesis World 1.0 from Genesis AI flips that equation: One hour in reality becomes 100 days in simulation. That turns a wall-clock bottleneck into a compute problem. And compute problems are solvable. The technical stack they rebuilt from scratch is serious: → GPU-accelerated cross-platform compiler via Quadrants, 10x faster launch time and up to 4.6x runtime vs the initial Genesis release → Penetration-free multi-physics contact solvers, the thing that makes simulation actually trustworthy → Unified rigid AND deformable physics in a single engine → Nyx, a high-performance path-traced rendering engine purpose-built for physical AI The sim-to-real gap has historically been the graveyard of robotics research. Policies that work beautifully in simulation fall apart on real hardware. Genesis World 1.0 is a direct attack on that problem. And it's fully open-source. The companies that master simulation infrastructure will train better robots faster than anyone else. Find it here: Genesis World 1.0: Quadrants: Nyx: Theophile Gervet, Zhou Xian congrats! 👏🏼 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

36,767 次观看 • 1 个月前

Robotics has a massive, silent bottleneck. It isn’t just data collection—it’s the brutal 1x speed of the physical world. Genesis AI Genesis AI just unveiled Genesis World 1.0, and they are attempting to turn the notorious Sim2Real gap into a pure compute problem. Evaluating a robotics foundation model across edge cases usually means hundreds of hours of physical lab testing. With Genesis World 1.0, what traditionally takes nearly a week of continuous, real-world operation is being compressed into 30 minutes in simulation. What makes this different from just dropping a robot model into an off-the-shelf game engine? 1️⃣ Nyx Renderer: A custom, real-time path-traced engine rendering noise-free 1080p frames in under 4ms. Game engines use rasterization tricks that confuse AI; Nyx uses physically accurate multi-bounce lighting so the model's "eyes" see exactly what real sensors see. 2️⃣ Quadrants Compiler: A custom Python-to-GPU compiler to run heavily parallelized multi-physics simulations (rigid bodies, fluids, deformables) natively across architectures. 3️⃣ Evaluation First: They aren't rushing to train on synthetic data. They are using this purely for closed-loop evaluation to perfect the physics first, currently claiming an impressive 89% correlation with real-world hardware tests. If the industry can accurately evaluate models in simulation without the physical world bottleneck, humanoid development stops moving at wall-clock time and starts scaling with compute.

Robotics has a massive, silent bottleneck. It isn’t just data collection—it’s the brutal 1x speed of the physical world. Genesis AI Genesis AI just unveiled Genesis World 1.0, and they are attempting to turn the notorious Sim2Real gap into a pure compute problem. Evaluating a robotics foundation model across edge cases usually means hundreds of hours of physical lab testing. With Genesis World 1.0, what traditionally takes nearly a week of continuous, real-world operation is being compressed into 30 minutes in simulation. What makes this different from just dropping a robot model into an off-the-shelf game engine? 1️⃣ Nyx Renderer: A custom, real-time path-traced engine rendering noise-free 1080p frames in under 4ms. Game engines use rasterization tricks that confuse AI; Nyx uses physically accurate multi-bounce lighting so the model's "eyes" see exactly what real sensors see. 2️⃣ Quadrants Compiler: A custom Python-to-GPU compiler to run heavily parallelized multi-physics simulations (rigid bodies, fluids, deformables) natively across architectures. 3️⃣ Evaluation First: They aren't rushing to train on synthetic data. They are using this purely for closed-loop evaluation to perfect the physics first, currently claiming an impressive 89% correlation with real-world hardware tests. If the industry can accurately evaluate models in simulation without the physical world bottleneck, humanoid development stops moving at wall-clock time and starts scaling with compute.

Humanoids daily

17,240 次观看 • 1 个月前

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually slow them down so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

Jim Fan

566,544 次观看 • 1 年前

Why the character movement in my custom game engine felt janky and how I fixed it. In a game engine, most often, a character moves using the physics engine. Meaning, the player is not just a coordinate in space but a physical body. It has velocity, it handles collisions, and it interacts with the world. Now, as you might know, physics engines need stability. If you run them at variable framerates, things start breaking. Objects phase through walls or fly off into space because the math becomes unpredictable. This is why most game engines lock their physics loop to a 60Hz fixed rate. But here’s the problem: If you have a high-end system, you don't want to limit it at 60 FPS. That's a waste of good hardware. Now, that said, if the GPU is rendering at 144 FPS but the player's position (physics driven) only updates 60 times a second, it creates a micro-stutter that ruins the "smooth" feel of the game. A good way to fix this is to treat the character as two separate things: 1. The Physics Body (Invisible part): This is the "real" character. It lives in the 60Hz physics world, it moves the player and handles collisions. 2. The Visual Model and Camera (Visible part): This is what the player actually sees. It doesn't care about collisions, its only job is to look nice and smooth at whatever framerate the GPU is pushing. Once you have this separation, you can use interpolation to keep them in sync. Every time the physics clock ticks, you save the previous position of the invisible body before moving it to the new one. Between those ticks, calculate how far we are between the last physics update and the next one. By using this to drive the visible parts of the game, the stutters disappear. The physics loop stays fixed behind the scenes, while the visuals slide smoothly between the snapshots. Example: - Right after a tick: blend_weight= 0.0 (The visual model stays at the old physics position). - Halfway to the next: blend_weight= 0.5 (The visual model slides to the middle point). - Just before the next: blend_weight= 0.9 (The visual model is almost at the new physics position). Pro-Tip A critical mistake I made initially, and one many devs make, is parenting the camera and visible parts directly to the player body. If you do this, the camera inherits the discrete 60Hz physics movement by default. In that setup, interpolation won't work because the camera is "stuck" to the physics clock. For this fix to work you must decouple the camera and visuals from the body and move them separately. Player movement processing in Detis Engine: - fixed_process: Physics runs at 60Hz. Handles collisions and raw movement. - process: Variable rate. Mainly used for player input caching in the player case. - late_process: Variable rate. Handles interpolated camera movement after physics and everything else is done being processed. - render. Submits the final interpolated transforms to the GPU. The test environment in the video is running on an old 2070-based laptop. Hopefully the video compression won't introduce any stutter... I’m sharing this in hopes it helps a fellow dev. Cheers.

Why the character movement in my custom game engine felt janky and how I fixed it. In a game engine, most often, a character moves using the physics engine. Meaning, the player is not just a coordinate in space but a physical body. It has velocity, it handles collisions, and it interacts with the world. Now, as you might know, physics engines need stability. If you run them at variable framerates, things start breaking. Objects phase through walls or fly off into space because the math becomes unpredictable. This is why most game engines lock their physics loop to a 60Hz fixed rate. But here’s the problem: If you have a high-end system, you don't want to limit it at 60 FPS. That's a waste of good hardware. Now, that said, if the GPU is rendering at 144 FPS but the player's position (physics driven) only updates 60 times a second, it creates a micro-stutter that ruins the "smooth" feel of the game. A good way to fix this is to treat the character as two separate things: 1. The Physics Body (Invisible part): This is the "real" character. It lives in the 60Hz physics world, it moves the player and handles collisions. 2. The Visual Model and Camera (Visible part): This is what the player actually sees. It doesn't care about collisions, its only job is to look nice and smooth at whatever framerate the GPU is pushing. Once you have this separation, you can use interpolation to keep them in sync. Every time the physics clock ticks, you save the previous position of the invisible body before moving it to the new one. Between those ticks, calculate how far we are between the last physics update and the next one. By using this to drive the visible parts of the game, the stutters disappear. The physics loop stays fixed behind the scenes, while the visuals slide smoothly between the snapshots. Example: - Right after a tick: blend_weight= 0.0 (The visual model stays at the old physics position). - Halfway to the next: blend_weight= 0.5 (The visual model slides to the middle point). - Just before the next: blend_weight= 0.9 (The visual model is almost at the new physics position). Pro-Tip A critical mistake I made initially, and one many devs make, is parenting the camera and visible parts directly to the player body. If you do this, the camera inherits the discrete 60Hz physics movement by default. In that setup, interpolation won't work because the camera is "stuck" to the physics clock. For this fix to work you must decouple the camera and visuals from the body and move them separately. Player movement processing in Detis Engine: - fixed_process: Physics runs at 60Hz. Handles collisions and raw movement. - process: Variable rate. Mainly used for player input caching in the player case. - late_process: Variable rate. Handles interpolated camera movement after physics and everything else is done being processed. - render. Submits the final interpolated transforms to the GPU. The test environment in the video is running on an old 2070-based laptop. Hopefully the video compression won't introduce any stutter... I’m sharing this in hopes it helps a fellow dev. Cheers.

Ioannis Koukourakis

48,518 次观看 • 6 个月前

Physics AI 🌊 Jensen just unveiled the NVIDIA Isaac GR00T Reference Humanoid Robot at GTC Taipei--the world’s first open-source humanoid robot reference design Built on Jetson Thor edge computing and the Isaac GR00T open platform, it features a Unitree H2 Plus body with Sharpa five-finger dexterous hands and 75 degrees of freedom. Even before Physical AI truly explodes, NVIDIA has built a complete end-to-end full-stack infrastructure: >chips (Jetson Thor + Blackwell), >synthetic data (GR00T-Dreams), >world models (Cosmos), >foundation models (GR00T N-series VLA), >high-fidelity simulation platforms (Isaac Sim + Isaac Lab + Omniverse). Dozens of humanoid robot companies ,including Agility Robotics, Aglie robotics,Boston Dynamics, Figure, 1X, NEURA Robotics, XPENG Robotics, and more are already running large-scale daily simulation training on this platform, rapidly iterating perception, reasoning, and actions in virtual environments, speeding up the jump from lab prototypes to real factory deployment. By providing an open reference design and full-stack tools, NVIDIA has dramatically lowered the barrier, letting every robotics team join the Physical AI wave with low cost and high efficiency.

Physics AI 🌊 Jensen just unveiled the NVIDIA Isaac GR00T Reference Humanoid Robot at GTC Taipei--the world’s first open-source humanoid robot reference design Built on Jetson Thor edge computing and the Isaac GR00T open platform, it features a Unitree H2 Plus body with Sharpa five-finger dexterous hands and 75 degrees of freedom. Even before Physical AI truly explodes, NVIDIA has built a complete end-to-end full-stack infrastructure: >chips (Jetson Thor + Blackwell), >synthetic data (GR00T-Dreams), >world models (Cosmos), >foundation models (GR00T N-series VLA), >high-fidelity simulation platforms (Isaac Sim + Isaac Lab + Omniverse). Dozens of humanoid robot companies ,including Agility Robotics, Aglie robotics,Boston Dynamics, Figure, 1X, NEURA Robotics, XPENG Robotics, and more are already running large-scale daily simulation training on this platform, rapidly iterating perception, reasoning, and actions in virtual environments, speeding up the jump from lab prototypes to real factory deployment. By providing an open reference design and full-stack tools, NVIDIA has dramatically lowered the barrier, letting every robotics team join the Physical AI wave with low cost and high efficiency.

CyberRobo

17,092 次观看 • 1 个月前

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 次观看 • 3 年前

Gemini 2.5 Flash demolishes my Galton Board test, I could not get 4omini, 4o mini high, or 03 to produce this. I found that Gemini 2.5 Flash understands my intents almost instantly, code produced is tight and neat. The prompt is a merging of various steps. It took me 5 steps to achieve this in Gemini 2.5 Flash, I gave up on OpenAI models after about half an hour. My iterations are obviously not exact. But people can test with this one prompt for more objective comparison. Please try this prompt on your end to confirm: -------------------------------------------------- Create a self-contained HTML file for a Galton board simulation using client-side JavaScript and a 2D physics engine (like Matter.js, included via CDN). The simulation should be rendered on an HTML5 canvas and meet the following criteria: 1. **Single File:** All necessary HTML, CSS, and JavaScript code must be within this single `.html` file. 2. **Canvas Size:** The overall simulation area (canvas) should be reasonably sized to fit on a standard screen without requiring extensive scrolling or zooming (e.g., around 500x700 pixels). 3. **Physics:** Utilize a 2D rigid body physics engine for realistic ball-peg and ball-wall interactions. 4. **Obstacles (Pegs):** Create static, circular pegs arranged in full-width horizontal rows extending across the usable width of the board (not just a triangle). The pegs should be small enough and spaced appropriately for balls to navigate and bounce between them. 5. **Containment:** * Include static, sufficiently thick side walls and a ground at the bottom to contain the balls within the board. * Implement *physical* static dividers between the collection bins at the bottom. These dividers must be thick enough to prevent balls from passing through them, ensuring accurate accumulation in each bin. 6. **Ball Dropping:** Balls should be dropped from a controlled, narrow area near the horizontal center at the top of the board to ensure they enter the peg field consistently. 7. **Bins:** The collection area at the bottom should be divided into distinct bins by the physical dividers. The height of the bins should be sufficient to clearly visualize the accumulation of balls. 8. **Visualization:** Use a high-contrast color scheme to clearly distinguish between elements. Specifically, use yellow for the structural elements (walls, top guides, physical bin dividers, ground), a contrasting color (like red) for the pegs, and a highly contrasting color (like dark grey or black) for the balls. 9. **Demonstration:** The simulation should visually demonstrate the formation of the normal (or binomial) distribution as multiple balls fall through the pegs and collect in the bins. Ensure the physics parameters (restitution, friction, density) and ball drop rate are tuned for a smooth and clear demonstration of the distribution. #OpenAI Sam Altman Greg Brockman AshutoshShrivastava Aidan McLaughlin

Gemini 2.5 Flash demolishes my Galton Board test, I could not get 4omini, 4o mini high, or 03 to produce this. I found that Gemini 2.5 Flash understands my intents almost instantly, code produced is tight and neat. The prompt is a merging of various steps. It took me 5 steps to achieve this in Gemini 2.5 Flash, I gave up on OpenAI models after about half an hour. My iterations are obviously not exact. But people can test with this one prompt for more objective comparison. Please try this prompt on your end to confirm: -------------------------------------------------- Create a self-contained HTML file for a Galton board simulation using client-side JavaScript and a 2D physics engine (like Matter.js, included via CDN). The simulation should be rendered on an HTML5 canvas and meet the following criteria: 1. Single File: All necessary HTML, CSS, and JavaScript code must be within this single `.html` file. 2. Canvas Size: The overall simulation area (canvas) should be reasonably sized to fit on a standard screen without requiring extensive scrolling or zooming (e.g., around 500x700 pixels). 3. Physics: Utilize a 2D rigid body physics engine for realistic ball-peg and ball-wall interactions. 4. Obstacles (Pegs): Create static, circular pegs arranged in full-width horizontal rows extending across the usable width of the board (not just a triangle). The pegs should be small enough and spaced appropriately for balls to navigate and bounce between them. 5. Containment: * Include static, sufficiently thick side walls and a ground at the bottom to contain the balls within the board. * Implement physical static dividers between the collection bins at the bottom. These dividers must be thick enough to prevent balls from passing through them, ensuring accurate accumulation in each bin. 6. Ball Dropping: Balls should be dropped from a controlled, narrow area near the horizontal center at the top of the board to ensure they enter the peg field consistently. 7. Bins: The collection area at the bottom should be divided into distinct bins by the physical dividers. The height of the bins should be sufficient to clearly visualize the accumulation of balls. 8. Visualization: Use a high-contrast color scheme to clearly distinguish between elements. Specifically, use yellow for the structural elements (walls, top guides, physical bin dividers, ground), a contrasting color (like red) for the pegs, and a highly contrasting color (like dark grey or black) for the balls. 9. Demonstration: The simulation should visually demonstrate the formation of the normal (or binomial) distribution as multiple balls fall through the pegs and collect in the bins. Ensure the physics parameters (restitution, friction, density) and ball drop rate are tuned for a smooth and clear demonstration of the distribution. #OpenAI Sam Altman Greg Brockman AshutoshShrivastava Aidan McLaughlin

RameshR

247,923 次观看 • 1 年前

Dolce & Gabbana (D&G) fashion show in summer of 2019; Ancient Greece inspired collection. As the collection was inspired by Ancient Greece, it was only fitting that the clothes were displayed for the first time at the Valley of the Temples in Agrigento, Sicily - Italy. The noble temple, reflecting the tangerine yellow of the setting sun, stood like a rock of ages – taking Sicily back thousands of years to the artistic culture of ancient Greece. The Valley of the Temples in Agrigento belonged – for one night only – to Dolce & Gabbana, whose Alta Moda collection was shown on this historic site. Civilization of ancient Greece flourished from 8th Century BC to 600 CE. It was located in southeastern Europe along the coast of Mediterranean Sea and included modern-day countries of Greece and parts of Türkiye, Italy and Bulgaria. Today, ancient Greeks are known for their contributions to philosophy, politics, art, architecture, and science. They were also known for their unique clothing styles, inspired by their environment and cultural traditions. In movies about ancient Greece, characters portraying Greek citizens, soldiers, or mythological figures often wear these clothing styles. Characteristics of traditional Greek fashion included long, flowing garments made of lightweight materials such as linen or wool. People draped these garments over the body in a manner that allowed for a range of movement and comfort. Both men and women wore tunics that were knee-length or longer. They could be worn as a standalone garment or layered over other clothing. People fastened their tunics at the shoulder with a pin or brooch and often wore a belt to cinch the waist. Overall, the clothing of ancient Greece was functional and practical but also imbued with cultural and artistic significance. Intricate patterns, designs, and embroidery often adorned garments, and they were often made of luxurious materials such as silk or gold-threaded fabric. A tunic was a garment that both men and women would commonly wear in ancient Greece. It was a sleeveless piece of clothing, knee-length or longer, that they wore over the upper body and fastened at the shoulder with a pin or brooch. Ancient Greeks typically made tunics of lightweight materials such as linen or wool and wore them as either a standalone garment or layered over other clothing. The tunic was a versatile garment that could be worn in a variety of settings and was suitable for both formal and casual occasions. It was also practical in ancient Greek dress, allowing for a range of movement and comfort. Ancient Greeks created their clothing using various techniques and tools, such as weaving fabrics on looms, sewing together pieces of fabric, and using decorative techniques like embroidery, beadwork, and the use of gold and silver thread. In ancient Greece, men and women wore tunics that they crafted out of lightweight materials like linen or wool and fastened at the shoulder with a pin or brooch. Men and women both wore a chiton, a long sleeveless garment, as well as cloaks called himatia made of heavier materials like wool or animal skin to keep warm. Men wore various headgear such as hats, helmet-like caps, and headbands, while women wore head coverings like veils, headbands, and hair ornaments. Ancient Greeks also wore garments like the peplos and the chlamys. The peplos was a long rectangular piece of fabric that women would often wear on formal occasions, fastened at the shoulders with a pair of fibulae (brooches), and made of luxurious materials such as silk with intricate patterns or designs. A chlamys was a short cloak-like garment worn by men over a tunic or chiton. It was made of lightweight materials like wool or linen, fastened at the shoulder with a pin or brooch, and often used for travel, outdoor activities, or in battle. 🎥© antiktarih (IG) #archaeohistories

Dolce & Gabbana (D&G) fashion show in summer of 2019; Ancient Greece inspired collection. As the collection was inspired by Ancient Greece, it was only fitting that the clothes were displayed for the first time at the Valley of the Temples in Agrigento, Sicily - Italy. The noble temple, reflecting the tangerine yellow of the setting sun, stood like a rock of ages – taking Sicily back thousands of years to the artistic culture of ancient Greece. The Valley of the Temples in Agrigento belonged – for one night only – to Dolce & Gabbana, whose Alta Moda collection was shown on this historic site. Civilization of ancient Greece flourished from 8th Century BC to 600 CE. It was located in southeastern Europe along the coast of Mediterranean Sea and included modern-day countries of Greece and parts of Türkiye, Italy and Bulgaria. Today, ancient Greeks are known for their contributions to philosophy, politics, art, architecture, and science. They were also known for their unique clothing styles, inspired by their environment and cultural traditions. In movies about ancient Greece, characters portraying Greek citizens, soldiers, or mythological figures often wear these clothing styles. Characteristics of traditional Greek fashion included long, flowing garments made of lightweight materials such as linen or wool. People draped these garments over the body in a manner that allowed for a range of movement and comfort. Both men and women wore tunics that were knee-length or longer. They could be worn as a standalone garment or layered over other clothing. People fastened their tunics at the shoulder with a pin or brooch and often wore a belt to cinch the waist. Overall, the clothing of ancient Greece was functional and practical but also imbued with cultural and artistic significance. Intricate patterns, designs, and embroidery often adorned garments, and they were often made of luxurious materials such as silk or gold-threaded fabric. A tunic was a garment that both men and women would commonly wear in ancient Greece. It was a sleeveless piece of clothing, knee-length or longer, that they wore over the upper body and fastened at the shoulder with a pin or brooch. Ancient Greeks typically made tunics of lightweight materials such as linen or wool and wore them as either a standalone garment or layered over other clothing. The tunic was a versatile garment that could be worn in a variety of settings and was suitable for both formal and casual occasions. It was also practical in ancient Greek dress, allowing for a range of movement and comfort. Ancient Greeks created their clothing using various techniques and tools, such as weaving fabrics on looms, sewing together pieces of fabric, and using decorative techniques like embroidery, beadwork, and the use of gold and silver thread. In ancient Greece, men and women wore tunics that they crafted out of lightweight materials like linen or wool and fastened at the shoulder with a pin or brooch. Men and women both wore a chiton, a long sleeveless garment, as well as cloaks called himatia made of heavier materials like wool or animal skin to keep warm. Men wore various headgear such as hats, helmet-like caps, and headbands, while women wore head coverings like veils, headbands, and hair ornaments. Ancient Greeks also wore garments like the peplos and the chlamys. The peplos was a long rectangular piece of fabric that women would often wear on formal occasions, fastened at the shoulders with a pair of fibulae (brooches), and made of luxurious materials such as silk with intricate patterns or designs. A chlamys was a short cloak-like garment worn by men over a tunic or chiton. It was made of lightweight materials like wool or linen, fastened at the shoulder with a pin or brooch, and often used for travel, outdoor activities, or in battle. 🎥© antiktarih (IG) #archaeohistories

Archaeo - Histories

144,107 次观看 • 2 年前

🦿Xpeng showed a humanoid robot called IRON whose movement looked so human that the team literally cut it open on stage to prove it is a machine. IRON uses a bionic body with a flexible spine, synthetic muscles, and soft skin so joints and torso can twist smoothly like a person. The system has 82 degrees of freedom in total with 22 in each hand for fine finger control. Compute runs on 3 custom AI chips rated at 2,250 TOPS (Tera Operations Per Second), which is far above typical laptop neural accelerators, so it can handle vision and motion planning on the robot. The AI stack focuses on turning camera input directly into body movement without routing through text, which reduces lag and makes the gait look natural. Xpeng staged the cut-open demo at AI Day in Guangzhou this week, addressing rumors that a performer was inside by exposing internal actuators, wiring, and cooling. Company materials also mention a large physical-world model and a multi-brain control setup for dialogue, perception, and locomotion, hinting at a path from stage demos to service work. Production is targeted for 2026, so near-term tasks will be limited, but the hardware shows a serious step toward human-scale manipulation.

🦿Xpeng showed a humanoid robot called IRON whose movement looked so human that the team literally cut it open on stage to prove it is a machine. IRON uses a bionic body with a flexible spine, synthetic muscles, and soft skin so joints and torso can twist smoothly like a person. The system has 82 degrees of freedom in total with 22 in each hand for fine finger control. Compute runs on 3 custom AI chips rated at 2,250 TOPS (Tera Operations Per Second), which is far above typical laptop neural accelerators, so it can handle vision and motion planning on the robot. The AI stack focuses on turning camera input directly into body movement without routing through text, which reduces lag and makes the gait look natural. Xpeng staged the cut-open demo at AI Day in Guangzhou this week, addressing rumors that a performer was inside by exposing internal actuators, wiring, and cooling. Company materials also mention a large physical-world model and a multi-brain control setup for dialogue, perception, and locomotion, hinting at a path from stage demos to service work. Production is targeted for 2026, so near-term tasks will be limited, but the hardware shows a serious step toward human-scale manipulation.

Rohan Paul

3,802,402 次观看 • 8 个月前

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the motion of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,380 次观看 • 1 年前

On an average, a household in urban India generates approx. 10 gms. of MLP waste every single day. These are typically the covers of various food products ranging from tetra-pack cartons, masala powder covers, milk powder, potato chips - various other FMCG consumables. MLP (Multi - layered packaging material) are NON recyclable plastic waste. Since it cannot be recycled, this waste has NO purchase value at the scrap shops. Even rag pickers ignore it. Every other waste like thick-thin plastic, paper, metal, electric , electronic etc. have resale value ... But not this bloke. Thus the MLP inevitably ends up on landfills, contaminating the soil and eventually, during flows into a water body on its journey to the ocean. There it hurts the marine life , breaks into micro plastics under the UV light rays from the Sun. In a classical RSVP, it gets into the clouds alongwith evaporated water and returns back to earth in rain water. Through water, all of us and the animals and birds, now have micro plastics in our bodies. Many diseases like cancer, etc have been attributed to this non-organic invader inside our bodies. Now, we cannot wish away the MLPs. They do a fantastic job in keeping the potato chips fresh and crispy for 6 months. A boon to the supply-chain of FMCG products. They are available as a raw material to the food processing. FMCG industries in abundance. Also, an environment friendly commercially viable alternate material in large quantity is not seen in the distant horizon. So, along with 2 other professionals we launched our start-up to address this problem. Ui- Unified Intelligence Pvt Ltd is the execution wing of our venture. Vaspar foundation is our not-for-profit foundation that lodges the innovation center. Here we are always working on developing multiple new products and solutions aimed at creating large scale employment. At the factory MLP waste covers brought in from dry waste collection centres are shredded, washed and then loaded on a sheet making unit that presses out 8'x4' sized sheets of a variety of thickness. As the sheets cool down, we get a light weight thermoset sheet that are extremely strong and have fantastic mechanical and thermal properties. The MLP waste are from food grade plastic and thus have almost zero VOC. Also, the sheets are wonderful heat insulators. We sandwich 3 sheets of these between honeycomb structures made of the same material to create structurally integrated construction grade panels. These 100 mm thick panels are then covered on all sides with 8mm cement boards to form a skin. These are ready to paint surface, are standard construction materials with fire resistance rating of 2 hours. These panels become the walls and roof of our building. The floor is a 6" concrete layer. We build homes, terrace living space, dormitories , class rooms, auditoriums, compound walls, public toilets etc. with these panels. At the rate of 10 grams of MLP packets that we throw out as waste from our homes per day, in a single year each home generates 3,650 grams (10 grams x 365 days) So if an apartment has 10 dwelling units, in a year the residents discharge 36.50 Kgs of MLP waste into the environment Last week we installed this tiny security cabin in our Apartments in Bangalore. There are 10 flats here . This 5'x5'x8' cabin was constructed with our panels in under 6 hours. The weight of the MLP waste consumed in the panels for the cabin is 500kgs. THAT is around 14 years of #MLP waste discharged together by all the VIRGO residents - is now stored in the walls of the cabin as our panels. The cabin has a life of 50 years. We 10 families , in our apartment thus atoned for our 'plastic-guilt' to an extent. For inquiries, E Mail paul@ui.ltd

On an average, a household in urban India generates approx. 10 gms. of MLP waste every single day. These are typically the covers of various food products ranging from tetra-pack cartons, masala powder covers, milk powder, potato chips - various other FMCG consumables. MLP (Multi - layered packaging material) are NON recyclable plastic waste. Since it cannot be recycled, this waste has NO purchase value at the scrap shops. Even rag pickers ignore it. Every other waste like thick-thin plastic, paper, metal, electric , electronic etc. have resale value ... But not this bloke. Thus the MLP inevitably ends up on landfills, contaminating the soil and eventually, during flows into a water body on its journey to the ocean. There it hurts the marine life , breaks into micro plastics under the UV light rays from the Sun. In a classical RSVP, it gets into the clouds alongwith evaporated water and returns back to earth in rain water. Through water, all of us and the animals and birds, now have micro plastics in our bodies. Many diseases like cancer, etc have been attributed to this non-organic invader inside our bodies. Now, we cannot wish away the MLPs. They do a fantastic job in keeping the potato chips fresh and crispy for 6 months. A boon to the supply-chain of FMCG products. They are available as a raw material to the food processing. FMCG industries in abundance. Also, an environment friendly commercially viable alternate material in large quantity is not seen in the distant horizon. So, along with 2 other professionals we launched our start-up to address this problem. Ui- Unified Intelligence Pvt Ltd is the execution wing of our venture. Vaspar foundation is our not-for-profit foundation that lodges the innovation center. Here we are always working on developing multiple new products and solutions aimed at creating large scale employment. At the factory MLP waste covers brought in from dry waste collection centres are shredded, washed and then loaded on a sheet making unit that presses out 8'x4' sized sheets of a variety of thickness. As the sheets cool down, we get a light weight thermoset sheet that are extremely strong and have fantastic mechanical and thermal properties. The MLP waste are from food grade plastic and thus have almost zero VOC. Also, the sheets are wonderful heat insulators. We sandwich 3 sheets of these between honeycomb structures made of the same material to create structurally integrated construction grade panels. These 100 mm thick panels are then covered on all sides with 8mm cement boards to form a skin. These are ready to paint surface, are standard construction materials with fire resistance rating of 2 hours. These panels become the walls and roof of our building. The floor is a 6" concrete layer. We build homes, terrace living space, dormitories , class rooms, auditoriums, compound walls, public toilets etc. with these panels. At the rate of 10 grams of MLP packets that we throw out as waste from our homes per day, in a single year each home generates 3,650 grams (10 grams x 365 days) So if an apartment has 10 dwelling units, in a year the residents discharge 36.50 Kgs of MLP waste into the environment Last week we installed this tiny security cabin in our Apartments in Bangalore. There are 10 flats here . This 5'x5'x8' cabin was constructed with our panels in under 6 hours. The weight of the MLP waste consumed in the panels for the cabin is 500kgs. THAT is around 14 years of #MLP waste discharged together by all the VIRGO residents - is now stored in the walls of the cabin as our panels. The cabin has a life of 50 years. We 10 families , in our apartment thus atoned for our 'plastic-guilt' to an extent. For inquiries, E Mail [email protected]

Paul Koshy

45,018 次观看 • 1 年前

The Genie 3 release is a perfect moment to have a discussion about the future of 3D But first it would be nice to make the terminology more clear, specifically: What is a “spatial representation” Implicit vs Explicit Generalization vs Specialization Reconstruction vs Generation Production vs Execution Let’s start: For me, a spatial representation is just a way to describe a thing in the physical world The core property that makes it useful is consistency You can enforce consistency explicitly via rendering equations, geometric constraints, and physics Or implicitly, purely through training data Then, your representation parameters can be explicit, like points, gaussians, triangles, voxels, etc. Or implicit, weights or latent vectors Parameters alone are not the representation. It’s a combination of the parameters, the process that produces them, and the way you materialize them through a function (physics-based rendering, simulation, neural network, etc.) Generalization means you take data from multiple scene observations, and then produce a map from desired input to representation parameters Specialization means you take single-scene observations and directly fit a function parameters to describe thar scene Many representations can serve both of the approaches, as long as you keep them differentiable Both of the above can be used for reconstruction, where the main goal is to explain observations through a lens of physics (hard constraint) On the other hand, generation needs generalization, and its task is to produce statistically plausible results that could be conditioned on observations (soft constraint) Both tasks are not solved yet and they can complement each other in various ways Yet another important aspect is the difference between production and execution Production = process of going from inputs to parameters Execution = process of going from parameters to result It’s important to separate these, because most usecases require fast execution to be viable which is severely constrained by the hardware So, are *world models* like Genie an important step forward? Yes Do they make other representations obsolete? Maybe some of them - but there are tons of economically valuable tasks that won’t be solved by it, at least in any observable future

The Genie 3 release is a perfect moment to have a discussion about the future of 3D But first it would be nice to make the terminology more clear, specifically: What is a “spatial representation” Implicit vs Explicit Generalization vs Specialization Reconstruction vs Generation Production vs Execution Let’s start: For me, a spatial representation is just a way to describe a thing in the physical world The core property that makes it useful is consistency You can enforce consistency explicitly via rendering equations, geometric constraints, and physics Or implicitly, purely through training data Then, your representation parameters can be explicit, like points, gaussians, triangles, voxels, etc. Or implicit, weights or latent vectors Parameters alone are not the representation. It’s a combination of the parameters, the process that produces them, and the way you materialize them through a function (physics-based rendering, simulation, neural network, etc.) Generalization means you take data from multiple scene observations, and then produce a map from desired input to representation parameters Specialization means you take single-scene observations and directly fit a function parameters to describe thar scene Many representations can serve both of the approaches, as long as you keep them differentiable Both of the above can be used for reconstruction, where the main goal is to explain observations through a lens of physics (hard constraint) On the other hand, generation needs generalization, and its task is to produce statistically plausible results that could be conditioned on observations (soft constraint) Both tasks are not solved yet and they can complement each other in various ways Yet another important aspect is the difference between production and execution Production = process of going from inputs to parameters Execution = process of going from parameters to result It’s important to separate these, because most usecases require fast execution to be viable which is severely constrained by the hardware So, are world models like Genie an important step forward? Yes Do they make other representations obsolete? Maybe some of them - but there are tons of economically valuable tasks that won’t be solved by it, at least in any observable future

Lucky Iyinbor

13,959 次观看 • 5 个月前

Special thanks to Google DeepMind for inviting me to try out Genie 3. I'm excited to share my thoughts on this early research prototype and also some of my live recordings below: I spent the whole day playing with the system and when it works, it is truly mind blowing🤯. It is the first neural game engine / world model I have tried that generalizes so well and has long term world consistency. Here’s a couple of examples from my live recording and some thoughts on what it means for the future of gaming, robotics, digital experiences and ASI. Where it shines: - Truly general-purpose and quick startup time. Works exceptionally well for gaming environments but also generalizes to other industrial and real-world scenarios. - It learns physics. Although there are systematic failures even for rigid body physics, it was clear to me that it can learn game engine and non-rigid physics without an underlying engine (and in limit learn from game engines via training data). - It works exceptionally well for stylized environments with characters walking around. This will have implications for concept artists, level designers and game devs. - It is way more fun than video models, indicating that there are high retention consumer experiences waiting to be built with this in the future - Photorealistic walk throughs and drone shots work exceptionally well - Global illumination and lighting works surprisingly well - Visual memory is quite powerful and the same objects approximately remain coherent under occlusion and longer time horizons Open Problems: - Physics is still hard and there are obvious failure cases when I tried the classical intuitive physics experiments from psychology (tower of blocks). - Social and multi-agent interactions are tricky to handle. 1vs1 combat games do not work - Long instruction following and simple combinatorial game logic fails (e.g. collect some points / keys etc, go to the door, unlock and so on) - Action space is limited - It is far from being a real game engines and has a long way to go but this is a clear glimpse into the future. The Future: - It is impressive enough for me to have strong conviction that this is going to disrupt the gaming industry. It is super early days and there are a lot of failures but the writing is on the wall. Lots of challenging scientific, engineering and scaling problems to be solved but it is going to happen in the next 5 years. - This is the final piece before we get full AGI and now I think we are well on our way to truly solve it once something like this is scaled up. In many ways it is more ASI than AGI but this is a matter of definitions. The fidelity and generalizability will reach human-level and quickly surpass humans - People are going to combine this with 3D AI and LLMs to build AAA games.

Special thanks to Google DeepMind for inviting me to try out Genie 3. I'm excited to share my thoughts on this early research prototype and also some of my live recordings below: I spent the whole day playing with the system and when it works, it is truly mind blowing🤯. It is the first neural game engine / world model I have tried that generalizes so well and has long term world consistency. Here’s a couple of examples from my live recording and some thoughts on what it means for the future of gaming, robotics, digital experiences and ASI. Where it shines: - Truly general-purpose and quick startup time. Works exceptionally well for gaming environments but also generalizes to other industrial and real-world scenarios. - It learns physics. Although there are systematic failures even for rigid body physics, it was clear to me that it can learn game engine and non-rigid physics without an underlying engine (and in limit learn from game engines via training data). - It works exceptionally well for stylized environments with characters walking around. This will have implications for concept artists, level designers and game devs. - It is way more fun than video models, indicating that there are high retention consumer experiences waiting to be built with this in the future - Photorealistic walk throughs and drone shots work exceptionally well - Global illumination and lighting works surprisingly well - Visual memory is quite powerful and the same objects approximately remain coherent under occlusion and longer time horizons Open Problems: - Physics is still hard and there are obvious failure cases when I tried the classical intuitive physics experiments from psychology (tower of blocks). - Social and multi-agent interactions are tricky to handle. 1vs1 combat games do not work - Long instruction following and simple combinatorial game logic fails (e.g. collect some points / keys etc, go to the door, unlock and so on) - Action space is limited - It is far from being a real game engines and has a long way to go but this is a clear glimpse into the future. The Future: - It is impressive enough for me to have strong conviction that this is going to disrupt the gaming industry. It is super early days and there are a lot of failures but the writing is on the wall. Lots of challenging scientific, engineering and scaling problems to be solved but it is going to happen in the next 5 years. - This is the final piece before we get full AGI and now I think we are well on our way to truly solve it once something like this is scaled up. In many ways it is more ASI than AGI but this is a matter of definitions. The fidelity and generalizability will reach human-level and quickly surpass humans - People are going to combine this with 3D AI and LLMs to build AAA games.

Tejas Kulkarni

87,917 次观看 • 11 个月前

Late night on Friday can only mean one thing… working on the coedz for our port of Grand Theft Auto 3 to the Sega Dreamcast… Went a little crazy with my C++20 VMU infrastructure, which now supports “apps” you can swap between in-game, each supporting multiple horizontal pages with vertical scrolling… The main “app” will be the GTA3-specific in-game display for player information such as HP, cash, ammo, and stars, with upcoming UI work done by some of our VMU pixel art rock stars. The other VMU “app” you already know as the real-time profiler that shipped with our alpha release. I’ve been upgrading the shit out of it to track various metrics and counters such as number of cars, pedestrians, rigid bodies, and building entities. Also added scene stats for the number of atomics, meshes, skins, and effect matrices that are drawn in the current frame. Finally, the most important stuff will be a page showing the CPU utilization of all of our threads along with a frame breakdown of the time it takes to process each major subsystem (rendering, collision, AI, streaming assets, physics simulation, etc). This will help drive future performance work as we continue to fine-tune performance. Oh, and before you freak out claiming that this profiling itself must be adding a bunch of overhead, I rolled my own auto assembling intrusively linked list structure of static atomic counter values, which all get linked together upon program initialization and automatically added to a central singleton list of stats which the profiler can later iterate over and inspect from its own thread… Not one goddamn malloc or new call, the VMU thread itself is only awakening every 200ms to update the display, and has a low thread priority, plus I just downsized its default 32KB stack to only 5KB, paying for any static memory costs of storing these metrics by over tenfold…

Late night on Friday can only mean one thing… working on the coedz for our port of Grand Theft Auto 3 to the Sega Dreamcast… Went a little crazy with my C++20 VMU infrastructure, which now supports “apps” you can swap between in-game, each supporting multiple horizontal pages with vertical scrolling… The main “app” will be the GTA3-specific in-game display for player information such as HP, cash, ammo, and stars, with upcoming UI work done by some of our VMU pixel art rock stars. The other VMU “app” you already know as the real-time profiler that shipped with our alpha release. I’ve been upgrading the shit out of it to track various metrics and counters such as number of cars, pedestrians, rigid bodies, and building entities. Also added scene stats for the number of atomics, meshes, skins, and effect matrices that are drawn in the current frame. Finally, the most important stuff will be a page showing the CPU utilization of all of our threads along with a frame breakdown of the time it takes to process each major subsystem (rendering, collision, AI, streaming assets, physics simulation, etc). This will help drive future performance work as we continue to fine-tune performance. Oh, and before you freak out claiming that this profiling itself must be adding a bunch of overhead, I rolled my own auto assembling intrusively linked list structure of static atomic counter values, which all get linked together upon program initialization and automatically added to a central singleton list of stats which the profiler can later iterate over and inspect from its own thread… Not one goddamn malloc or new call, the VMU thread itself is only awakening every 200ms to update the display, and has a low thread priority, plus I just downsized its default 32KB stack to only 5KB, paying for any static memory costs of storing these metrics by over tenfold…

Falco Girgis

11,547 次观看 • 1 年前