Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Excited to share that our paper 🌊🤺 “CFC: Simulating Character–Fluid Coupling using a Two-Level World Model” has been accepted to #SIGGRAPHASIA2025! In this work, we build a two-level world model (neural physics) for rigid-body–fluid interaction and use it to train physics-based character controllers efficiently. We study: (1) learning to...

frankzydou's profile picture

Zhiyang (Frank) Dou

2,115 subscribers

20,247 görüntüleme • 9 ay önce •via X (Twitter)

#SIGGRAPHASIA2025
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Today we are announcing Genie 3, a general purpose world model by Google DeepMind that can generate dynamic, interactive environments with a single text prompt. World models are AI that understand facets of the world (like Veo's knowledge of intuitive physics or Genie's mastery of new environments), and serve as a key stepping stone on the path to AGI. Genie 3 is our first world model to allow interaction in real-time, while also improving consistency and realism compared to Genie 2. Learn more ➡️
0:33

Today we are announcing Genie 3, a general purpose world model by Google DeepMind that can generate dynamic, interactive environments with a single text prompt. World models are AI that understand facets of the world (like Veo's knowledge of intuitive physics or Genie's mastery of new environments), and serve as a key stepping stone on the path to AGI. Genie 3 is our first world model to allow interaction in real-time, while also improving consistency and realism compared to Genie 2. Learn more ➡️

Google AI

102,711 görüntüleme • 1 yıl önce

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.
0:50

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

Jim Fan

567,327 görüntüleme • 1 yıl önce

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
53:45

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,772 görüntüleme • 3 ay önce

🚨 New Paper Hybrid Neural World Models. We build neural world models for discontinuous environments and our method discovers when they're making error, with no ground truth. Our neural models run physics fast, leaping many simulator steps in a single pass. How we find points where not to trust it: ask the model for the same future two ways, and the disagreement between the two answers marks where it's failing without needing labels. The same recipe holds on three systems with nothing in common: a chemical wave, compressible gas with shocks, rigid bodies colliding in 3D. 👇
0:46

🚨 New Paper Hybrid Neural World Models. We build neural world models for discontinuous environments and our method discovers when they're making error, with no ground truth. Our neural models run physics fast, leaping many simulator steps in a single pass. How we find points where not to trust it: ask the model for the same future two ways, and the disagreement between the two answers marks where it's failing without needing labels. The same recipe holds on three systems with nothing in common: a chemical wave, compressible gas with shocks, rigid bodies colliding in 3D. 👇

Lossfunk

15,171 görüntüleme • 3 ay önce

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." Rayan Krishnan Jennifer Li
1:02

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." Rayan Krishnan Jennifer Li

a16z

34,164 görüntüleme • 2 gün önce

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:
1:00

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:

Heng Yang

19,191 görüntüleme • 6 ay önce

The reception since we launched the first unified audio + text + image to video model in production last week has been incredible! We wanted to take a moment and highlight some firsts: - the first omnimodal modal that jointly operates over text, image, and audio. Fun fact: we shipped our first audio-to-video model Character-1 last June! - Character-3 is generating 5 minute uncut clips, and capable of temporally stable longform prediction. - Character-3 handles realistic humans, cartoons, side profiles, dynamic backgrounds, anime, animals, and more. - Character-3 sets a new standard for fast & efficient video diffusion at HD and SD resolutions, leveraging our advancements in realtime video generation. We'll have an announcement soon, but you can read more about an earlier version of this line of work in our paper here:
0:55

The reception since we launched the first unified audio + text + image to video model in production last week has been incredible! We wanted to take a moment and highlight some firsts: - the first omnimodal modal that jointly operates over text, image, and audio. Fun fact: we shipped our first audio-to-video model Character-1 last June! - Character-3 is generating 5 minute uncut clips, and capable of temporally stable longform prediction. - Character-3 handles realistic humans, cartoons, side profiles, dynamic backgrounds, anime, animals, and more. - Character-3 sets a new standard for fast & efficient video diffusion at HD and SD resolutions, leveraging our advancements in realtime video generation. We'll have an announcement soon, but you can read more about an earlier version of this line of work in our paper here:

Hedra

180,676 görüntüleme • 1 yıl önce

New Fable 5 beats Opus 4.8 on real world physics simulations We gave both models the same three prompts and asked them to build self contained HTML5 sims with real physics and no libraries: 1. Chaotic double pendulum 2. Galton board 3. Water in a spinning drum (WCSPH) Generation cost Fable 5: $3.35 on 68.7k tokens, time 14m 47s Opus 4.8: $0.93 on 38.9k tokens, time 8m 10s Fable clearly did better on the water simulation, producing a much more solid and continuous body of water. Opus left larger gaps near the walls, scattered particles around the scene, and struggled to keep the fluid stable.
0:22

New Fable 5 beats Opus 4.8 on real world physics simulations We gave both models the same three prompts and asked them to build self contained HTML5 sims with real physics and no libraries: 1. Chaotic double pendulum 2. Galton board 3. Water in a spinning drum (WCSPH) Generation cost Fable 5: $3.35 on 68.7k tokens, time 14m 47s Opus 4.8: $0.93 on 38.9k tokens, time 8m 10s Fable clearly did better on the water simulation, producing a much more solid and continuous body of water. Opus left larger gaps near the walls, scattered particles around the scene, and struggled to keep the fluid stable.

atomic.chat

1,587,969 görüntüleme • 3 ay önce

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.
5:19

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

MrNeRF

34,782 görüntüleme • 1 yıl önce

The final lecture of my course is an intro to character training! This is a topic that I've been quietly very invested in for ~18 months, as it: * Has potential for high real world impact * Clearly used extensively at frontier labs * Almost no empirical literature exists * More accessible on academic compute This lecture covers what character training is, reviews model specs, constitutions, the differences, the motivations in real world events, some example research papers I like, and open questions in how it relates to post-training/model use generally. Hopefully this brings more people into the field (and reach out if you have questions). It is one of the more research-y chapters in my book, but one that I felt needed the reference. There is still so little, educational content on the topic online. 0:00 Intro 6:22 Part 1: Fundamentals — character, constitutions, and model specs 19:21 Part 2: Character training in practice 23:23 Part 3: Character elicitation without gradient steps 28:03 Part 4: Open questions (and the end of the course) 32:27 The course, complete Thanks for watching. No need to like and subscribe now that the course is done, you definitely wouldn't! h/t to Sharan for leading the technical work I got to do in the space, and Zafir Stojanovski for investing a lot of attention at this book chapter.
32:55

The final lecture of my course is an intro to character training! This is a topic that I've been quietly very invested in for ~18 months, as it: * Has potential for high real world impact * Clearly used extensively at frontier labs * Almost no empirical literature exists * More accessible on academic compute This lecture covers what character training is, reviews model specs, constitutions, the differences, the motivations in real world events, some example research papers I like, and open questions in how it relates to post-training/model use generally. Hopefully this brings more people into the field (and reach out if you have questions). It is one of the more research-y chapters in my book, but one that I felt needed the reference. There is still so little, educational content on the topic online. 0:00 Intro 6:22 Part 1: Fundamentals — character, constitutions, and model specs 19:21 Part 2: Character training in practice 23:23 Part 3: Character elicitation without gradient steps 28:03 Part 4: Open questions (and the end of the course) 32:27 The course, complete Thanks for watching. No need to like and subscribe now that the course is done, you definitely wouldn't! h/t to Sharan for leading the technical work I got to do in the space, and Zafir Stojanovski for investing a lot of attention at this book chapter.

Nathan Lambert

57,151 görüntüleme • 1 ay önce

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀
0:50

Combinatorial complexity is often the bane of imitation learning - including VLA models! Jesse Zhang and Marius Memmel proposed a way around this, using VLMs to perform problem reduction for imitation. The insight is simple - 1) High-level VLM takes a complex scene/task and reducing it a minimal representation (via masking and path prediction) that is needed to act in the world. 2) A low-level policy then takes this reduced representation and generates actions to be executed in the world. The high-level policy absorbs all the combinatorial complexity of the problem, leaving the low-level to focus on dexterity and geometric reasoning. Super simple, works really well across policy classes and problem settings! - 41.4× sim2real improvement (3DDA) and 2–3.5× boosts for π₀ and ACT in the real world. Paper: Website: Demo: Fun collaboration led by Jesse Zhang Marius Memmel with lots of collaborators! Let us know what you think 😀

Abhishek Gupta

22,354 görüntüleme • 11 ay önce

I’m so excited to finally talk about our new model, Atlas, and share some generations from it..so excited, in fact, that I made a fan trailer for it 😆 It’s an incredibly versatile, multi-task model with a wide range of capabilities. There are many powerful video models out there right now, but after working with Atlas and experimenting with it for so long, I’ve come to really appreciate its elegance and malleability...all rooted in the framework’s simple autoregressive design. As you can see in this video, Atlas is, at a high level, a camera-conditioned visual generation model. But its formulation allows it to learn and perform a variety of tasks remarkably efficiently!
0:23

I’m so excited to finally talk about our new model, Atlas, and share some generations from it..so excited, in fact, that I made a fan trailer for it 😆 It’s an incredibly versatile, multi-task model with a wide range of capabilities. There are many powerful video models out there right now, but after working with Atlas and experimenting with it for so long, I’ve come to really appreciate its elegance and malleability...all rooted in the framework’s simple autoregressive design. As you can see in this video, Atlas is, at a high level, a camera-conditioned visual generation model. But its formulation allows it to learn and perform a variety of tasks remarkably efficiently!

Gowthami

16,804 görüntüleme • 10 gün önce

In our research lab, we are building “real-time dreaming” - the ability to generate fully playable video worlds prompted from any text or image. Our real-time, action conditioned world model (currently running internally at 16fps at 832x480p) is trained on a combination of data, including proprietary Roblox 3D avatar/world interaction data. World models are different from multiplayer engines in that they store state and memory in video latents. Roblox is multiplayer, and we are actively researching optimal ways to simultaneously store state for thousands of players, and keep them in sync with their environment. Our world model leverages database technology which stores all user interactions on Roblox in a vector format that can be used to re-render video and interaction from any camera angle. We see several immediate uses for our Roblox world model. We will use it side-by-side text, image and video prompts as a way to launch auto-generation of immersive worlds. In Roblox Studio, a creator could walk around and use prompts to “paint” a world and then convert it into a 3D representation or direct to Roblox native as a way for many people to play simultaneously. All of this comes alive as we explore the notion of a “Dream Theater” - where one user is dreaming, while others watch and prompt them. 2/4
1:17

In our research lab, we are building “real-time dreaming” - the ability to generate fully playable video worlds prompted from any text or image. Our real-time, action conditioned world model (currently running internally at 16fps at 832x480p) is trained on a combination of data, including proprietary Roblox 3D avatar/world interaction data. World models are different from multiplayer engines in that they store state and memory in video latents. Roblox is multiplayer, and we are actively researching optimal ways to simultaneously store state for thousands of players, and keep them in sync with their environment. Our world model leverages database technology which stores all user interactions on Roblox in a vector format that can be used to re-render video and interaction from any camera angle. We see several immediate uses for our Roblox world model. We will use it side-by-side text, image and video prompts as a way to launch auto-generation of immersive worlds. In Roblox Studio, a creator could walk around and use prompts to “paint” a world and then convert it into a 3D representation or direct to Roblox native as a way for many people to play simultaneously. All of this comes alive as we explore the notion of a “Dream Theater” - where one user is dreaming, while others watch and prompt them. 2/4

Roblox

4,407,793 görüntüleme • 7 ay önce

We are excited to share our #CORL2024 paper on learning quadrotor obstacle avoidance from the visual stream of a single #eventcamera! Trained entirely in simulation! We demonstrate obstacle avoidance both in the dark and in a forest up to 5m/s. PDF: Video: Project page: Event cameras are sensors that output per-pixel-level intensity changes at microsecond latency resolution; they feature nearly zero motion blur and high dynamic range but produce a very large volume of events under significant ego-motion and further lack a high-fidelity continuous-time sensor model in simulation, making direct #sim2real transfer not possible. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance policy with “approximated” simulated events and then fine-tune the perception component with limited events-and-depth real-world data. This technique bridges the sim2real gap for #eventcameras! As at the current state, there is no continuous-time sensor model for event cameras, we hope that this work can finally spur future research leveraging simulation for training event-vision-based policies to create faster, agile robots! Kudos to Anish Bhattacharya, @marcocannic, Vijay Kumar Nikolai Matni UZH Science University of Zurich UZH Space Hub UZH IfI European Research Council (ERC) GRASP Laboratory Penn Engineering
1:25

We are excited to share our #CORL2024 paper on learning quadrotor obstacle avoidance from the visual stream of a single #eventcamera! Trained entirely in simulation! We demonstrate obstacle avoidance both in the dark and in a forest up to 5m/s. PDF: Video: Project page: Event cameras are sensors that output per-pixel-level intensity changes at microsecond latency resolution; they feature nearly zero motion blur and high dynamic range but produce a very large volume of events under significant ego-motion and further lack a high-fidelity continuous-time sensor model in simulation, making direct #sim2real transfer not possible. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance policy with “approximated” simulated events and then fine-tune the perception component with limited events-and-depth real-world data. This technique bridges the sim2real gap for #eventcameras! As at the current state, there is no continuous-time sensor model for event cameras, we hope that this work can finally spur future research leveraging simulation for training event-vision-based policies to create faster, agile robots! Kudos to Anish Bhattacharya, @marcocannic, Vijay Kumar Nikolai Matni UZH Science University of Zurich UZH Space Hub UZH IfI European Research Council (ERC) GRASP Laboratory Penn Engineering

Davide Scaramuzza

17,219 görüntüleme • 1 yıl önce

Our general understanding of the characteristics of the physical world are largely restricted by the limited range of our senses. The world appears to be comprised of tangible objects positioned within empty space, separate and distinct. Neither of these characterizations are true. Our unaided senses generate a false interpretation of the true state of reality. Space is note empty, it is substantive with a quantifiable and measurable energy density. Space, in terms of quantum vacuum fluctuations, can be considered as a veritable sea of oscillating energy, like a fluid, quantized at the Planck scale (a billion trillion trillion times smaller than a centimeter) as Planck spherical units, and these tiny oscillators make up the fluid medium of space and comprise the "material" stuff as well. Imagine being able to directly perceive this level of reality. Our photodetector proteins in our eyes are sensitive to electromagnetic radiation in the frequency range of 400 to 800 terahertz (trillions of oscillations per second), and we call this "visible light". If, however, we could see light at the Planck scale, were photons oscillate at the Planck frequency— a mass-energy value that makes the electromagnetic component at order of unity with spacetime curvature— then we would theoretically see directly the substantive fluid medium of space and our sight would relay a world that is integrally interconnected and all one substance; objects would not appear as separate and distinct or even fundamentally different than the substance comprising the bulk space. "Material" objects would appear just as patterned vortices of the fluid that is the very substance of space. So, tangible objects are made of the same substance as space, and only seem physical to our limited senses because of electromagnetic repulsive forces. The electromagnetic repulsive forces are generated by how these PSUs circulate and flow within the structured patterns of space that we call particles and atoms. In this way, the coherent phases, circulation, flow, and pressure forces of this Planck plasma fluid are the source of mass, force, and charge. We are now coming to an understanding of these dynamics at a fundamental level, exemplified in the publication The Origin of Mass and the Nature of Gravity 🔗
0:45

Our general understanding of the characteristics of the physical world are largely restricted by the limited range of our senses. The world appears to be comprised of tangible objects positioned within empty space, separate and distinct. Neither of these characterizations are true. Our unaided senses generate a false interpretation of the true state of reality. Space is note empty, it is substantive with a quantifiable and measurable energy density. Space, in terms of quantum vacuum fluctuations, can be considered as a veritable sea of oscillating energy, like a fluid, quantized at the Planck scale (a billion trillion trillion times smaller than a centimeter) as Planck spherical units, and these tiny oscillators make up the fluid medium of space and comprise the "material" stuff as well. Imagine being able to directly perceive this level of reality. Our photodetector proteins in our eyes are sensitive to electromagnetic radiation in the frequency range of 400 to 800 terahertz (trillions of oscillations per second), and we call this "visible light". If, however, we could see light at the Planck scale, were photons oscillate at the Planck frequency— a mass-energy value that makes the electromagnetic component at order of unity with spacetime curvature— then we would theoretically see directly the substantive fluid medium of space and our sight would relay a world that is integrally interconnected and all one substance; objects would not appear as separate and distinct or even fundamentally different than the substance comprising the bulk space. "Material" objects would appear just as patterned vortices of the fluid that is the very substance of space. So, tangible objects are made of the same substance as space, and only seem physical to our limited senses because of electromagnetic repulsive forces. The electromagnetic repulsive forces are generated by how these PSUs circulate and flow within the structured patterns of space that we call particles and atoms. In this way, the coherent phases, circulation, flow, and pressure forces of this Planck plasma fluid are the source of mass, force, and charge. We are now coming to an understanding of these dynamics at a fundamental level, exemplified in the publication The Origin of Mass and the Nature of Gravity 🔗

Nassim Haramein

15,937 görüntüleme • 2 yıl önce

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.
0:51

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

AK

367,249 görüntüleme • 2 yıl önce

Tracking Anything with Decoupled Video Segmentation paper page: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation.
0:32

Tracking Anything with Decoupled Video Segmentation paper page: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation.

AK

305,683 görüntüleme • 3 yıl önce

We are excited to share our latest work, "Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation", where a policy learns to adapt in the real world to unknown disturbances within 5 seconds, both with and without explicit state estimation, directly from visual features. Code released! PDF: Project Page: Starting from a simple analytical dynamics model, the system continuously learns residual dynamics from real-world data and embeds the refined model into a differentiable simulator. This enables fast, gradient-based policy updates that are far more sample-efficient than classical #ReinforcementLearning. We demonstrate rapid adaptation in <5 seconds in agile quadrotor control under challenging conditions, including added payloads, wind disturbances, and large sim-to-real gaps. In real-world experiments, our method reduces hovering error by up to 81% compared to L1-MPC and 55% compared to PPO-based adaptive methods. It also operates directly from visual features without explicit state estimation. Reference: “Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation” IEEE Robotics and Automation Letters, 2026 PDF: Video: Code: Website: Kudos to Michael Pan, Jiaxu Xing, Rudolf Reiter, Yifan Zhai, Elie Aljalbout! UZH Space Hub UZH IfI European Research Council (ERC) AUTOASSESS UZH Science University of Zurich
1:06

We are excited to share our latest work, "Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation", where a policy learns to adapt in the real world to unknown disturbances within 5 seconds, both with and without explicit state estimation, directly from visual features. Code released! PDF: Project Page: Starting from a simple analytical dynamics model, the system continuously learns residual dynamics from real-world data and embeds the refined model into a differentiable simulator. This enables fast, gradient-based policy updates that are far more sample-efficient than classical #ReinforcementLearning. We demonstrate rapid adaptation in <5 seconds in agile quadrotor control under challenging conditions, including added payloads, wind disturbances, and large sim-to-real gaps. In real-world experiments, our method reduces hovering error by up to 81% compared to L1-MPC and 55% compared to PPO-based adaptive methods. It also operates directly from visual features without explicit state estimation. Reference: “Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation” IEEE Robotics and Automation Letters, 2026 PDF: Video: Code: Website: Kudos to Michael Pan, Jiaxu Xing, Rudolf Reiter, Yifan Zhai, Elie Aljalbout! UZH Space Hub UZH IfI European Research Council (ERC) AUTOASSESS UZH Science University of Zurich

Davide Scaramuzza

19,283 görüntüleme • 8 ay önce