Загрузка видео...

Не удалось загрузить видео

На главную

We present EgoReAct: Real-time 3D human reaction generation from streaming egocentric video. 🌟Reacting to streaming egocentric video is something humans do every day. We hope EgoReAct makes human motion more human-like. 🔎 What we found: existing ego-reaction data can be spatially inconsistent (e.g., moving reactions paired with fixed-camera videos),...

frankzydou's profile picture

Zhiyang (Frank) Dou

2,047 subscribers

11,336 просмотров • 6 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI
3:58

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Zhiyang (Frank) Dou

14,610 просмотров • 1 год назад

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.
1:29

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

148,998 просмотров • 1 год назад

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:
1:07

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

SpAItial AI

175,909 просмотров • 7 месяцев назад

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:
1:21

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,380 просмотров • 1 год назад

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!
0:30

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

95,918 просмотров • 6 месяцев назад

🚨 SIGGRAPH Asia 2025 Paper Alert 🚨 ➡️Paper Title: WorldExplorer: Towards Generating Fully Navigable 3D Scenes 🌟Few pointers from the paper 🎯Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce stretched-out and noisy artifacts when moving beyond central or panoramic perspectives. 🎯 To this end, authors of this paper proposed “WorldExplorer”, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. 🎯They initialize their scenes by creating multi-view consistent images corresponding to a 360 degree panorama. 🎯Then, they expanded it by leveraging video diffusion models in an iterative scene generation pipeline. 🎯Concretely, they generated multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. 🎯Their novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. 🎯Finally,they fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. 🎯Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. 🎯They believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments. 🏢Organization: TU München 🧙Paper Authors: Manuel-Andreas Schneider, Lukas Höllein , Matthias Niessner 📝 Read the Full Paper here: 🗂️ Project Page: 🧑‍💻 Code: 🎥 Be sure to watch the attached Technical Summary Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #SIGGRAPHAsia2025
3:36

🚨 SIGGRAPH Asia 2025 Paper Alert 🚨 ➡️Paper Title: WorldExplorer: Towards Generating Fully Navigable 3D Scenes 🌟Few pointers from the paper 🎯Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce stretched-out and noisy artifacts when moving beyond central or panoramic perspectives. 🎯 To this end, authors of this paper proposed “WorldExplorer”, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. 🎯They initialize their scenes by creating multi-view consistent images corresponding to a 360 degree panorama. 🎯Then, they expanded it by leveraging video diffusion models in an iterative scene generation pipeline. 🎯Concretely, they generated multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. 🎯Their novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. 🎯Finally,they fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. 🎯Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. 🎯They believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments. 🏢Organization: TU München 🧙Paper Authors: Manuel-Andreas Schneider, Lukas Höllein , Matthias Niessner 📝 Read the Full Paper here: 🗂️ Project Page: 🧑‍💻 Code: 🎥 Be sure to watch the attached Technical Summary Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #SIGGRAPHAsia2025

naveen manwani

10,578 просмотров • 9 месяцев назад

After 8 months of building in stealth and testing our infrastructure on 10000+ hours of real-world data and hundreds of unique environments, we're bringing FPV Labs into the open today. FPV Labs started with the following bet - if human data proves to be the underlying factor that determines scaling laws in general-purpose robotics, it will trigger the largest economic transformation in human history, and the underlying infrastructure that captures that data will determine how fast we get there. We will achieve this by building the full-stack infrastructure for capturing, processing, transferring, and evaluating human experience into spatial, temporal, and semantic knowledge for machines. Despite all the research novelty behind ChatGPT, its success can be attributed to one foundational fact - the scaling law of transformers. We believe the same dynamics have made their way into robotics. Recent studies showed task completion rates jumping from 30% to 70% when human demonstration data scaled from 1,000 to 20,000 hours, a log-linear trend that mirrors exactly what we saw in language and vision. Seeing these emergent signs of scaling law curves in robotics, we believe we are entering the era of general-purpose robotics policies, which makes the next few years the most exciting time in the history of this field. But the library of physical interactions required to train general-purpose robot policies does not exist yet. Over the last 8 months, we've seen dozens of companies emerge in this space. We were really happy to see new companies pushing this space forward, but we also saw the same pattern repeat: every egocentric data company was making some tradeoffs between quality, scale, and diversity. We have built FPV labs on the core principle that high-quality data is orders of magnitude more valuable than sheer volume. Case in point, self-driving cars collect thousands of hours of data per day, but only a small fraction of that data is actually useful for training better models. Several studies, like RT-2, have shown that as little as 1% of data improves as much as 25% on task success. The quality and diversity of data matter a lot more than scale, so there is clearly a power law curve in the downstream impact of data. We've spent months obsessing over data quality by building our stack, discarding it, rebuilding it, and iterating until we found a formula that doesn't compromise downstream quality at scale. We believe the downstream impact here is far more profound than most people realize. Workers globally are paid around $60 trillion per year in aggregate, and a lion's share of that compensation goes to physical labor - tasks that require navigating real spaces, manipulating real objects, and negotiating the infinite variability of the physical world. Human-to-robot transfer will be one of the most important infrastructures that will shape our society in the near future, and if it works, the economic impact will dwarf every technology transition that came before it in an exponential manner and lead to the creation of goods and services we can’t imagine today. Our mission is to lay the groundwork for us to transition into this future - the future of abundance. We are deeply grateful to our earliest believers, Paras Chopra and Lossfunk, who played a critical role in shaping our thinking.
0:36

After 8 months of building in stealth and testing our infrastructure on 10000+ hours of real-world data and hundreds of unique environments, we're bringing FPV Labs into the open today. FPV Labs started with the following bet - if human data proves to be the underlying factor that determines scaling laws in general-purpose robotics, it will trigger the largest economic transformation in human history, and the underlying infrastructure that captures that data will determine how fast we get there. We will achieve this by building the full-stack infrastructure for capturing, processing, transferring, and evaluating human experience into spatial, temporal, and semantic knowledge for machines. Despite all the research novelty behind ChatGPT, its success can be attributed to one foundational fact - the scaling law of transformers. We believe the same dynamics have made their way into robotics. Recent studies showed task completion rates jumping from 30% to 70% when human demonstration data scaled from 1,000 to 20,000 hours, a log-linear trend that mirrors exactly what we saw in language and vision. Seeing these emergent signs of scaling law curves in robotics, we believe we are entering the era of general-purpose robotics policies, which makes the next few years the most exciting time in the history of this field. But the library of physical interactions required to train general-purpose robot policies does not exist yet. Over the last 8 months, we've seen dozens of companies emerge in this space. We were really happy to see new companies pushing this space forward, but we also saw the same pattern repeat: every egocentric data company was making some tradeoffs between quality, scale, and diversity. We have built FPV labs on the core principle that high-quality data is orders of magnitude more valuable than sheer volume. Case in point, self-driving cars collect thousands of hours of data per day, but only a small fraction of that data is actually useful for training better models. Several studies, like RT-2, have shown that as little as 1% of data improves as much as 25% on task success. The quality and diversity of data matter a lot more than scale, so there is clearly a power law curve in the downstream impact of data. We've spent months obsessing over data quality by building our stack, discarding it, rebuilding it, and iterating until we found a formula that doesn't compromise downstream quality at scale. We believe the downstream impact here is far more profound than most people realize. Workers globally are paid around $60 trillion per year in aggregate, and a lion's share of that compensation goes to physical labor - tasks that require navigating real spaces, manipulating real objects, and negotiating the infinite variability of the physical world. Human-to-robot transfer will be one of the most important infrastructures that will shape our society in the near future, and if it works, the economic impact will dwarf every technology transition that came before it in an exponential manner and lead to the creation of goods and services we can’t imagine today. Our mission is to lay the groundwork for us to transition into this future - the future of abundance. We are deeply grateful to our earliest believers, Paras Chopra and Lossfunk, who played a critical role in shaping our thinking.

Abhishek Anand

81,285 просмотров • 3 месяцев назад

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:
1:17

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Jim Fan

225,239 просмотров • 4 месяцев назад

Loneliness On The Path & Transcending The Matrix Many of us can relate to the sense and experience of feeling alone as we shed the matrix-cultural social programming and sincerely start to seek truth. As we embark on the path, we start to learn, often remember, realize, and begin to see a deeper reality within and without us. This new sight is contrary to what the masses believe in and how they live their lives. It is also opposed to whatever we found to be true at first. But there was a splinter in the mind - a seed deep within that inspired us to question what we've been told and taught most often after a period of intense suffering responding to the yearning of the soul's call to awaken. It pushed us to answer the call and step into the unknown, and down the rabbit hole we tumble. As we progress on our path, going through our trials initiations and process of disillusionment, truth starts to become stranger than fiction. We can no longer relate to pleasures, social interactions, and enjoyments we may have engaged in enthusiastically in the past. Goals, ambitions, and desires we once had and were very attached to start to feel completely foreign to us. We realize that most of our so-called needs and desires, wants, including group and tribal identifications, were based on lies, wounds, and matrix programs, which we mistook stemming from our true self. They do not lead us to nor reflect true fulfillment joy, and love - the pure state of our soul hidden deep within us beyond the ego personality - our unique soul embodied individuality centered in the Divine. We start to see through appearances and all the pretense - the mechanical unconscious social interactions with the unconscious feeding mechanisms between people - emotionally, energetically sexually - where almost everyone is a copy of a copy sleepwalking and dreaming of being awake. We realize how little of real individuality and authenticity there is in the world, and how people mistake their conditioned and programmed ego personality for authenticity and also mistake all kinds of feelings, desires, and cravings within them for the soul being. All of which we have done in the past as well to varying degrees and still may do. We realize how little of true love there is. If we are honest with ourselves, we also question what love truly is and if we have really ever experienced it. The more we uncover within us and work through our shadow aspects, mechanical behavior, wounds, and traumas, the more we see all that in others. As the masks fall away, we start to see the unseen and a whole new world - the real world opens up. At first this view is not pleasant at all, and we may question our sanity, feelings of despair, depression, shame, guilt, anger, sadness, and loneliness come up everything that we have been hiding and have suppressed for lifetimes starts to come to the surface the darkness within and without is being made conscious as we descend into the underworld. The higher we want to rise, the deeper we need to go, The more light we possess, the more darkness we uncover. It is the esoteric law of descent and ascent at the same time. If we are sincere in our process, we start to realize thatwe cannot blame others or anything externally anymore. We are asked to take full self responsibility with radical self-honesty. The more we awaken, the more we are also confronted with the illusion of control, the ego likes to cling to. There is something else, a higher divine force that is pulling us out of inertia and sleep, asking us to surrender to it while, at the same time, the lower forces of nature want to keep us chained to our past, our wounds, programs, and identifications trapped in the victim-blame cycle. The occult adverse forces tag into our lower nature using temptations, egohooks and thought injections targeting our blind spots and wounds to keep us plugged into the matrix. Ultimately, we realize that the battle is and always has been through us. We then also start to see and sense the mighty occult hyperdimensional forces beyond our five sensory cognition that manipulate not only us but all people like puppets with strings attached. We realize that there is something else hidden from the mind of ordinary consciousness that controls and manipulates human beings from occult realms - something that feeds on humanity and keeps us in a frequency prison of fear and survival-based egoic service to self-consciousness. We also start to see the wounds and traumas people carry masked and armored under a thick social mask disconnected from their bodies. In fact, we begin to see how most people suffer deep inside but don't know that they are suffering for all the attention and energy is used to escape it by dissociation and then externalizing or suppressing their pain to avoid the discomfort of feeling it within. Like unconscious drug addicts, most people are constantly looking for a fix, trying to fill a bottomless hole in, always looking for love, fulfillment, and happiness externally- just like we have done in our unconscious sleeping state, dreaming of being awake. We then start to encounter humans who don't seem to have a soul but are empty shells occupied by something else. The eyes always give it away - all of which becomes more obvious as we cleanse our perception by internal work. We don't get fooled by appearances anymore. We begin to see via inner vision and our guidance comes more and more from within, not influenced by the group hive mind or the conditioned mind with all its fragmented thought loops. We let go of comparison, ambition and competition and surrender to our soul's calling to unite with the Divine. However, not only do we experience isolation and loneliness pulling back more and more from exterior life we also become subject to the projections and attacks of people still wandering in the wilderness of the collective matrix program. We realize that longtime friends and even family members or our spouse and partner are not at all who we thought they were - that these connections were often based on the illusion of love, trauma bonds, and social programming. We realize how we mistook love for all kinds of behaviors, feelings, and things that are not love but were based on wounds and unconscious feeding mechanisms. When we tell others about our discoveries and realizations we are most often met with ridicule made fun of or even attacked and shamed. We are being told that we are negative, crazy and live in fear or engage in fear-mongering-when, in fact, this reaction is their unconscious fear of the truth projected onto us. It is their false mechanical ego personality trying to protect itself from disillusionment with occult forces, working through them, using them as unconscious matrix agents and attempting to keep us from awakening, speaking truth, and anchoring the divine force. Hence, we also realize that we cannot possibly take anything personally anymore because it is not them doing anything. Moreover, we also recognize that these attacks and interferences serve a teaching function in the bigger picture of the evolution of consciousness and in the process of our unique soul evolution for all there is are lessons. These realizations and confronting the lies we've been telling ourselves do indeed lead to disillusionment - a moral bankruptcy. It's the necessary death everyone has to go through to be reborn in the true self. It is the way out of the matrix. Yet this loneliness we experience and the inability to fit in anywhere has its purpose, including the suffering we experience. If we keep on the track with faith and trust, learning our lessons with sincerity, patience, humility, and courage, it will lead us to true love, real companionship, and a conscious connection with Spirit as we find our way back home, reuniting with the Divine. We start to experience magical positive synchronicities and there is help and support of the divine force that responds to our call and supports and guides us in unexpected ways. But we can only perceive this guidance if we get out of the tunnel vision of ego identification and out of our heads. Doors open where there have been no doors before. Out of the dark, we emerge into the light, and we realize and experience on an embodied level—not merely philosophically or intellectually—that we were never alone but under a spell of separation from all that is, as we experience and perceive the Divine in all. Godspeed. Bernhard Guenther “We must become so alone, so utterly alone, that we withdraw into our innermost self. It is a way of bitter suffering. But then our solitude is overcome, we are no longer alone, for we find that our innermost self is the spirit, that it is Creator, the indivisible. And suddenly, we find ourselves in the midst of the world, yet undisturbed by its multiplicity, for our innermost soul we know ourselves to be one with all being.” - Hermann Hesse --------------------- Written and narrated by Bernhard Guenther Video and audio editing by Victor Fontane
11:52

Loneliness On The Path & Transcending The Matrix Many of us can relate to the sense and experience of feeling alone as we shed the matrix-cultural social programming and sincerely start to seek truth. As we embark on the path, we start to learn, often remember, realize, and begin to see a deeper reality within and without us. This new sight is contrary to what the masses believe in and how they live their lives. It is also opposed to whatever we found to be true at first. But there was a splinter in the mind - a seed deep within that inspired us to question what we've been told and taught most often after a period of intense suffering responding to the yearning of the soul's call to awaken. It pushed us to answer the call and step into the unknown, and down the rabbit hole we tumble. As we progress on our path, going through our trials initiations and process of disillusionment, truth starts to become stranger than fiction. We can no longer relate to pleasures, social interactions, and enjoyments we may have engaged in enthusiastically in the past. Goals, ambitions, and desires we once had and were very attached to start to feel completely foreign to us. We realize that most of our so-called needs and desires, wants, including group and tribal identifications, were based on lies, wounds, and matrix programs, which we mistook stemming from our true self. They do not lead us to nor reflect true fulfillment joy, and love - the pure state of our soul hidden deep within us beyond the ego personality - our unique soul embodied individuality centered in the Divine. We start to see through appearances and all the pretense - the mechanical unconscious social interactions with the unconscious feeding mechanisms between people - emotionally, energetically sexually - where almost everyone is a copy of a copy sleepwalking and dreaming of being awake. We realize how little of real individuality and authenticity there is in the world, and how people mistake their conditioned and programmed ego personality for authenticity and also mistake all kinds of feelings, desires, and cravings within them for the soul being. All of which we have done in the past as well to varying degrees and still may do. We realize how little of true love there is. If we are honest with ourselves, we also question what love truly is and if we have really ever experienced it. The more we uncover within us and work through our shadow aspects, mechanical behavior, wounds, and traumas, the more we see all that in others. As the masks fall away, we start to see the unseen and a whole new world - the real world opens up. At first this view is not pleasant at all, and we may question our sanity, feelings of despair, depression, shame, guilt, anger, sadness, and loneliness come up everything that we have been hiding and have suppressed for lifetimes starts to come to the surface the darkness within and without is being made conscious as we descend into the underworld. The higher we want to rise, the deeper we need to go, The more light we possess, the more darkness we uncover. It is the esoteric law of descent and ascent at the same time. If we are sincere in our process, we start to realize thatwe cannot blame others or anything externally anymore. We are asked to take full self responsibility with radical self-honesty. The more we awaken, the more we are also confronted with the illusion of control, the ego likes to cling to. There is something else, a higher divine force that is pulling us out of inertia and sleep, asking us to surrender to it while, at the same time, the lower forces of nature want to keep us chained to our past, our wounds, programs, and identifications trapped in the victim-blame cycle. The occult adverse forces tag into our lower nature using temptations, egohooks and thought injections targeting our blind spots and wounds to keep us plugged into the matrix. Ultimately, we realize that the battle is and always has been through us. We then also start to see and sense the mighty occult hyperdimensional forces beyond our five sensory cognition that manipulate not only us but all people like puppets with strings attached. We realize that there is something else hidden from the mind of ordinary consciousness that controls and manipulates human beings from occult realms - something that feeds on humanity and keeps us in a frequency prison of fear and survival-based egoic service to self-consciousness. We also start to see the wounds and traumas people carry masked and armored under a thick social mask disconnected from their bodies. In fact, we begin to see how most people suffer deep inside but don't know that they are suffering for all the attention and energy is used to escape it by dissociation and then externalizing or suppressing their pain to avoid the discomfort of feeling it within. Like unconscious drug addicts, most people are constantly looking for a fix, trying to fill a bottomless hole in, always looking for love, fulfillment, and happiness externally- just like we have done in our unconscious sleeping state, dreaming of being awake. We then start to encounter humans who don't seem to have a soul but are empty shells occupied by something else. The eyes always give it away - all of which becomes more obvious as we cleanse our perception by internal work. We don't get fooled by appearances anymore. We begin to see via inner vision and our guidance comes more and more from within, not influenced by the group hive mind or the conditioned mind with all its fragmented thought loops. We let go of comparison, ambition and competition and surrender to our soul's calling to unite with the Divine. However, not only do we experience isolation and loneliness pulling back more and more from exterior life we also become subject to the projections and attacks of people still wandering in the wilderness of the collective matrix program. We realize that longtime friends and even family members or our spouse and partner are not at all who we thought they were - that these connections were often based on the illusion of love, trauma bonds, and social programming. We realize how we mistook love for all kinds of behaviors, feelings, and things that are not love but were based on wounds and unconscious feeding mechanisms. When we tell others about our discoveries and realizations we are most often met with ridicule made fun of or even attacked and shamed. We are being told that we are negative, crazy and live in fear or engage in fear-mongering-when, in fact, this reaction is their unconscious fear of the truth projected onto us. It is their false mechanical ego personality trying to protect itself from disillusionment with occult forces, working through them, using them as unconscious matrix agents and attempting to keep us from awakening, speaking truth, and anchoring the divine force. Hence, we also realize that we cannot possibly take anything personally anymore because it is not them doing anything. Moreover, we also recognize that these attacks and interferences serve a teaching function in the bigger picture of the evolution of consciousness and in the process of our unique soul evolution for all there is are lessons. These realizations and confronting the lies we've been telling ourselves do indeed lead to disillusionment - a moral bankruptcy. It's the necessary death everyone has to go through to be reborn in the true self. It is the way out of the matrix. Yet this loneliness we experience and the inability to fit in anywhere has its purpose, including the suffering we experience. If we keep on the track with faith and trust, learning our lessons with sincerity, patience, humility, and courage, it will lead us to true love, real companionship, and a conscious connection with Spirit as we find our way back home, reuniting with the Divine. We start to experience magical positive synchronicities and there is help and support of the divine force that responds to our call and supports and guides us in unexpected ways. But we can only perceive this guidance if we get out of the tunnel vision of ego identification and out of our heads. Doors open where there have been no doors before. Out of the dark, we emerge into the light, and we realize and experience on an embodied level—not merely philosophically or intellectually—that we were never alone but under a spell of separation from all that is, as we experience and perceive the Divine in all. Godspeed. Bernhard Guenther “We must become so alone, so utterly alone, that we withdraw into our innermost self. It is a way of bitter suffering. But then our solitude is overcome, we are no longer alone, for we find that our innermost self is the spirit, that it is Creator, the indivisible. And suddenly, we find ourselves in the midst of the world, yet undisturbed by its multiplicity, for our innermost soul we know ourselves to be one with all being.” - Hermann Hesse --------------------- Written and narrated by Bernhard Guenther Video and audio editing by Victor Fontane

Bernhard Guenther

136,740 просмотров • 1 год назад

12 hours ago I made the decision to fly out to El Salvador to be the first ever to make an AI generated short film for a country. I’m now 4 hours away from arriving there. (Talk about executing fast) I will take my time writing this message because the reason why has layers. A few months ago I founded ARQ and bootstrapped a platform that lets anyone create high-quality AI productions from start to finish. the best models a multi-agent creative assistant a built-in video editor All in one place. Alongside the product, I’ve built a strong community where I livestream daily, teaching creators how to use AI to tell better stories and get paid for their work. Shortly after we got dozens of offers to make commercials for companies. This was fun, until we came to the conclusion that we were wasting a lot of potential by focussing on companies, while we could also tell the story of nations, movements, communities and narratives that the world needs to hear. So we’re starting with El Salvador. Why? Because Nayib Bukele is one of the most inspiring leaders in the world, not only embracing, but embodying change, turning the country from one of the most dangerous places on earth, to one of the safest and innovation friendly nations. Humanity is experiencing a shift. New technology, new philosophies, new artists, new story tellers, new builders. A new world. So the argument is not AI vs. Non AI. The argument is: Soul Vs. Soulless Pro Human vs. Anti Human Building Vs. Gambling Effort Vs. Effortless This is what ARQ stands for. Not just a SaaS but a safe haven for the soul. It feels like my moral responsibility to speak not only for the AI community but for every human creator. So I decided to do something that requires so much effort and human involvement; no one can deny that AI serves only as a tool in the process. Instead of making 100 commercials for companies, I want to prove what 2 young humans can do with balls and the tools available today. “You can just do things” Yesterday evening me and Ya-Sirr D left our comfortable office in Dubai and are now on our way to El Salvador. The goal: to create a 3-5 Minute AI powered short film telling THEIR story. We will spend every day live streaming, showcasing the process of creating the video while pushing the boundaries of every tool available on the market. Think Google DeepMind , Hailuo AI (MiniMax), Kling AI, Reve, Midjourney, xAI All through ARQ. Who knows who we inspire? Who knows who we meet? We might as well end up meeting the president himself. We want to get to know the country, its people and imagine its future. We want to visit innovative projects in El Salvador, hear the stories of it’s citizens and include them in the video: MURPHSLIFE Stacy Herbert 🇸🇻🚀 @BuildSalvador The Bitcoin Office Bitcoin Beach Anyone who can help us tell the story. Not just any story, one of the most inspiring stories of the past 10 years. Like I said. You can just do things. We have to stop thinking in steps. I know where I will be in 30 days if I stay in the office. More followers. More beta testers. More live streams. More revenue. But by taking this leap of faith. We can be a case study. An inspiration. An ARQ. I remove expectations and increase chances of special things happening. If anyone in El Salvador has any recommendations on who and what places I should visit, send me a DM. We will post updates on X and create an entire “vlog/documentary” for youtube. Join the discord for daily live streams and lessons on how to improve your creation skills. We expect that it will take 7-9 days to finish the entire short film. It will be posted on all social platforms. ARQ. A safe haven for the soul.
2:33

12 hours ago I made the decision to fly out to El Salvador to be the first ever to make an AI generated short film for a country. I’m now 4 hours away from arriving there. (Talk about executing fast) I will take my time writing this message because the reason why has layers. A few months ago I founded ARQ and bootstrapped a platform that lets anyone create high-quality AI productions from start to finish. the best models a multi-agent creative assistant a built-in video editor All in one place. Alongside the product, I’ve built a strong community where I livestream daily, teaching creators how to use AI to tell better stories and get paid for their work. Shortly after we got dozens of offers to make commercials for companies. This was fun, until we came to the conclusion that we were wasting a lot of potential by focussing on companies, while we could also tell the story of nations, movements, communities and narratives that the world needs to hear. So we’re starting with El Salvador. Why? Because Nayib Bukele is one of the most inspiring leaders in the world, not only embracing, but embodying change, turning the country from one of the most dangerous places on earth, to one of the safest and innovation friendly nations. Humanity is experiencing a shift. New technology, new philosophies, new artists, new story tellers, new builders. A new world. So the argument is not AI vs. Non AI. The argument is: Soul Vs. Soulless Pro Human vs. Anti Human Building Vs. Gambling Effort Vs. Effortless This is what ARQ stands for. Not just a SaaS but a safe haven for the soul. It feels like my moral responsibility to speak not only for the AI community but for every human creator. So I decided to do something that requires so much effort and human involvement; no one can deny that AI serves only as a tool in the process. Instead of making 100 commercials for companies, I want to prove what 2 young humans can do with balls and the tools available today. “You can just do things” Yesterday evening me and Ya-Sirr D left our comfortable office in Dubai and are now on our way to El Salvador. The goal: to create a 3-5 Minute AI powered short film telling THEIR story. We will spend every day live streaming, showcasing the process of creating the video while pushing the boundaries of every tool available on the market. Think Google DeepMind , Hailuo AI (MiniMax), Kling AI, Reve, Midjourney, xAI All through ARQ. Who knows who we inspire? Who knows who we meet? We might as well end up meeting the president himself. We want to get to know the country, its people and imagine its future. We want to visit innovative projects in El Salvador, hear the stories of it’s citizens and include them in the video: MURPHSLIFE Stacy Herbert 🇸🇻🚀 @BuildSalvador The Bitcoin Office Bitcoin Beach Anyone who can help us tell the story. Not just any story, one of the most inspiring stories of the past 10 years. Like I said. You can just do things. We have to stop thinking in steps. I know where I will be in 30 days if I stay in the office. More followers. More beta testers. More live streams. More revenue. But by taking this leap of faith. We can be a case study. An inspiration. An ARQ. I remove expectations and increase chances of special things happening. If anyone in El Salvador has any recommendations on who and what places I should visit, send me a DM. We will post updates on X and create an entire “vlog/documentary” for youtube. Join the discord for daily live streams and lessons on how to improve your creation skills. We expect that it will take 7-9 days to finish the entire short film. It will be posted on all social platforms. ARQ. A safe haven for the soul.

Amir D

54,566 просмотров • 7 месяцев назад

THE DESENSITIZED GENERATION — Immune To Novelty By AI Video Fatigue With just a few hours of testing OpenAI’s Sora2 Social Network for my clients and just finished a 218 page report, the thing I thought about for decades is upon us: Full Desensitization Of Visual Novelty I knew so many years ago, and one reason I have posted on the account so many novel images and videos that one day we will rapidly become immune to “oh damn, it’s over” images and video. With Sora2 this desensitization will rapidly accelerate as “you and your friends” build endless videos where you and they are inserted. Like all things that get abused, the first phase is “wow look at this”, the next phase is “can it do that” and the last phase is “nothing is real and I don’t care”. The shock value of the entire technology become removed but in the final phases it will be pushed to extremes of insults and potentially purposeful directed reputation damage. This will take place so rapidly as Sora2 and every other social network rushes to take whatever nicely oxygen that is left in the room for more and more “it’s over, we are so cooked” fantastical videos. They will continue to get so much better that even the “debunker” experts will give up their “look at the fingers-type troupe”, they honestly will not know and soon burn out. My easy point is get ready to be bored by the novelty of humans that in the past took potentially hours of thier lives to create novelty in reality. The hard point is what now? Once a society takes something that has increasingly more and more amazing possibilities like AI and rapidly becomes immune to the possibilities the entire industry become devalued because the front facing aspect to AI will be Sora2-like social media. Humanity’s entire past of astonishing achievements, the high points and the low points and the incredible art we can make will equally become devalued because this fatigue will make us not care and not trust what was real. Everything will become a forever now where there was no real past because it would take impossible effort to verify it if at all. Why? Because not only are we the Desensitized Generation we are also the Amnesia Generation where Petabytes of all forms of media, books, papers, microfilm/microfiche, film, VHS, audio tapes, records, etc is being thrown away or decayed. Not to mention the daily removal of text, images, video and audio from the internet. We simply will not have the receipts, we won’t have the memory and the fatigue is too large to care. It took 5 dark nights for the Library to burn, the sum total of human knowledge. In our age the burn is slow and rapidly increasing till the pot boils over. What are the answers? I have thought about this for decades. I will continue to write about it a lot. But one thing you can do is understand we are rapidly losing our past because almost no one cares. I should not be the only one dumpster diving to save what I can. Also protect your mind and preserve your “novelty” sensitivity because once it is burned out with fatigue, it will never return. And I ask you to have grace and compassion to those that don’t know what we know. You are the 1% of the 1% and must serve a place to help those that can understand what this means and have grace to those who can’t. I am serious, be gentle with them as they find the bottom of the hole and hopefully find a way back. This video below was novel and she risked her life. You are the last generation to understand this. The post reposted of a CEO head in a toilet, is AI rendered it is not novelty.
2:03

THE DESENSITIZED GENERATION — Immune To Novelty By AI Video Fatigue With just a few hours of testing OpenAI’s Sora2 Social Network for my clients and just finished a 218 page report, the thing I thought about for decades is upon us: Full Desensitization Of Visual Novelty I knew so many years ago, and one reason I have posted on the account so many novel images and videos that one day we will rapidly become immune to “oh damn, it’s over” images and video. With Sora2 this desensitization will rapidly accelerate as “you and your friends” build endless videos where you and they are inserted. Like all things that get abused, the first phase is “wow look at this”, the next phase is “can it do that” and the last phase is “nothing is real and I don’t care”. The shock value of the entire technology become removed but in the final phases it will be pushed to extremes of insults and potentially purposeful directed reputation damage. This will take place so rapidly as Sora2 and every other social network rushes to take whatever nicely oxygen that is left in the room for more and more “it’s over, we are so cooked” fantastical videos. They will continue to get so much better that even the “debunker” experts will give up their “look at the fingers-type troupe”, they honestly will not know and soon burn out. My easy point is get ready to be bored by the novelty of humans that in the past took potentially hours of thier lives to create novelty in reality. The hard point is what now? Once a society takes something that has increasingly more and more amazing possibilities like AI and rapidly becomes immune to the possibilities the entire industry become devalued because the front facing aspect to AI will be Sora2-like social media. Humanity’s entire past of astonishing achievements, the high points and the low points and the incredible art we can make will equally become devalued because this fatigue will make us not care and not trust what was real. Everything will become a forever now where there was no real past because it would take impossible effort to verify it if at all. Why? Because not only are we the Desensitized Generation we are also the Amnesia Generation where Petabytes of all forms of media, books, papers, microfilm/microfiche, film, VHS, audio tapes, records, etc is being thrown away or decayed. Not to mention the daily removal of text, images, video and audio from the internet. We simply will not have the receipts, we won’t have the memory and the fatigue is too large to care. It took 5 dark nights for the Library to burn, the sum total of human knowledge. In our age the burn is slow and rapidly increasing till the pot boils over. What are the answers? I have thought about this for decades. I will continue to write about it a lot. But one thing you can do is understand we are rapidly losing our past because almost no one cares. I should not be the only one dumpster diving to save what I can. Also protect your mind and preserve your “novelty” sensitivity because once it is burned out with fatigue, it will never return. And I ask you to have grace and compassion to those that don’t know what we know. You are the 1% of the 1% and must serve a place to help those that can understand what this means and have grace to those who can’t. I am serious, be gentle with them as they find the bottom of the hole and hopefully find a way back. This video below was novel and she risked her life. You are the last generation to understand this. The post reposted of a CEO head in a toilet, is AI rendered it is not novelty.

Brian Roemmele

584,290 просмотров • 9 месяцев назад

I vibe coded and built a sprite animation pipeline 🛠️ (Day 22 of making the engine+game) ⬇️ Watch the video if you don't wanna read the wall of text - it directly shows what I do. Shoutout to Jidé ✨ for showing me a paper on black/white combination to get alpha, it's the cleanest method yet, and to Cursor for enabling this entire journey. If you prefer the wall of text here you go: The hardest part of using general image models for 2D sprites isn’t getting a nice-looking frame, it’s getting consistent motion across a whole sprite sheet. You can fake a sheet, but frames won’t align, timing drifts, and you end up with weird artifacts. Even if you manually cut frames + interpolate, the animation often looks “off” because each frame is basically a new interpretation, not the same character evolving over time. This is especially noticeable with public API models like gpt-image-1.5 and Nano Banana. Some custom LoRAs for open models exist, but this is intended for less techy folks. My workaround: use a video model first, then post-process into a sprite sheet. Render the animation over a solid background (white/black/magenta/green), then chroma-key it out (my engine tool supports this). If the motion stays inside the silhouette, this works surprisingly well. You can do this in almost any video editing software too! The catch: keying almost always leaves an “aura” (edge spill). My best results come from interpolating the keyed animation with a clean base sprite, so you keep crisp edges and only “borrow” motion/detail where needed. If the animation extends outside the silhouette (tree branches, hair wisps, foliage), I usually skip “true sprite animation” and do it with shaders instead. Keying can’t fully remove halos there, no matter how much feathering/tuning you do. Another annoying issue: pixel corruption. AI rarely generates a perfectly flat background (pure #000000 or #FF00FF). That tiny noise breaks clean extraction and creates crawling garbage pixels around the subject. For clean base sprites (and even PBR maps), a useful trick is generating the same asset on white + black backgrounds and deriving alpha from the difference. This is basically a matte workflow: white = opaque, black = transparent. It fixes aura… but you’d need it per-frame to fix animation, which is still hard. For simple pixel art (single-digit frames), you can sometimes generate a sprite sheet, then ask the model to recreate it on black/white while preserving alignment… but it’s still manual-heavy. Honestly, at this point, for some projects it’s easier to go 3D → 2D and render clean sprites/maps directly. But I still love pushing “pure 2D” and seeing how far we can take it. Thanks for reading! Follow/bookmark/repost if interested in this kind of content!
3:33

I vibe coded and built a sprite animation pipeline 🛠️ (Day 22 of making the engine+game) ⬇️ Watch the video if you don't wanna read the wall of text - it directly shows what I do. Shoutout to Jidé ✨ for showing me a paper on black/white combination to get alpha, it's the cleanest method yet, and to Cursor for enabling this entire journey. If you prefer the wall of text here you go: The hardest part of using general image models for 2D sprites isn’t getting a nice-looking frame, it’s getting consistent motion across a whole sprite sheet. You can fake a sheet, but frames won’t align, timing drifts, and you end up with weird artifacts. Even if you manually cut frames + interpolate, the animation often looks “off” because each frame is basically a new interpretation, not the same character evolving over time. This is especially noticeable with public API models like gpt-image-1.5 and Nano Banana. Some custom LoRAs for open models exist, but this is intended for less techy folks. My workaround: use a video model first, then post-process into a sprite sheet. Render the animation over a solid background (white/black/magenta/green), then chroma-key it out (my engine tool supports this). If the motion stays inside the silhouette, this works surprisingly well. You can do this in almost any video editing software too! The catch: keying almost always leaves an “aura” (edge spill). My best results come from interpolating the keyed animation with a clean base sprite, so you keep crisp edges and only “borrow” motion/detail where needed. If the animation extends outside the silhouette (tree branches, hair wisps, foliage), I usually skip “true sprite animation” and do it with shaders instead. Keying can’t fully remove halos there, no matter how much feathering/tuning you do. Another annoying issue: pixel corruption. AI rarely generates a perfectly flat background (pure #000000 or #FF00FF). That tiny noise breaks clean extraction and creates crawling garbage pixels around the subject. For clean base sprites (and even PBR maps), a useful trick is generating the same asset on white + black backgrounds and deriving alpha from the difference. This is basically a matte workflow: white = opaque, black = transparent. It fixes aura… but you’d need it per-frame to fix animation, which is still hard. For simple pixel art (single-digit frames), you can sometimes generate a sprite sheet, then ask the model to recreate it on black/white while preserving alignment… but it’s still manual-heavy. Honestly, at this point, for some projects it’s easier to go 3D → 2D and render clean sprites/maps directly. But I still love pushing “pure 2D” and seeing how far we can take it. Thanks for reading! Follow/bookmark/repost if interested in this kind of content!

Startracker 🔺

20,181 просмотров • 5 месяцев назад

🎉 new skill unlocked: 20s uninterrupted, unstitched, single render from our new ai video engine: Nami. This is my birb (#7531) from the Moonbirds collection, idling in the library. patent: "Intra-Latent Semantic Injection via Cross-Spatial Encoding and Decoding during Multi-Pass Inference for Generative AI Video Creation" At Scrypted we've been quietly working on an agentic generative AI stack for two years: • integrating and testing w/ partners across the games & entertainment sectors • stealthily building a community of early believers through AVB • showcasing some of what we're doing with amazing projects like H011yw00d Agent. -- about Nami -- Nami is an agentic orchestration layer for AI video models: it unlocks their inner superpowers without making them rely on custom LoRAs or fine-tunings. Instead of throwing raw training power and tens of millions of dollars at training yet another ai video model: we figured out new ways to use what we have. Nami harnesses a multi-agent system to perform the work needed in taking a simple prompt or image and turning it into something bigger - much bigger. The agentic steps are allowed to manipulate latent space, digging into tensors, yet doing so in semantically aware chunks - meaning that Nami inherently supports video generation of arbitrary length, though it's bound to O(n) rendering time. (We do have some cool sharding tech that allows us to cut the generative time in half for a reference pose idle-animation like this demo). It's also fairly agnostic, picking and choosing the right tools for the job, and plays really well with emerging tech like FLUX Kontext, FramePack, or <- without being limited by any of them. -- use cases -- Even just a year or two ago the 20 second render below would cost a company, paying an agency, around $10k start-to-finish. This one cost me $6.25 on our dev hardware in an unoptimized environment. There's something mind-blowing about the state-of-the-art when we reduce costs to 0.0625% - less than 1% - of what we used to pay. It's also empowering. For creators. Game developers. Content influencers: you name it. -- superpowers -- 1. it does the things you ask for, in the order you asked for it 2. consistency is king 3. single-shot text or image-to-video 4. future videos can reference previous ones to seamlessly maintain style 5. semantic stitching: can't wait to showcase this -- gtm -- We think Generative AI Video, like image generation, like text, like games, should be a publicly accessible common good. We believe democratizing access to Nami in web3, via x402 payments proposed by Drew Coffman, or in World's mini-apps, is a bold step forward for digital freedom. Permissionless, decentralized, generative ai video. Naturally, we'll also soon release a web platform for using Nami in a traditionally SaaSy way: bring your own images, videos, or prompts and we'll take care of the rest. In the mid-term, Scrypted is building a stack of agentic skills (we call it AVB) and making them available to projects like H011yw00d Agent on Virtuals Protocol and other platforms. -- long-term vision -- Scrypted's mission is to decentralize the things that can't be decentralized. We participated in a16z crypto's CSX (London 2024) during our pre-seed specifically to research a new consensus protocol for hard things like AI video and AI agents: where there's no "one right answer". When Zero-Knowledge Proofs (ZKP) can't secure it, and Trusted Execution Environments (TEEs) are too small, we've got you covered with our upcoming Inori Network. -- how you can help -- 1. Are you a GPU farm? We're gonna need more flops. 2. Do you represent an L1 or L2? We want to build bridges. 3. Do you represent a Wallet or App creator? Let's get an endpoint exposed. 4. Are you an investor? Let's chat. 5. Like, repost, share! -- team background -- We come from a background of AI in the Video Game industry with each founder having over 20 years of experience at companies like Electronic Arts & Square Enix. -- contact -- DMs are open, reach out if you want to be an early tester for your site, game, collection, or project! -- try it out -- Go anywhere on X and tag H011yw00d Agent with a prompt and she'll give you a free 2 second render. Have fun making cinematic shorts or meme videos! -- thanks -- AWS Startups has been an incredible help scaling our prototypes. Also, shout out to all loyal beans 🫘 in the Autonomous Virtuals Beings (AVB) community. Nami has a very important role in the upcoming XP agent platform, can't wait to show you all. AVbeings
0:23

🎉 new skill unlocked: 20s uninterrupted, unstitched, single render from our new ai video engine: Nami. This is my birb (#7531) from the Moonbirds collection, idling in the library. patent: "Intra-Latent Semantic Injection via Cross-Spatial Encoding and Decoding during Multi-Pass Inference for Generative AI Video Creation" At Scrypted we've been quietly working on an agentic generative AI stack for two years: • integrating and testing w/ partners across the games & entertainment sectors • stealthily building a community of early believers through AVB • showcasing some of what we're doing with amazing projects like H011yw00d Agent. -- about Nami -- Nami is an agentic orchestration layer for AI video models: it unlocks their inner superpowers without making them rely on custom LoRAs or fine-tunings. Instead of throwing raw training power and tens of millions of dollars at training yet another ai video model: we figured out new ways to use what we have. Nami harnesses a multi-agent system to perform the work needed in taking a simple prompt or image and turning it into something bigger - much bigger. The agentic steps are allowed to manipulate latent space, digging into tensors, yet doing so in semantically aware chunks - meaning that Nami inherently supports video generation of arbitrary length, though it's bound to O(n) rendering time. (We do have some cool sharding tech that allows us to cut the generative time in half for a reference pose idle-animation like this demo). It's also fairly agnostic, picking and choosing the right tools for the job, and plays really well with emerging tech like FLUX Kontext, FramePack, or <- without being limited by any of them. -- use cases -- Even just a year or two ago the 20 second render below would cost a company, paying an agency, around $10k start-to-finish. This one cost me $6.25 on our dev hardware in an unoptimized environment. There's something mind-blowing about the state-of-the-art when we reduce costs to 0.0625% - less than 1% - of what we used to pay. It's also empowering. For creators. Game developers. Content influencers: you name it. -- superpowers -- 1. it does the things you ask for, in the order you asked for it 2. consistency is king 3. single-shot text or image-to-video 4. future videos can reference previous ones to seamlessly maintain style 5. semantic stitching: can't wait to showcase this -- gtm -- We think Generative AI Video, like image generation, like text, like games, should be a publicly accessible common good. We believe democratizing access to Nami in web3, via x402 payments proposed by Drew Coffman, or in World's mini-apps, is a bold step forward for digital freedom. Permissionless, decentralized, generative ai video. Naturally, we'll also soon release a web platform for using Nami in a traditionally SaaSy way: bring your own images, videos, or prompts and we'll take care of the rest. In the mid-term, Scrypted is building a stack of agentic skills (we call it AVB) and making them available to projects like H011yw00d Agent on Virtuals Protocol and other platforms. -- long-term vision -- Scrypted's mission is to decentralize the things that can't be decentralized. We participated in a16z crypto's CSX (London 2024) during our pre-seed specifically to research a new consensus protocol for hard things like AI video and AI agents: where there's no "one right answer". When Zero-Knowledge Proofs (ZKP) can't secure it, and Trusted Execution Environments (TEEs) are too small, we've got you covered with our upcoming Inori Network. -- how you can help -- 1. Are you a GPU farm? We're gonna need more flops. 2. Do you represent an L1 or L2? We want to build bridges. 3. Do you represent a Wallet or App creator? Let's get an endpoint exposed. 4. Are you an investor? Let's chat. 5. Like, repost, share! -- team background -- We come from a background of AI in the Video Game industry with each founder having over 20 years of experience at companies like Electronic Arts & Square Enix. -- contact -- DMs are open, reach out if you want to be an early tester for your site, game, collection, or project! -- try it out -- Go anywhere on X and tag H011yw00d Agent with a prompt and she'll give you a free 2 second render. Have fun making cinematic shorts or meme videos! -- thanks -- AWS Startups has been an incredible help scaling our prototypes. Also, shout out to all loyal beans 🫘 in the Autonomous Virtuals Beings (AVB) community. Nami has a very important role in the upcoming XP agent platform, can't wait to show you all. AVbeings

Tim Cotten

12,617 просмотров • 1 год назад

DarkFi's goal is to create an agorist parallel economy completely free of state power using anonymous crypto and p2p tech. Our future vision is a confederation of autonomous sovereign nations. Nations often overlap and solve their needs locally. We can even have cyber nations. Politics is not voting every 5 years for a political pageant. It is a core human activity for our freedom, security and needs. DarkFi removes the state and middle men from politics. The DarkFi blockchain is fully programmable with ZK smart contracts, and a very advanced base layer tech stack. We have spent years polishing it. We aim to create parallel markets for agorists. The DarkFi tech stack We went further. We created our own stack to organize community. We are on a mission to conquer cyberspace. To wrest power from the state. To mobilize freedom movements by activating revolutionaries. We created the world's strongest anonymous p2p chat, and an anonymous task manager. In 2021, people thought DAOs would change the world and usher in the post-corporate crypto governance. But it never had strength to change anything. Instead it devolved into wagmi nihilism. People lost belief. Why didn't DAOs change anything? DAOs didn't change anything because they were transparent. DAOs must be strongly anon otherwise your movement will go to jail. The status quo does not tolerate competition. Every cell in nature has a cell wall. Defense is integral to organism's survival. We are not hippy pacifists who want to degrow out of existence. We are here to expand. As Satoshi said: "we can win a major battle in the arms race and gain a new territory of freedom for several years." We created the world's first fully anonymous DAO. Nobody has done this yet. That means a token weighted anon treasury with all participants, proposals and activity completely hidden. Nations require sovereignty and the people should be in charge. It won't happen naturally. We have to use tech to exit and form parallel society. DarkFi is now in the final phase of the release cycle. After mainnet, we will begin the next phase of products. Our vision is to revolutionize society/culture, law, education, economy, governance and self-defense. For society and culture we have built the strongest uncensored p2p tech, and we will continue to iterate and double down here. It's essential for humanity, freedom and nature. We have to rethink law without the state. Contracts between individuals, and decentralized polycentric law. This was the original reason why smart contracts were invented. Yet this has not yet emerged. We cannot mobilize at scale without thinking how to enable new forms of human association and bottom-up non-coercive opt-in organizations. Imagine markets where you can opt into legal codes and digital jurisdictions. The education system is broken. Schools create uncreative thinker drones. University is a groupthink institution. We need to go back to the ancient greek ideal of the academy. Students self-learn under the guidance of a mentor. Education is a mix of math, dev, philosophy and sports. DarkFi already is training hacker devs, and in the future will create the first academies to train the next generation of tech leaders. For economy, we need decentralized markets, distributed reputation, private asset issuance, encrypted shopfronts, uncensorable crowdfunding platforms, stock platforms and information markets. It's incredible that in over a decade, nobody has built this yet. DarkFi is going to do it, just wait. Self-defense comes from freedom of information, by giving people the means to defend their community such as 3D printed guns. Lastly governance. We need information markets for decentralized anon discovery. This is how you ensure transparency of governance - by having full protection for individuals to get the word out. Everyone else in crypto is too afraid and incompetent. No other project will build the tools humanity needs. It's why DarkFi exists. DarkFi will come and grow day by day. It is unstoppable and will ultimately win. Post mainnet we will start working on v2 of DarkFi which will be the next gen in super scalable ZK succinct & anonymous blockchain. So we're not worried for scaling either. With smart contracts, arbitration & escrow, decentralized reputation (not identity!), we can pioneer alternative ownership models. The UK ultimately won over spain in the 1500s going on to conquer the world because of inventing the central bank model where the state was fiscalized. Imagine now we about to again overturn the logic of violence with parallel economies where the money will flow. Nothing can stop this. With this new ownership models where communities own their infra and economy can flourish. For example agricultural coops and small farmers can create their own tax-free internal economies with anon token issuance linking up into confederations outside the state. The burden of tax is ultimately highest on small producers. Imagine how powerful this section of the market can be. We then have true sovereignty outside the state. This is more than "numba go up", it's the true real innovation community. It is strong and steady. Other projects want this, but they cannot have this because they are captured ideologically. Mark these words. We are in the turning of a new era where crypto changes the logic of violence. Zoom out and see the big picture. We are destined to win, yet we have to start fighting. ⚔️ Let there be dark.
0:27

DarkFi's goal is to create an agorist parallel economy completely free of state power using anonymous crypto and p2p tech. Our future vision is a confederation of autonomous sovereign nations. Nations often overlap and solve their needs locally. We can even have cyber nations. Politics is not voting every 5 years for a political pageant. It is a core human activity for our freedom, security and needs. DarkFi removes the state and middle men from politics. The DarkFi blockchain is fully programmable with ZK smart contracts, and a very advanced base layer tech stack. We have spent years polishing it. We aim to create parallel markets for agorists. The DarkFi tech stack We went further. We created our own stack to organize community. We are on a mission to conquer cyberspace. To wrest power from the state. To mobilize freedom movements by activating revolutionaries. We created the world's strongest anonymous p2p chat, and an anonymous task manager. In 2021, people thought DAOs would change the world and usher in the post-corporate crypto governance. But it never had strength to change anything. Instead it devolved into wagmi nihilism. People lost belief. Why didn't DAOs change anything? DAOs didn't change anything because they were transparent. DAOs must be strongly anon otherwise your movement will go to jail. The status quo does not tolerate competition. Every cell in nature has a cell wall. Defense is integral to organism's survival. We are not hippy pacifists who want to degrow out of existence. We are here to expand. As Satoshi said: "we can win a major battle in the arms race and gain a new territory of freedom for several years." We created the world's first fully anonymous DAO. Nobody has done this yet. That means a token weighted anon treasury with all participants, proposals and activity completely hidden. Nations require sovereignty and the people should be in charge. It won't happen naturally. We have to use tech to exit and form parallel society. DarkFi is now in the final phase of the release cycle. After mainnet, we will begin the next phase of products. Our vision is to revolutionize society/culture, law, education, economy, governance and self-defense. For society and culture we have built the strongest uncensored p2p tech, and we will continue to iterate and double down here. It's essential for humanity, freedom and nature. We have to rethink law without the state. Contracts between individuals, and decentralized polycentric law. This was the original reason why smart contracts were invented. Yet this has not yet emerged. We cannot mobilize at scale without thinking how to enable new forms of human association and bottom-up non-coercive opt-in organizations. Imagine markets where you can opt into legal codes and digital jurisdictions. The education system is broken. Schools create uncreative thinker drones. University is a groupthink institution. We need to go back to the ancient greek ideal of the academy. Students self-learn under the guidance of a mentor. Education is a mix of math, dev, philosophy and sports. DarkFi already is training hacker devs, and in the future will create the first academies to train the next generation of tech leaders. For economy, we need decentralized markets, distributed reputation, private asset issuance, encrypted shopfronts, uncensorable crowdfunding platforms, stock platforms and information markets. It's incredible that in over a decade, nobody has built this yet. DarkFi is going to do it, just wait. Self-defense comes from freedom of information, by giving people the means to defend their community such as 3D printed guns. Lastly governance. We need information markets for decentralized anon discovery. This is how you ensure transparency of governance - by having full protection for individuals to get the word out. Everyone else in crypto is too afraid and incompetent. No other project will build the tools humanity needs. It's why DarkFi exists. DarkFi will come and grow day by day. It is unstoppable and will ultimately win. Post mainnet we will start working on v2 of DarkFi which will be the next gen in super scalable ZK succinct & anonymous blockchain. So we're not worried for scaling either. With smart contracts, arbitration & escrow, decentralized reputation (not identity!), we can pioneer alternative ownership models. The UK ultimately won over spain in the 1500s going on to conquer the world because of inventing the central bank model where the state was fiscalized. Imagine now we about to again overturn the logic of violence with parallel economies where the money will flow. Nothing can stop this. With this new ownership models where communities own their infra and economy can flourish. For example agricultural coops and small farmers can create their own tax-free internal economies with anon token issuance linking up into confederations outside the state. The burden of tax is ultimately highest on small producers. Imagine how powerful this section of the market can be. We then have true sovereignty outside the state. This is more than "numba go up", it's the true real innovation community. It is strong and steady. Other projects want this, but they cannot have this because they are captured ideologically. Mark these words. We are in the turning of a new era where crypto changes the logic of violence. Zoom out and see the big picture. We are destined to win, yet we have to start fighting. ⚔️ Let there be dark.

Amir Taaki ⛛

27,132 просмотров • 1 год назад

Fast Company just published a great piece on World Labs , Fei-Fei Li , Marble, and the idea that spatial intelligence / world models may be one of the next big shifts in AI. I was happy to be quoted in the article, but I also wanted to share more context about my own experience with World Labs and Marble, and why this direction is especially interesting to me. My starting point: volumetric capture — For the past few years I’ve been exploring and using volumetric capture and reconstruction (photogrammetry, NeRFs, 3D Gaussian Splats) mostly capturing locations around Montreal. Alleys, museums, urban interiors. I love every step of it: the capture itself, the pipeline, and what can be done with the output. Turning real spaces into real-time explorable systems. I do this personally, sharing explorations here, and professionally as chief technologist, and co-founder of Dpt. Physical reality + generative manipulation — In my work I’m especially drawn to mixing physical reality with generative and digital manipulation: using physical interfaces (light, clay, ink, ... ) to drive generative AI pipelines, building mixed reality prototypes that reshape your surroundings, or starting from real captured spaces and transforming them using tools like Marble. Like many people, I saw the World Labs announcement on Twitter in September 2024, and Marble when it surfaced in early December. But by then, I already had a sense something was coming. The first conversation — As someone deep into volumetric capture and radiance fields, I obviously knew about Ben Mildenhall and his pioneering work on NeRF. To my surprise, Ben reached out to me in late June 2024. He’d been following some of my experiments and wanted to chat about my process and workflows and how I was using this “stuff” creatively. At that point he didn’t share what he was building, but we had a genuinely great conversation about radiance fields, AI, and my work. He was curious about the creative perspective, not just the technical one. When the World Labs announcement dropped a few months later, it all made sense. I understood what Ben had been working on, and why the creative angle mattered to them. Then in August 2025, he invited me to try the Marble beta, and I’ve been experimenting with it since. Experimenting with Marble — The first thing I used Marble for was materializing scene and world concepts during ideation at the studio, and seeing if and how it could fit into our production pipeline. In parallel, I dove into a series of experiments focused on world manipulation: starting from real captured spaces and transforming them using Marble. I’d already been exploring that idea using img2img diffusion with ControlNet on NeRF renders, real-time video streams, and even mixed reality using headset camera feeds. But Marble brings something different. It generates persistent, spatially cohesive 3D worlds that can be rendered in real time across a wide range of devices. That’s a real shift. Experiment 01: Parallel Realities — The first experiment, Parallel Realities, starts from a volumetric capture of a real location, reconstructed as 3D Gaussian Splats. Using Marble, I generate an alternate version of that same space, something informed by the original architecture: abandoned, nature-reclaimed, alternate era. Then, using Spark (World Labs’ 3D Gaussian Splatting renderer for THREE.js) I make both realities coexist in the same spatial coordinate system. From there, I use a portal UX mechanic to let the user step between the real reconstruction and the Marble-generated version. Experiment 02: Hidden Depth The second experiment, Hidden Depth, does not transform a space as much as expand it. A captured location has a visual boundary (a mural, a doorway, a dark corridor) and Marble generates what exists beyond it. For example: a Montreal alley has a painted mural; step through it and you’re inside a world informed by what is actually depicted there. World Labs showcased part of this work here: And in their Spark 2.0 post: The project page is here: Why this matters to me — Being able to start from a real 3D Gaussian Splat scene and manipulate it with Marble opens up a lot of ideas. The 3DGS pipeline is becoming an increasingly compelling foundation for exploration, experimentation, and storytelling. What matters most to me right now is more control. The more I can steer the generated scene or world, the more useful the tool becomes. I want more features like the already existing multiple input images and Chisel, the blockout-based approach. I would like better local control, the ability to expand a generated world more and more while preserving coherence, and the ability to directly import 3D Gaussian Splat scenes to be used as a starting point. I want more ways to shape the result, not just a “prompt and hope” approach. — It is exciting to see this field moving from research and demos toward actual creative workflows.
1:04

Fast Company just published a great piece on World Labs , Fei-Fei Li , Marble, and the idea that spatial intelligence / world models may be one of the next big shifts in AI. I was happy to be quoted in the article, but I also wanted to share more context about my own experience with World Labs and Marble, and why this direction is especially interesting to me. My starting point: volumetric capture — For the past few years I’ve been exploring and using volumetric capture and reconstruction (photogrammetry, NeRFs, 3D Gaussian Splats) mostly capturing locations around Montreal. Alleys, museums, urban interiors. I love every step of it: the capture itself, the pipeline, and what can be done with the output. Turning real spaces into real-time explorable systems. I do this personally, sharing explorations here, and professionally as chief technologist, and co-founder of Dpt. Physical reality + generative manipulation — In my work I’m especially drawn to mixing physical reality with generative and digital manipulation: using physical interfaces (light, clay, ink, ... ) to drive generative AI pipelines, building mixed reality prototypes that reshape your surroundings, or starting from real captured spaces and transforming them using tools like Marble. Like many people, I saw the World Labs announcement on Twitter in September 2024, and Marble when it surfaced in early December. But by then, I already had a sense something was coming. The first conversation — As someone deep into volumetric capture and radiance fields, I obviously knew about Ben Mildenhall and his pioneering work on NeRF. To my surprise, Ben reached out to me in late June 2024. He’d been following some of my experiments and wanted to chat about my process and workflows and how I was using this “stuff” creatively. At that point he didn’t share what he was building, but we had a genuinely great conversation about radiance fields, AI, and my work. He was curious about the creative perspective, not just the technical one. When the World Labs announcement dropped a few months later, it all made sense. I understood what Ben had been working on, and why the creative angle mattered to them. Then in August 2025, he invited me to try the Marble beta, and I’ve been experimenting with it since. Experimenting with Marble — The first thing I used Marble for was materializing scene and world concepts during ideation at the studio, and seeing if and how it could fit into our production pipeline. In parallel, I dove into a series of experiments focused on world manipulation: starting from real captured spaces and transforming them using Marble. I’d already been exploring that idea using img2img diffusion with ControlNet on NeRF renders, real-time video streams, and even mixed reality using headset camera feeds. But Marble brings something different. It generates persistent, spatially cohesive 3D worlds that can be rendered in real time across a wide range of devices. That’s a real shift. Experiment 01: Parallel Realities — The first experiment, Parallel Realities, starts from a volumetric capture of a real location, reconstructed as 3D Gaussian Splats. Using Marble, I generate an alternate version of that same space, something informed by the original architecture: abandoned, nature-reclaimed, alternate era. Then, using Spark (World Labs’ 3D Gaussian Splatting renderer for THREE.js) I make both realities coexist in the same spatial coordinate system. From there, I use a portal UX mechanic to let the user step between the real reconstruction and the Marble-generated version. Experiment 02: Hidden Depth The second experiment, Hidden Depth, does not transform a space as much as expand it. A captured location has a visual boundary (a mural, a doorway, a dark corridor) and Marble generates what exists beyond it. For example: a Montreal alley has a painted mural; step through it and you’re inside a world informed by what is actually depicted there. World Labs showcased part of this work here: And in their Spark 2.0 post: The project page is here: Why this matters to me — Being able to start from a real 3D Gaussian Splat scene and manipulate it with Marble opens up a lot of ideas. The 3DGS pipeline is becoming an increasingly compelling foundation for exploration, experimentation, and storytelling. What matters most to me right now is more control. The more I can steer the generated scene or world, the more useful the tool becomes. I want more features like the already existing multiple input images and Chisel, the blockout-based approach. I would like better local control, the ability to expand a generated world more and more while preserving coherence, and the ability to directly import 3D Gaussian Splat scenes to be used as a starting point. I want more ways to shape the result, not just a “prompt and hope” approach. — It is exciting to see this field moving from research and demos toward actual creative workflows.

Hugues Bruyère

64,336 просмотров • 27 дней назад

I think the Singularity could be BORING We were promised flying cars and warp drives. We got same-day delivery and better autocomplete. And somehow, impossibly, we’re bored by it. This is the Boring Singularity. The idea that the most transformative period in human history will feel, to the people living through it, like a long and uneventful Tuesday. I want to explain why this happens. It comes down to three layers. The first is neurological. The second is architectural. The third is physical. Together they create a perfect storm of invisible progress that our minds are designed to ignore. Layer One: The Neurological Filter Here is a thought experiment. Imagine a caveman breaks his arm. For weeks he is miserable. He cannot hunt, cannot gather, cannot contribute. Then the bone heals. Within days of regaining function, he has completely forgotten the misery. The memory of suffering serves no purpose once the threat has passed. Evolution deleted it so he could return to baseline and focus on survival. We do this with everything. We did it with antibiotics. We did it with smartphones. We will do it with longevity. Psychologists call this hedonic adaptation. The human brain is an adaptation machine that returns us to a baseline level of experience regardless of how much our circumstances improve. And here is the critical finding. It only takes about three months for the “new normal” to cement itself. Any change that plays out over months or years, no matter how revolutionary, simply becomes background noise. Think about what this means for the Singularity. If anti-gravity cars were introduced tomorrow, they would be miraculous for a month, a status symbol for a year, and a frustrating utility that needs maintenance by year three. The internet is a collective telepathic hive mind that moves petabits at lightspeed. It is genuinely god-like power. We experience it as checking emails. The Singularity might already be here. We just cannot feel it because our brains are not designed to feel sustained amazement. They are designed to adapt and move on. Layer Two: The Hidden Infrastructure We expected Blade Runner. Neon towers and chrome robots serving drinks at the bar. What we are actually getting is something I call “Reverse Trantor.” In classic science fiction, advanced civilizations build upward and inward. They create city-planets like Trantor in Foundation or Coruscant in Star Wars, layer upon layer of visible technology. Our trajectory is the opposite. We are pushing the infrastructure outward and downward, into spaces humans never see. Consider where the robots actually are. They are not walking down the street. They are in mines and fulfillment centers and vertical farms. The real automation revolution is happening in dark warehouses where no human needs to flip a light switch. You order a package and it arrives faster than it used to. That is the entire perceptible output of a massive transformation in logistics. That’s not to say that you’ll never see humanoid robots milling around, only that the vast majority of them will be away from the public. The same principle applies to computation itself. In hindsight, it will look like the entire purpose of inventing computers was to run AI, and everything else was just the bootloader. We are heading toward a world where 99% of all CPU and GPU cycles are dedicated to machine-to-machine processes, and less than one percent is for human-facing tasks. The vast majority of the intelligence infrastructure will be completely invisible to us, humming along as background noise. And the really heavy stuff will be in space. Earth has a finite ability to dissipate heat. To run truly massive AI systems, we will likely move the servers to orbital platforms or Lagrange points where they can vent entropy into the void. The megastructures will exist. They will just be invisible points of light, indistinguishable from stars and space dust. This is the architectural reality of the Boring Singularity. The magic gets hidden in the walls and launched into orbit. What remains on Earth is green and quiet and looks suspiciously like a return to the pastoral. That’s not a bad thing, and it’s not to say that we return to a “steady state” equilibrium forever. Layer Three: The Hard Limits The final layer is the most sobering. We are hitting the physical ceiling of discovery itself. The Golden Age of science fiction emerged during a specific historical anomaly. Between 1905 and 1970, in a single human lifetime, we went from the Wright Brothers to the Moon, from Newtonian physics to quantum mechanics and the structure of DNA. That created an expectation of constant improvement. Fundamental discoveries happen every decade. The exponential curve goes up forever. Star Trek promised we would keep finding new energy sources and new physics for centuries. The data suggests otherwise. Research on scientific progress shows that we must double research effort every thirteen years just to maintain the same rate of economic growth. Studies of citation patterns reveal that the disruptiveness of new scientific papers dropped by ninety percent between 1945 and 2010. We are publishing more but saying less. The low-hanging fruit is gone. The cost curve tells the story most clearly. In the 1930s, you could discover a new particle with a tabletop experiment in a university lab for a few thousand dollars. It only took a few days to duplicate the splitting of the atom. To confirm the Higgs Boson, however, we needed the Large Hadron Collider, which cost nearly five billion dollars and took decades to build. The next generation of particle physics might require a hundred billion dollars, or trillions. And the math suggests that to probe the truly fundamental structure of reality at the Planck scale, you would need an accelerator the size of a galaxy. We cannot build that. So physics becomes theoretical not because we lack curiosity, but because we can no longer afford to test our hypotheses. Meanwhile, our imagination has outpaced physical reality. We grew up on fiction that treats the laws of physics as suggestions that can be bypassed with clever engineering. And that felt true (at the time) because we kept finding cool exploits, like fiber optics and nuclear fission. But the speed of light appears to be absolute. Thermodynamics is non-negotiable. We can imagine teleportation and warp drives, but there is no known physics that could enable them. This is the Sigmoid Curve in action. Progress is not an exponential line to infinity. It is an S-curve. We have likely passed the steepest part of fundamental discovery, and we are entering the plateau. The Inverted Star Wars So where does this leave us? I think the best model is actually Star Wars, just inverted. In Star Wars, they have had faster-than-light travel and droids for thousands of years. The technology has faded completely into the background. A hyperdrive failure is treated like a flat tire. It is annoying, not existential. Because the tech tree is fully unlocked, all the drama shifts to politics and governance and ideology. The Empire versus the Republic. Trade routes and treaties and coups. We are heading somewhere similar, with one crucial inversion. Our droids will be smarter, but our ships will be slower. We are likely trapped in this solar system by the speed of light. There is no Outer Rim to escape to if you dislike the politics. But our AI systems will be genuinely superintelligent, an invisible omniscient layer managing supply chains and governance and the allocation of resources. This intensifies the politics because there is no exit valve. We are stuck here with each other and with very powerful tools. The optimistic reading is that this represents maturity. For the last century, technology has moved faster than culture, causing constant anxiety. Future shock, always. If technology moves to a plateau, culture finally has time to catch up. Human decisions, not technological accidents, become what determines history. We stop waiting for a gadget to save us. We realize that if we want a better world, we have to build it with the tools we already have, because no new fundamental laws of reality are coming to rescue us. The Verdict The Boring Singularity is not a prediction of stagnation. Things will still change. We will probably see radical longevity and hyper-efficient energy and algorithmic governance that makes traffic and logistics invisible. It will be, by any historical standard, a utopia of convenience. But it will not feel like the future we were promised. The changes will be incremental enough that our brains adapt before we can appreciate them. The infrastructure will be hidden in warehouses and orbiting platforms we never see. And the truly magical discoveries, the new forces of nature and new physics, may simply be too expensive and complex to pursue. The Singularity is not ending with a bang or a whimper. It is ending with a shrug. And because we are humans, we will probably find something to complain about anyway.
27:09

I think the Singularity could be BORING We were promised flying cars and warp drives. We got same-day delivery and better autocomplete. And somehow, impossibly, we’re bored by it. This is the Boring Singularity. The idea that the most transformative period in human history will feel, to the people living through it, like a long and uneventful Tuesday. I want to explain why this happens. It comes down to three layers. The first is neurological. The second is architectural. The third is physical. Together they create a perfect storm of invisible progress that our minds are designed to ignore. Layer One: The Neurological Filter Here is a thought experiment. Imagine a caveman breaks his arm. For weeks he is miserable. He cannot hunt, cannot gather, cannot contribute. Then the bone heals. Within days of regaining function, he has completely forgotten the misery. The memory of suffering serves no purpose once the threat has passed. Evolution deleted it so he could return to baseline and focus on survival. We do this with everything. We did it with antibiotics. We did it with smartphones. We will do it with longevity. Psychologists call this hedonic adaptation. The human brain is an adaptation machine that returns us to a baseline level of experience regardless of how much our circumstances improve. And here is the critical finding. It only takes about three months for the “new normal” to cement itself. Any change that plays out over months or years, no matter how revolutionary, simply becomes background noise. Think about what this means for the Singularity. If anti-gravity cars were introduced tomorrow, they would be miraculous for a month, a status symbol for a year, and a frustrating utility that needs maintenance by year three. The internet is a collective telepathic hive mind that moves petabits at lightspeed. It is genuinely god-like power. We experience it as checking emails. The Singularity might already be here. We just cannot feel it because our brains are not designed to feel sustained amazement. They are designed to adapt and move on. Layer Two: The Hidden Infrastructure We expected Blade Runner. Neon towers and chrome robots serving drinks at the bar. What we are actually getting is something I call “Reverse Trantor.” In classic science fiction, advanced civilizations build upward and inward. They create city-planets like Trantor in Foundation or Coruscant in Star Wars, layer upon layer of visible technology. Our trajectory is the opposite. We are pushing the infrastructure outward and downward, into spaces humans never see. Consider where the robots actually are. They are not walking down the street. They are in mines and fulfillment centers and vertical farms. The real automation revolution is happening in dark warehouses where no human needs to flip a light switch. You order a package and it arrives faster than it used to. That is the entire perceptible output of a massive transformation in logistics. That’s not to say that you’ll never see humanoid robots milling around, only that the vast majority of them will be away from the public. The same principle applies to computation itself. In hindsight, it will look like the entire purpose of inventing computers was to run AI, and everything else was just the bootloader. We are heading toward a world where 99% of all CPU and GPU cycles are dedicated to machine-to-machine processes, and less than one percent is for human-facing tasks. The vast majority of the intelligence infrastructure will be completely invisible to us, humming along as background noise. And the really heavy stuff will be in space. Earth has a finite ability to dissipate heat. To run truly massive AI systems, we will likely move the servers to orbital platforms or Lagrange points where they can vent entropy into the void. The megastructures will exist. They will just be invisible points of light, indistinguishable from stars and space dust. This is the architectural reality of the Boring Singularity. The magic gets hidden in the walls and launched into orbit. What remains on Earth is green and quiet and looks suspiciously like a return to the pastoral. That’s not a bad thing, and it’s not to say that we return to a “steady state” equilibrium forever. Layer Three: The Hard Limits The final layer is the most sobering. We are hitting the physical ceiling of discovery itself. The Golden Age of science fiction emerged during a specific historical anomaly. Between 1905 and 1970, in a single human lifetime, we went from the Wright Brothers to the Moon, from Newtonian physics to quantum mechanics and the structure of DNA. That created an expectation of constant improvement. Fundamental discoveries happen every decade. The exponential curve goes up forever. Star Trek promised we would keep finding new energy sources and new physics for centuries. The data suggests otherwise. Research on scientific progress shows that we must double research effort every thirteen years just to maintain the same rate of economic growth. Studies of citation patterns reveal that the disruptiveness of new scientific papers dropped by ninety percent between 1945 and 2010. We are publishing more but saying less. The low-hanging fruit is gone. The cost curve tells the story most clearly. In the 1930s, you could discover a new particle with a tabletop experiment in a university lab for a few thousand dollars. It only took a few days to duplicate the splitting of the atom. To confirm the Higgs Boson, however, we needed the Large Hadron Collider, which cost nearly five billion dollars and took decades to build. The next generation of particle physics might require a hundred billion dollars, or trillions. And the math suggests that to probe the truly fundamental structure of reality at the Planck scale, you would need an accelerator the size of a galaxy. We cannot build that. So physics becomes theoretical not because we lack curiosity, but because we can no longer afford to test our hypotheses. Meanwhile, our imagination has outpaced physical reality. We grew up on fiction that treats the laws of physics as suggestions that can be bypassed with clever engineering. And that felt true (at the time) because we kept finding cool exploits, like fiber optics and nuclear fission. But the speed of light appears to be absolute. Thermodynamics is non-negotiable. We can imagine teleportation and warp drives, but there is no known physics that could enable them. This is the Sigmoid Curve in action. Progress is not an exponential line to infinity. It is an S-curve. We have likely passed the steepest part of fundamental discovery, and we are entering the plateau. The Inverted Star Wars So where does this leave us? I think the best model is actually Star Wars, just inverted. In Star Wars, they have had faster-than-light travel and droids for thousands of years. The technology has faded completely into the background. A hyperdrive failure is treated like a flat tire. It is annoying, not existential. Because the tech tree is fully unlocked, all the drama shifts to politics and governance and ideology. The Empire versus the Republic. Trade routes and treaties and coups. We are heading somewhere similar, with one crucial inversion. Our droids will be smarter, but our ships will be slower. We are likely trapped in this solar system by the speed of light. There is no Outer Rim to escape to if you dislike the politics. But our AI systems will be genuinely superintelligent, an invisible omniscient layer managing supply chains and governance and the allocation of resources. This intensifies the politics because there is no exit valve. We are stuck here with each other and with very powerful tools. The optimistic reading is that this represents maturity. For the last century, technology has moved faster than culture, causing constant anxiety. Future shock, always. If technology moves to a plateau, culture finally has time to catch up. Human decisions, not technological accidents, become what determines history. We stop waiting for a gadget to save us. We realize that if we want a better world, we have to build it with the tools we already have, because no new fundamental laws of reality are coming to rescue us. The Verdict The Boring Singularity is not a prediction of stagnation. Things will still change. We will probably see radical longevity and hyper-efficient energy and algorithmic governance that makes traffic and logistics invisible. It will be, by any historical standard, a utopia of convenience. But it will not feel like the future we were promised. The changes will be incremental enough that our brains adapt before we can appreciate them. The infrastructure will be hidden in warehouses and orbiting platforms we never see. And the truly magical discoveries, the new forces of nature and new physics, may simply be too expensive and complex to pursue. The Singularity is not ending with a bang or a whimper. It is ending with a shrug. And because we are humans, we will probably find something to complain about anyway.

David Shapiro (L/0)

14,113 просмотров • 5 месяцев назад

People are undoubtedly a little alarmed at having unwittingly helped build a 3D map of the world for Niantic by contributing 30 billion crowdsourced images. I interviewed Niantic's CTO Brian McClendon about exactly this in a TED interview last year -- he's also the guy who co-created Google Earth. But let's put it in perspective. Pokestop data isn't what you think it is. It's not a surveillance panopticon of your neighborhood. These are static captures of parks, statues, murals, landmarks -- the places people congregate. Brian described it as "building the map from the bottom up, from the locations where people spend time." Think of these 20 million waypoints as basically the inverse of what Google mapped with Street View. Google mapped the drivable streets. Niantic mapped where people actually hang out. Cool data, genuinely useful for visual positioning -- but very different from what the headlines imply. And lest we forget that Niantic is just one of many companies quietly building their own map of the world right now -- and they're all capturing different facets of reality: >🚶 person-level: Axon body cams on hundreds of thousands of officers. Meta Ray-Ban glasses capturing first-person POV at scale -- overseas operators reviewing images every time someone says "Hey Meta." > 🚗 vehicle-level: Tesla dashcams on every car in the fleet, massive onboard compute extracting and distilling data to the cloud. Waymo with cm-accurate 3D maps of every city they operate in. Fleet telematics cameras on delivery vehicles globally. > 🏠 street & home-level: Flock Safety deploying CCTV across neighborhoods and cities. Amazon with Ring cameras on every doorstep and mailroom (recently got dragged over that Super Bowl commercial about fusing all these cams together to find your dog) plus dashcams on every Prime delivery van. Roomba mapping your floor plan every time it vacuums -- Amazon wanted that data badly enough to try acquiring iRobot for $1.7B before regulators shut it down. > 🥽 headset-level: Apple Vision Pro and Meta Quest build a 3D model of whatever room you're in every time you put them on. Between Ring, Roomba, and your headset, your entire home is being spatially understood by at least three different companies. >📍platform-level: Google with Street View cars, aerial planes, satellite imagery, and live location from every Android phone in your pocket. Apple doing the same with mapping cars AND every LiDAR iPhone is quietly a 3D scanner. And yeah, despite the "Apple is too privacy-conscious" narrative, they're collecting location data too. >🏃 trajectory-level: Strava mapped every running and cycling trail on Earth -- and accidentally exposed secret military bases in Afghanistan and Syria because soldiers logged their jogs. When you aggregate enough individual trajectories, patterns emerge that were never supposed to be visible. > 🛰️ space-level: Planet Labs imaging the entire Earth's landmass every single day from orbit. Vantor capturing it in higher detail. Iceye doing it in 3D using SAR. If something changes anywhere on the planet -- a building goes up, a forest burns down, a military convoy moves -- before-and-after imagery within 24 hours. Fused together -- we have everything from body cam to dashcam to doorbell to phone to satellite -- every layer of physical reality is being mapped by somebody right now. Different sensors, different angles, different purposes. Same pattern. The interesting part is how they incentivize it. Google spends billions. Mapillary tried altruism. Hivemapper grinds with crypto. Pokémon GO cracked something none of them could: a game mechanic that subsidizes the scanning behavior. You're not building a map. You're catching pokemon. The map is just a side effect. 3D scanning is still a niche hobby for reality capture nerds like me. The moment somebody gamifies dense 3D capture at scale -- not posed photos but actual geometry -- that's when this blows wide open. Niantic sold the games for $3.5B but kept the spatial platform, with a data-sharing agreement in place. One team makes the game great, the other builds the spatial infrastructure underneath. Incentives finally aligned. Gaming is becoming a way for humans to contribute real-world trajectories that help physical AI learn about the real world. Google does it with live traffic. Tesla does it with autopilot. The mechanic is different but the pattern is identical -- and most people are already part of at least one -- if not a majority -- of these datasets whether they realize it or not.
0:37

People are undoubtedly a little alarmed at having unwittingly helped build a 3D map of the world for Niantic by contributing 30 billion crowdsourced images. I interviewed Niantic's CTO Brian McClendon about exactly this in a TED interview last year -- he's also the guy who co-created Google Earth. But let's put it in perspective. Pokestop data isn't what you think it is. It's not a surveillance panopticon of your neighborhood. These are static captures of parks, statues, murals, landmarks -- the places people congregate. Brian described it as "building the map from the bottom up, from the locations where people spend time." Think of these 20 million waypoints as basically the inverse of what Google mapped with Street View. Google mapped the drivable streets. Niantic mapped where people actually hang out. Cool data, genuinely useful for visual positioning -- but very different from what the headlines imply. And lest we forget that Niantic is just one of many companies quietly building their own map of the world right now -- and they're all capturing different facets of reality: >🚶 person-level: Axon body cams on hundreds of thousands of officers. Meta Ray-Ban glasses capturing first-person POV at scale -- overseas operators reviewing images every time someone says "Hey Meta." > 🚗 vehicle-level: Tesla dashcams on every car in the fleet, massive onboard compute extracting and distilling data to the cloud. Waymo with cm-accurate 3D maps of every city they operate in. Fleet telematics cameras on delivery vehicles globally. > 🏠 street & home-level: Flock Safety deploying CCTV across neighborhoods and cities. Amazon with Ring cameras on every doorstep and mailroom (recently got dragged over that Super Bowl commercial about fusing all these cams together to find your dog) plus dashcams on every Prime delivery van. Roomba mapping your floor plan every time it vacuums -- Amazon wanted that data badly enough to try acquiring iRobot for $1.7B before regulators shut it down. > 🥽 headset-level: Apple Vision Pro and Meta Quest build a 3D model of whatever room you're in every time you put them on. Between Ring, Roomba, and your headset, your entire home is being spatially understood by at least three different companies. >📍platform-level: Google with Street View cars, aerial planes, satellite imagery, and live location from every Android phone in your pocket. Apple doing the same with mapping cars AND every LiDAR iPhone is quietly a 3D scanner. And yeah, despite the "Apple is too privacy-conscious" narrative, they're collecting location data too. >🏃 trajectory-level: Strava mapped every running and cycling trail on Earth -- and accidentally exposed secret military bases in Afghanistan and Syria because soldiers logged their jogs. When you aggregate enough individual trajectories, patterns emerge that were never supposed to be visible. > 🛰️ space-level: Planet Labs imaging the entire Earth's landmass every single day from orbit. Vantor capturing it in higher detail. Iceye doing it in 3D using SAR. If something changes anywhere on the planet -- a building goes up, a forest burns down, a military convoy moves -- before-and-after imagery within 24 hours. Fused together -- we have everything from body cam to dashcam to doorbell to phone to satellite -- every layer of physical reality is being mapped by somebody right now. Different sensors, different angles, different purposes. Same pattern. The interesting part is how they incentivize it. Google spends billions. Mapillary tried altruism. Hivemapper grinds with crypto. Pokémon GO cracked something none of them could: a game mechanic that subsidizes the scanning behavior. You're not building a map. You're catching pokemon. The map is just a side effect. 3D scanning is still a niche hobby for reality capture nerds like me. The moment somebody gamifies dense 3D capture at scale -- not posed photos but actual geometry -- that's when this blows wide open. Niantic sold the games for $3.5B but kept the spatial platform, with a data-sharing agreement in place. One team makes the game great, the other builds the spatial infrastructure underneath. Incentives finally aligned. Gaming is becoming a way for humans to contribute real-world trajectories that help physical AI learn about the real world. Google does it with live traffic. Tesla does it with autopilot. The mechanic is different but the pattern is identical -- and most people are already part of at least one -- if not a majority -- of these datasets whether they realize it or not.

Bilawal Sidhu

203,550 просмотров • 4 месяцев назад

If an AI can control 1,000 robots to perform 1 million skills in 1 billion different simulations, then it may "just work" in our real world, which is simply another point in the vast space of possible realities. This is the fundamental principle behind why simulation works so effectively for robotics. Real-world teleoperation data scales linearly with human time (< 24 hrs/robot/day). Sim data scales exponentially with compute. There are 3 big trends for simulators in the near future: 1. Massive parallelization on large clusters. Physics equations are "just" matrix math at their core. I hear GPUs are good at matrix math 🔥. One can run 100K copies of simulation on a single GPU. To put this number in perspective: 1 hour of wallclock compute time gives a robot 10 years (!!) of training experience. That's how Neo was able to learn martial arts in a blink of an eye in the Matrix Dojo. 2. Generative graphics pipeline. Traditionally, simulators require a huge amount of manual effort from artists: 3D assets, textures, scene layouts, etc. But every component in the workflow can be automated: text-to-image, text-to-3D mesh, and LLMs that write Universal Scene Description (USD) files as a coding exercise. RoboCasa is one example of a prior work. 3. End2end neural net that acts as simulator itself. This is still bluesky research and quite far from replacing a graphics pipeline, but we are seeing some exciting signs-of-life based on video gen models: Sora, Veo2, CogVideoX, Hunyuan (text-to-video); and action-driven world models: GameNGen, Oasis, Genie-2, etc. Genesis does great on (1) for certain tasks, shows good promises on (2), and could become a data generation tool for reaching (3). Its sim2real capabilities for locomotion are good, but there's still a long way to go for contact-rich, dexterous manipulation. It shows a bold vision and is on the right path to providing a virtual cradle for embodied AI. It is open-source and puts a streamlined user journey at the front and center. I had the privilege to know Zhou Xian and play a small part in his project since a year ago. Xian has been crunching code non-stop on Genesis with a very small group of core devs. He often replied to my messages at 3 am. Zhenjia Xu from our GEAR team helped with sim2real experiments in his spare time. Genesis is truly a grassroot effort with an intense focus on quality engineering. Nothing gives me more joy than seeing the simulation ecosystem bloom. Robotics should be a moonshot initiative owned by all of humanity. Congratulations.
3:26

If an AI can control 1,000 robots to perform 1 million skills in 1 billion different simulations, then it may "just work" in our real world, which is simply another point in the vast space of possible realities. This is the fundamental principle behind why simulation works so effectively for robotics. Real-world teleoperation data scales linearly with human time (< 24 hrs/robot/day). Sim data scales exponentially with compute. There are 3 big trends for simulators in the near future: 1. Massive parallelization on large clusters. Physics equations are "just" matrix math at their core. I hear GPUs are good at matrix math 🔥. One can run 100K copies of simulation on a single GPU. To put this number in perspective: 1 hour of wallclock compute time gives a robot 10 years (!!) of training experience. That's how Neo was able to learn martial arts in a blink of an eye in the Matrix Dojo. 2. Generative graphics pipeline. Traditionally, simulators require a huge amount of manual effort from artists: 3D assets, textures, scene layouts, etc. But every component in the workflow can be automated: text-to-image, text-to-3D mesh, and LLMs that write Universal Scene Description (USD) files as a coding exercise. RoboCasa is one example of a prior work. 3. End2end neural net that acts as simulator itself. This is still bluesky research and quite far from replacing a graphics pipeline, but we are seeing some exciting signs-of-life based on video gen models: Sora, Veo2, CogVideoX, Hunyuan (text-to-video); and action-driven world models: GameNGen, Oasis, Genie-2, etc. Genesis does great on (1) for certain tasks, shows good promises on (2), and could become a data generation tool for reaching (3). Its sim2real capabilities for locomotion are good, but there's still a long way to go for contact-rich, dexterous manipulation. It shows a bold vision and is on the right path to providing a virtual cradle for embodied AI. It is open-source and puts a streamlined user journey at the front and center. I had the privilege to know Zhou Xian and play a small part in his project since a year ago. Xian has been crunching code non-stop on Genesis with a very small group of core devs. He often replied to my messages at 3 am. Zhenjia Xu from our GEAR team helped with sim2real experiments in his spare time. Genesis is truly a grassroot effort with an intense focus on quality engineering. Nothing gives me more joy than seeing the simulation ecosystem bloom. Robotics should be a moonshot initiative owned by all of humanity. Congratulations.

Jim Fan

157,165 просмотров • 1 год назад

Congratulations ZackeryFox🌏2️⃣👽 for your 2D concept art submission our team then took & brought to life in 3D for #Earth2🌍✌️This skin took longer given the detail & quality of the modelling but was definitely worth it! Read below for more important details & updates: The first variations of #CybercoreX will drop in about 8 hours. As mentioned in the article when we first started releasing skins, after #E2V1 officially launches we will be cutting back on #LimitedEdition skins & will predominantly focus on unlimited skins. In some cases these future unlimited skins will not be available for resale. This was always the plan as these early #LimitedEdition skins were released especially for early supporters. We will still release #LimitedEditions from time to time, but if & when we do so it will be at higher prices - which has also been mentioned numerous times in the past but I am just providing this as a reminder. I think we have 2-3 Limited Edition skins remaining before we transition into the standard skin system with unlimited skins & periodic availability - so there are not many left and these final Limited Edition drops will contain a few more rarer variations. #CybercoreX took a lot of additional work & team members strongly suggested the skin should be sold at a premium price, however I decided to keep the pricing in line with the other pre-#E2V1 Limited Edition releases. It is also important to note that as with many of the other Limited Edition skin releases with guns, the assault rifle & pistol will be sold separately. I'd also like to provide a quick update on the #E2V1 public release. The results from the initial testing group have been good so far, there are some issues we are working on fixing but nothing major. Considering it was our very first ever software release of the #Earth2 Launcher AND #E2V1, anyone with real experience releasing major software products would agree it went very smooth. Some #Players downloaded the launcher and were inside #E2V1 within 20 mins without any issues! You have major game or software developers releasing products after decades of experience & a million things go wrong .. often simple things .. our release was the dream type scenario & a real testament to our teams dedication, talent & ability! I'd like to include the option to spawn #avatars for the pre-alpha public release. The team has implemented the majority of this system with a few things remaining, which are actively being worked on at present. I just want to be clear that this NOT the avatar spawn system you have seen in demo videos to date where we just randomly spawn an avatar anywhere, that is simple & we could easily release that specific feature but it was only for demo purposes to showcase avatars can spawn anywhere in the world. To be transparent, what we are working to release now is the actual avatar spawn system intended for long term use where Players need to synthesise their #avatar from their #Mentar on their property to begin its actual life inside #E2V1. It's important to point out this involves numerous API integration, UI/UX updates and is not a simple random spawn. We anticipate there will almost surely be a number of potential bugs with this release so the plan is to first have it released to testers (WITH BUGS EXPECTED!) then stabilise & prepare for the public #E2V1 release. If you see people publishing footage of bugs online, remember we are in pre-alpha which we released to testers less than a month ago & testers are there to help us test and find issues we need to resolve. Our team is committed to fixing any major problems as they arise to improve the platform and keep moving forward. We should have avatars into the hands of testers this week. Remember, once you are inside your avatar you will not have that free camera flying ability. This makes teleportation very important for #Players who want to travel to different locations. So we will also need to start testing the teleportation system, and there is some potentially exciting news I have been keeping close to my chest which will soon be revealed. As always, thanks for your patience and support! Remember, we are literally less than one month since releasing #E2V1 to testers (less than a month ago there were still those who claimed we pre-rendered everything) so these are still early days! It may feel a little slow at first but we plan to release things to do inside of #E2V1 progressively and before you know it there will be too much to do! #Earth2 #Metaverse #E2V1
3:47

Congratulations ZackeryFox🌏2️⃣👽 for your 2D concept art submission our team then took & brought to life in 3D for #Earth2🌍✌️This skin took longer given the detail & quality of the modelling but was definitely worth it! Read below for more important details & updates: The first variations of #CybercoreX will drop in about 8 hours. As mentioned in the article when we first started releasing skins, after #E2V1 officially launches we will be cutting back on #LimitedEdition skins & will predominantly focus on unlimited skins. In some cases these future unlimited skins will not be available for resale. This was always the plan as these early #LimitedEdition skins were released especially for early supporters. We will still release #LimitedEditions from time to time, but if & when we do so it will be at higher prices - which has also been mentioned numerous times in the past but I am just providing this as a reminder. I think we have 2-3 Limited Edition skins remaining before we transition into the standard skin system with unlimited skins & periodic availability - so there are not many left and these final Limited Edition drops will contain a few more rarer variations. #CybercoreX took a lot of additional work & team members strongly suggested the skin should be sold at a premium price, however I decided to keep the pricing in line with the other pre-#E2V1 Limited Edition releases. It is also important to note that as with many of the other Limited Edition skin releases with guns, the assault rifle & pistol will be sold separately. I'd also like to provide a quick update on the #E2V1 public release. The results from the initial testing group have been good so far, there are some issues we are working on fixing but nothing major. Considering it was our very first ever software release of the #Earth2 Launcher AND #E2V1, anyone with real experience releasing major software products would agree it went very smooth. Some #Players downloaded the launcher and were inside #E2V1 within 20 mins without any issues! You have major game or software developers releasing products after decades of experience & a million things go wrong .. often simple things .. our release was the dream type scenario & a real testament to our teams dedication, talent & ability! I'd like to include the option to spawn #avatars for the pre-alpha public release. The team has implemented the majority of this system with a few things remaining, which are actively being worked on at present. I just want to be clear that this NOT the avatar spawn system you have seen in demo videos to date where we just randomly spawn an avatar anywhere, that is simple & we could easily release that specific feature but it was only for demo purposes to showcase avatars can spawn anywhere in the world. To be transparent, what we are working to release now is the actual avatar spawn system intended for long term use where Players need to synthesise their #avatar from their #Mentar on their property to begin its actual life inside #E2V1. It's important to point out this involves numerous API integration, UI/UX updates and is not a simple random spawn. We anticipate there will almost surely be a number of potential bugs with this release so the plan is to first have it released to testers (WITH BUGS EXPECTED!) then stabilise & prepare for the public #E2V1 release. If you see people publishing footage of bugs online, remember we are in pre-alpha which we released to testers less than a month ago & testers are there to help us test and find issues we need to resolve. Our team is committed to fixing any major problems as they arise to improve the platform and keep moving forward. We should have avatars into the hands of testers this week. Remember, once you are inside your avatar you will not have that free camera flying ability. This makes teleportation very important for #Players who want to travel to different locations. So we will also need to start testing the teleportation system, and there is some potentially exciting news I have been keeping close to my chest which will soon be revealed. As always, thanks for your patience and support! Remember, we are literally less than one month since releasing #E2V1 to testers (less than a month ago there were still those who claimed we pre-rendered everything) so these are still early days! It may feel a little slow at first but we plan to release things to do inside of #E2V1 progressively and before you know it there will be too much to do! #Earth2 #Metaverse #E2V1

Shane Isaac 🌍2️⃣

14,098 просмотров • 1 год назад