Загрузка видео...

Не удалось загрузить видео

На главную

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme....

davsca1's profile picture

Davide Scaramuzza

18,917 subscribers

12,692 просмотров • 2 лет назад •via X (Twitter)

#PAMI
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 1

Фото профиля David Jin
David Jin2 лет назад

Sorry to ask a question before reading the paper. May I ask if I need to train for a different data set or it will work with your pretrained checkpoints?

Похожие видео

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.
5:18

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

AK

280,570 просмотров • 3 лет назад

We are excited to share our #CORL2024 paper on learning quadrotor obstacle avoidance from the visual stream of a single #eventcamera! Trained entirely in simulation! We demonstrate obstacle avoidance both in the dark and in a forest up to 5m/s. PDF: Video: Project page: Event cameras are sensors that output per-pixel-level intensity changes at microsecond latency resolution; they feature nearly zero motion blur and high dynamic range but produce a very large volume of events under significant ego-motion and further lack a high-fidelity continuous-time sensor model in simulation, making direct #sim2real transfer not possible. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance policy with “approximated” simulated events and then fine-tune the perception component with limited events-and-depth real-world data. This technique bridges the sim2real gap for #eventcameras! As at the current state, there is no continuous-time sensor model for event cameras, we hope that this work can finally spur future research leveraging simulation for training event-vision-based policies to create faster, agile robots! Kudos to Anish Bhattacharya, @marcocannic, Vijay Kumar Nikolai Matni UZH Science University of Zurich UZH Space Hub UZH IfI European Research Council (ERC) GRASP Laboratory Penn Engineering
1:25

We are excited to share our #CORL2024 paper on learning quadrotor obstacle avoidance from the visual stream of a single #eventcamera! Trained entirely in simulation! We demonstrate obstacle avoidance both in the dark and in a forest up to 5m/s. PDF: Video: Project page: Event cameras are sensors that output per-pixel-level intensity changes at microsecond latency resolution; they feature nearly zero motion blur and high dynamic range but produce a very large volume of events under significant ego-motion and further lack a high-fidelity continuous-time sensor model in simulation, making direct #sim2real transfer not possible. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance policy with “approximated” simulated events and then fine-tune the perception component with limited events-and-depth real-world data. This technique bridges the sim2real gap for #eventcameras! As at the current state, there is no continuous-time sensor model for event cameras, we hope that this work can finally spur future research leveraging simulation for training event-vision-based policies to create faster, agile robots! Kudos to Anish Bhattacharya, @marcocannic, Vijay Kumar Nikolai Matni UZH Science University of Zurich UZH Space Hub UZH IfI European Research Council (ERC) GRASP Laboratory Penn Engineering

Davide Scaramuzza

17,263 просмотров • 2 лет назад

Image-to-Pixel 3.0 is out 🎨 Turn images and video into pixel art right in your browser. Free, MIT open source. Try it: Or use it in code: npm i image-to-pixel
1:34

Image-to-Pixel 3.0 is out 🎨 Turn images and video into pixel art right in your browser. Free, MIT open source. Try it: Or use it in code: npm i image-to-pixel

Tezumie

169,626 просмотров • 17 дней назад

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)
0:43

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 просмотров • 1 год назад

Get Ready to Earn $PIXEL in the Forgotten Runiverse! ✨ Use Quanta to claim $PIXEL ✨ Use $PIXEL in-game to purchase Mana and Boosts ✨ Get exclusive Pixels themed rewards Start early and earn Quanta now! 🏃 The Pixels Partner Token Event is coming soon.
0:21

Get Ready to Earn $PIXEL in the Forgotten Runiverse! ✨ Use Quanta to claim $PIXEL ✨ Use $PIXEL in-game to purchase Mana and Boosts ✨ Get exclusive Pixels themed rewards Start early and earn Quanta now! 🏃 The Pixels Partner Token Event is coming soon.

The Forgotten Runiverse

97,961 просмотров • 1 год назад

Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where we make linear attention sparse in space, recurrent in time, and parallel in training, enabling the first purely-linear-attention-based neural network for asynchronous object detection with #EventCameras, outperforming the previous best asynchronous method with 20x less computation with truly event-by-event inference on CPU! Code released! Paper: Code: Video: Event cameras promise extremely low-latency vision, but to fully exploit them, the neural network must be low-latency too. We introduce #SpatiallySparseLinearAttention (#SSLA) for asynchronous object detection directly from raw events. Linear attention is particularly appealing for event cameras: it can be trained efficiently in parallel on long event sequences, while at inference it operates recurrently, updating its prediction every time a new event arrives. The problem is that conventional linear attention updates its entire state for every event. For object detection, where fine spatial resolution matters, this quickly becomes expensive. Our key idea is simple: an event only carries information about a small spatial region, so why update the entire spatial state? SSLA updates only the relevant parts of the state, enabling fine-grained spatial representations while keeping per-event computation low. We achieve: - >20× lower per-event computation than the strongest prior asynchronous baseline - State-of-the-art accuracy among asynchronous object detection methods - Truly event-by-event inference on CPU, designed to preserve the latency advantage of event cameras Come to our poster on Friday September 11, 2026 from 4-6pm at ExHall #389 Reference: Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention ECCV, 2026 Prophesee SynSense University of Zurich UZH Science European Research Council (ERC) UZHai UZH IfI Tesla BYD #EventCameras #ComputerVision #Robotics #DeepLearning #NeuromorphicVision #AI
2:40

Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where we make linear attention sparse in space, recurrent in time, and parallel in training, enabling the first purely-linear-attention-based neural network for asynchronous object detection with #EventCameras, outperforming the previous best asynchronous method with 20x less computation with truly event-by-event inference on CPU! Code released! Paper: Code: Video: Event cameras promise extremely low-latency vision, but to fully exploit them, the neural network must be low-latency too. We introduce #SpatiallySparseLinearAttention (#SSLA) for asynchronous object detection directly from raw events. Linear attention is particularly appealing for event cameras: it can be trained efficiently in parallel on long event sequences, while at inference it operates recurrently, updating its prediction every time a new event arrives. The problem is that conventional linear attention updates its entire state for every event. For object detection, where fine spatial resolution matters, this quickly becomes expensive. Our key idea is simple: an event only carries information about a small spatial region, so why update the entire spatial state? SSLA updates only the relevant parts of the state, enabling fine-grained spatial representations while keeping per-event computation low. We achieve: - >20× lower per-event computation than the strongest prior asynchronous baseline - State-of-the-art accuracy among asynchronous object detection methods - Truly event-by-event inference on CPU, designed to preserve the latency advantage of event cameras Come to our poster on Friday September 11, 2026 from 4-6pm at ExHall #389 Reference: Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention ECCV, 2026 Prophesee SynSense University of Zurich UZH Science European Research Council (ERC) UZHai UZH IfI Tesla BYD #EventCameras #ComputerVision #Robotics #DeepLearning #NeuromorphicVision #AI

Davide Scaramuzza

52,969 просмотров • 2 месяцев назад

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.
0:24

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

AK

20,668 просмотров • 2 лет назад

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.
0:29

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

MrNeRF

10,968 просмотров • 1 год назад

🔥Your Videos Just Became 4D Worlds (1/2) #OmniX predicts 3D trajectories for every pixel across any viewpoint and time - all in one forward pass. Our key insight: 3D motion is low-rank, so a small set of dynamic tokens is enough to track everything. Code is here👇 #ECCV'26
0:21

🔥Your Videos Just Became 4D Worlds (1/2) #OmniX predicts 3D trajectories for every pixel across any viewpoint and time - all in one forward pass. Our key insight: 3D motion is low-rank, so a small set of dynamic tokens is enough to track everything. Code is here👇 #ECCV'26

Tengfei Wang

32,106 просмотров • 3 месяцев назад

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.
0:33

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

AK

122,567 просмотров • 3 лет назад

OPTICAL AIDS CAUSE 28 DAY MONTHS? NCS does not recommend that we use optical aids for a moonsighting system in Britain for many reasons. The primary reason is that we are seeking a unified solution, and unity is in the sunnah - local, naked eye sightings. Not everybody believes that optical aids are permissible, but naked eye sightings are wholly uncontroversial. Another issue is that optical aids would lead to more and more examples of the moon being “seen” a day early, and out of phase, on the 28th of a lunar month. This has ALREADY HAPPENED to those who use optical aids in the UK - yet has never happened to naked eye moonsighters for over 100 months. Whilst there are procedures from the sunnah of what to do if you were to see a moon on the 28th, we cannot really justify asking people to end their fast a day early on the 28th because of a moon which is invisible to the eye and only visible through optical aids. Hence our stance. Remember - we ourselves use optical aids - and had them on our livestream. We are experienced in them, and it is from our experience we share with you that we have found that it would leave to problems. In any case why are you in a rush to end Ramadan, using technology, and away from the sunnah?
1:02

OPTICAL AIDS CAUSE 28 DAY MONTHS? NCS does not recommend that we use optical aids for a moonsighting system in Britain for many reasons. The primary reason is that we are seeking a unified solution, and unity is in the sunnah - local, naked eye sightings. Not everybody believes that optical aids are permissible, but naked eye sightings are wholly uncontroversial. Another issue is that optical aids would lead to more and more examples of the moon being “seen” a day early, and out of phase, on the 28th of a lunar month. This has ALREADY HAPPENED to those who use optical aids in the UK - yet has never happened to naked eye moonsighters for over 100 months. Whilst there are procedures from the sunnah of what to do if you were to see a moon on the 28th, we cannot really justify asking people to end their fast a day early on the 28th because of a moon which is invisible to the eye and only visible through optical aids. Hence our stance. Remember - we ourselves use optical aids - and had them on our livestream. We are experienced in them, and it is from our experience we share with you that we have found that it would leave to problems. In any case why are you in a rush to end Ramadan, using technology, and away from the sunnah?

New Crescent Society

18,343 просмотров • 7 месяцев назад

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.
0:26

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

AK

76,380 просмотров • 3 лет назад

Introducing Continuous Thought Machines New Blog: Modern AI is powerful, but it’s still distinct from human-like flexible intelligence. We believe neural timing is key. Our Continuous Thought Machine is built from the ground up to use neural dynamics as a powerful representation for intelligence. Thought takes time, and reasoning is a process. Biological brains inspire us with their complex neural activity, where neural timing is critical to intelligence. We’re exploring how to bring that power to AI. The Continuous Thought Machine (CTM) incorporates neuron-level temporal processing and neural synchronization, moving beyond current AI limitations. Our approach has two core innovations: (1) neuron-level temporal processing, where each neuron uses unique parameters to process a history of incoming signals for fine-grained temporal dynamics, and (2) neural synchronization, used as a direct latent representation to modulate data and produce outputs, encoding information directly in the timing of neural activity. Learn more about our approach: Interactive Report: Full Paper: GitHub :
0:57

Introducing Continuous Thought Machines New Blog: Modern AI is powerful, but it’s still distinct from human-like flexible intelligence. We believe neural timing is key. Our Continuous Thought Machine is built from the ground up to use neural dynamics as a powerful representation for intelligence. Thought takes time, and reasoning is a process. Biological brains inspire us with their complex neural activity, where neural timing is critical to intelligence. We’re exploring how to bring that power to AI. The Continuous Thought Machine (CTM) incorporates neuron-level temporal processing and neural synchronization, moving beyond current AI limitations. Our approach has two core innovations: (1) neuron-level temporal processing, where each neuron uses unique parameters to process a history of incoming signals for fine-grained temporal dynamics, and (2) neural synchronization, used as a direct latent representation to modulate data and produce outputs, encoding information directly in the timing of neural activity. Learn more about our approach: Interactive Report: Full Paper: GitHub :

Sakana AI

290,107 просмотров • 1 год назад

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.
7:12

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 просмотров • 3 лет назад

Our token, $PIXEL, is officially live on Binance, the world’s largest crypto exchange 🔥 We’ve been working towards this moment for over 2 years, and we couldn’t be more excited to share this massive milestone with you all. This is just the beginning and the best is yet to come for the Pixels community. New here? Welcome to Pixels: Pixels is a free-to-play farming MMO with a play-to-earn (P2E) model. Gameplay revolves around resource management, strategic exploration and community building. Over the past year, we’ve grown to one of Web3’s largest games, with daily active players of 180k+. At Pixels, we build with our community. We're far from a traditional game studio, and our development process reflects that. Over the past 2 years, we’ve had 60+ public updates. This approach enables us to iterate and experiment more quickly, giving players a true sense of ownership over the game's direction. When you join our community, you are not just a player of Pixels, but a creator of it. Our Mission: We’re excited by the potential of Web3 technology and its ability to significantly enhance the gaming experience. We aim to experiment, build, and iterate on the features that make Web3 gaming unique. We’re here to set a new standard. Here are some of the interesting areas we've leaned into: ➡️ P2E: earn $PIXEL by participating in gameplay ➡️ Guilds: friendtech + competition ➡️ On-chain Reputation & Player Personas: creating a sustainable ecosystem of real players ➡️ Interoperability: using on-chain data to bring communities together We're eager to explore more possibilities in this space and push Web3 even further. Meet $PIXEL ✨ $PIXEL is an in-game utility token. It allows players to enhance their gameplay experience in various ways: ➡️ Purchase VIP memberships ➡️ Create and join guilds ➡️ Mint a Pixels Pet ➡️ Upgrade lands We believe Web3 games don’t need to look exactly like Web2 games. However, P2E formats need sustainable structures and economics. With that, we’ve focused heavily on persona types: why would each user want to use this token? We want to ensure we’ve created an environment where first: ➡️ Our game is providing real enjoyment and entertainment to our users ➡️ Our currency can help amplify that experience without affecting the experience of other users How do I get $PIXEL? Users can earn $PIXEL in-game right now. Head over to Buck’s Galore and chat with Hazel. She will provide you with your daily list of tasks. Within these tasks, you may be lucky enough to be given one that rewards in $PIXEL. Final Thoughts: In 2024, Pixels is stepping into its next chapter of growth fueled by the listing of $PIXEL on Binance. To the Pixels community, today is a historic moment for all of us. We can’t wait to welcome even more members to our community in the days, weeks, and years to come.
1:27

Our token, $PIXEL, is officially live on Binance, the world’s largest crypto exchange 🔥 We’ve been working towards this moment for over 2 years, and we couldn’t be more excited to share this massive milestone with you all. This is just the beginning and the best is yet to come for the Pixels community. New here? Welcome to Pixels: Pixels is a free-to-play farming MMO with a play-to-earn (P2E) model. Gameplay revolves around resource management, strategic exploration and community building. Over the past year, we’ve grown to one of Web3’s largest games, with daily active players of 180k+. At Pixels, we build with our community. We're far from a traditional game studio, and our development process reflects that. Over the past 2 years, we’ve had 60+ public updates. This approach enables us to iterate and experiment more quickly, giving players a true sense of ownership over the game's direction. When you join our community, you are not just a player of Pixels, but a creator of it. Our Mission: We’re excited by the potential of Web3 technology and its ability to significantly enhance the gaming experience. We aim to experiment, build, and iterate on the features that make Web3 gaming unique. We’re here to set a new standard. Here are some of the interesting areas we've leaned into: ➡️ P2E: earn $PIXEL by participating in gameplay ➡️ Guilds: friendtech + competition ➡️ On-chain Reputation & Player Personas: creating a sustainable ecosystem of real players ➡️ Interoperability: using on-chain data to bring communities together We're eager to explore more possibilities in this space and push Web3 even further. Meet $PIXEL ✨ $PIXEL is an in-game utility token. It allows players to enhance their gameplay experience in various ways: ➡️ Purchase VIP memberships ➡️ Create and join guilds ➡️ Mint a Pixels Pet ➡️ Upgrade lands We believe Web3 games don’t need to look exactly like Web2 games. However, P2E formats need sustainable structures and economics. With that, we’ve focused heavily on persona types: why would each user want to use this token? We want to ensure we’ve created an environment where first: ➡️ Our game is providing real enjoyment and entertainment to our users ➡️ Our currency can help amplify that experience without affecting the experience of other users How do I get $PIXEL? Users can earn $PIXEL in-game right now. Head over to Buck’s Galore and chat with Hazel. She will provide you with your daily list of tasks. Within these tasks, you may be lucky enough to be given one that rewards in $PIXEL. Final Thoughts: In 2024, Pixels is stepping into its next chapter of growth fueled by the listing of $PIXEL on Binance. To the Pixels community, today is a historic moment for all of us. We can’t wait to welcome even more members to our community in the days, weeks, and years to come.

Pixels

1,462,764 просмотров • 2 лет назад

With every generation of Pixel, Google delivers innovation to consumers, and our latest devices are no different. Hear about how Pixel uses AI to bring helpful experiences to life from VP of Product Management at Pixel, Venkat Rapaka. Learn more →
2:04

With every generation of Pixel, Google delivers innovation to consumers, and our latest devices are no different. Hear about how Pixel uses AI to bring helpful experiences to life from VP of Product Management at Pixel, Venkat Rapaka. Learn more →

Made by Google

22,409 просмотров • 1 год назад

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)
0:43

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 просмотров • 1 год назад

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.
0:43

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

AK

179,061 просмотров • 3 лет назад