Загрузка видео...

Не удалось загрузить видео

На главную

An interesting work on Physical AI: PhysX-Omni. First unified sim-ready generation framework for rigid, deformable, and articulated objects, with a diverse dataset and new benchmark. 🌐 💻 📦

ziqi_huang_'s profile picture

Ziqi Huang

2,884 subscribers

27,024 просмотров • 3 месяцев назад •via X (Twitter)

ОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Together with the Ego4D consortium, we recently released Ego-Exo4D: A diverse, large-scale multi-modal, multi-view, video dataset and benchmark. Learn more about the work ➡️ Access the dataset ➡️ This work could help to advance AI models' understanding of complex human skills & enable new applications for AR systems, robotics & more.
0:49

Together with the Ego4D consortium, we recently released Ego-Exo4D: A diverse, large-scale multi-modal, multi-view, video dataset and benchmark. Learn more about the work ➡️ Access the dataset ➡️ This work could help to advance AI models' understanding of complex human skills & enable new applications for AR systems, robotics & more.

AI at Meta

78,091 просмотров • 2 лет назад

🔥 Ming-UniAudio: The 「Nano Banana」moment for speech is here! A single model for universal understanding, generation & free-form editing. First Unified Continuous Tokenizer 「MingTok-Audio」and Unified Und & Gen Speech LLM built on it. First Universal Free-form Speech Editing Model Without Timestamp Condition. First FreeForm Audio Editing Benchmark. 💻 GitHub: 🤗 Tokenizer: 🤗 Model: base: edit: 🤗 Benchmark: 🌍 blog: #AI #Speech #SpeechLLM #LLM #GenerativeAI #Audio #ASR #TTS #SpeechEditing
2:52

🔥 Ming-UniAudio: The 「Nano Banana」moment for speech is here! A single model for universal understanding, generation & free-form editing. First Unified Continuous Tokenizer 「MingTok-Audio」and Unified Und & Gen Speech LLM built on it. First Universal Free-form Speech Editing Model Without Timestamp Condition. First FreeForm Audio Editing Benchmark. 💻 GitHub: 🤗 Tokenizer: 🤗 Model: base: edit: 🤗 Benchmark: 🌍 blog: #AI #Speech #SpeechLLM #LLM #GenerativeAI #Audio #ASR #TTS #SpeechEditing

Ant Ling

21,365 просмотров • 10 месяцев назад

LiteRT is here. The universal framework for on-device AI. 📱💻 ✨ 1.4x faster GPU performance ✨ Unified NPU acceleration ✨ Seamless PyTorch and JAX support ✨ Ready for Gemma and gen AI Build faster, simpler, and everywhere. Read the blog:
0:49

LiteRT is here. The universal framework for on-device AI. 📱💻 ✨ 1.4x faster GPU performance ✨ Unified NPU acceleration ✨ Seamless PyTorch and JAX support ✨ Ready for Gemma and gen AI Build faster, simpler, and everywhere. Read the blog:

Google for Developers

194,389 просмотров • 6 месяцев назад

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,960 просмотров • 9 месяцев назад

In my past research experience, finding or developing an appropriate simulation environment, dataset, and benchmark has always been a challenge. Missing features, limited support, or unexpected bugs often occupied my days and nights. Moreover, current simulation platforms are relatively fragmented—making it challenging to replicate the success of the RT-X dataset in unifying community efforts. Introducing RoboVerse, we provide a unified platform, dataset, and benchmark for scalable and generalizable robot learning. We hope to build a shared foundation to combine the community efforts. RoboVerse includes: MetaSim: We carefully designed a configuration system and a universal interface to align current robotic simulators. With MetaSim, you can use any simulator with the same code—bringing together the community’s diverse efforts under one framework! RoboVerse Dataset and Benchmark: We unify popular simulation environments and benchmarks into a single cohesive system and introduce the RoboVerse dataset—a large-scale, high-quality synthetic dataset. Additionally, we propose a standardized benchmark across both imitation learning and reinforcement learning. A cool feature enabled by our unified framework: Hybrid Simulation! You can now integrate physics engines and renderers from different simulators—e.g., using MuJoCo precise physics with Isaac photorealistic rendering. This not only elevates simulation fidelity but also significantly enhances real-world transfer performance across complex robotic applications. Hopefully, our team’s efforts could serve the robotic community to thrive vibrantly in the years to come. RoboVerse is open-sourced🥳!!! Project Page: Documentation: Github Repo: Paper:
2:21

In my past research experience, finding or developing an appropriate simulation environment, dataset, and benchmark has always been a challenge. Missing features, limited support, or unexpected bugs often occupied my days and nights. Moreover, current simulation platforms are relatively fragmented—making it challenging to replicate the success of the RT-X dataset in unifying community efforts. Introducing RoboVerse, we provide a unified platform, dataset, and benchmark for scalable and generalizable robot learning. We hope to build a shared foundation to combine the community efforts. RoboVerse includes: MetaSim: We carefully designed a configuration system and a universal interface to align current robotic simulators. With MetaSim, you can use any simulator with the same code—bringing together the community’s diverse efforts under one framework! RoboVerse Dataset and Benchmark: We unify popular simulation environments and benchmarks into a single cohesive system and introduce the RoboVerse dataset—a large-scale, high-quality synthetic dataset. Additionally, we propose a standardized benchmark across both imitation learning and reinforcement learning. A cool feature enabled by our unified framework: Hybrid Simulation! You can now integrate physics engines and renderers from different simulators—e.g., using MuJoCo precise physics with Isaac photorealistic rendering. This not only elevates simulation fidelity but also significantly enhances real-world transfer performance across complex robotic applications. Hopefully, our team’s efforts could serve the robotic community to thrive vibrantly in the years to come. RoboVerse is open-sourced🥳!!! Project Page: Documentation: Github Repo: Paper:

Haoran Geng

84,318 просмотров • 1 год назад

The latest update from NemoVideo sets a new benchmark for AI-powered video creation. With NemoVideo now integrated with Seedance 2, the workflow is seamless—structured scripting, refined visual generation, and production-ready output within minutes. A meaningful advancement for creators focused on efficiency and quality. Explore it here: #NemoVideo #Seedance2
0:21

The latest update from NemoVideo sets a new benchmark for AI-powered video creation. With NemoVideo now integrated with Seedance 2, the workflow is seamless—structured scripting, refined visual generation, and production-ready output within minutes. A meaningful advancement for creators focused on efficiency and quality. Explore it here: #NemoVideo #Seedance2

Aryan Rakib

250,353 просмотров • 6 месяцев назад

Robotics needs data. OVER delivers it. Robots aren’t just machines anymore, they’re becoming autonomous agents. The secret? A new generation of Physical AI Models. And the fuel? A global dataset built for scale. Let’s dive in 👇🧵
0:31

Robotics needs data. OVER delivers it. Robots aren’t just machines anymore, they’re becoming autonomous agents. The secret? A new generation of Physical AI Models. And the fuel? A global dataset built for scale. Let’s dive in 👇🧵

Over the Reality 🌐

24,975 просмотров • 11 месяцев назад

🚀 Struggling with the lack of high-quality data for AI-driven human-object interaction research? We've got you covered! Introducing HUMOTO, a groundbreaking 4D dataset for human-object interaction, developed with a combination of wearable motion capture, SOTA 6D pose estimation vision models, LLM, and the professional refining works of multiple animation studios. HUMOTO features: ✅ Over 700 diverse daily activities ✅ Interactions with 60+ objects, 70+ articulated parts. ✅ Fine-grained text annotations ✅ Detailed hand and finger movements We hope HUMOTO will fuel your Humanoid AI research and drive new advancements! For research or commercial license inquiries, please contact yizho@adobe.com. Explore the dataset: 👉 HUMOTO Dataset Website Learn more: 👉 HUMOTO Project Page Jiaxin Lu Qixing Huang #AI #HumanObjectInteraction #HumanoidAI #MotionCapture #HUMOTO #AdobeResearch
0:30

🚀 Struggling with the lack of high-quality data for AI-driven human-object interaction research? We've got you covered! Introducing HUMOTO, a groundbreaking 4D dataset for human-object interaction, developed with a combination of wearable motion capture, SOTA 6D pose estimation vision models, LLM, and the professional refining works of multiple animation studios. HUMOTO features: ✅ Over 700 diverse daily activities ✅ Interactions with 60+ objects, 70+ articulated parts. ✅ Fine-grained text annotations ✅ Detailed hand and finger movements We hope HUMOTO will fuel your Humanoid AI research and drive new advancements! For research or commercial license inquiries, please contact [email protected]. Explore the dataset: 👉 HUMOTO Dataset Website Learn more: 👉 HUMOTO Project Page Jiaxin Lu Qixing Huang #AI #HumanObjectInteraction #HumanoidAI #MotionCapture #HUMOTO #AdobeResearch

Yi Zhou

71,609 просмотров • 1 год назад

Robotics: coding agents’ next frontier. So how good are they? We introduce CaP-X: an open-source framework and benchmark for coding agents, where they write code for robot perception and control, execute it on sim and real robots, observe the outcomes, and iteratively improve code reliability. From NVIDIA Berkeley AI Research CMU Robotics Institute Stanford AI Lab 🧵
1:31

Robotics: coding agents’ next frontier. So how good are they? We introduce CaP-X: an open-source framework and benchmark for coding agents, where they write code for robot perception and control, execute it on sim and real robots, observe the outcomes, and iteratively improve code reliability. From NVIDIA Berkeley AI Research CMU Robotics Institute Stanford AI Lab 🧵

Max Fu

176,519 просмотров • 4 месяцев назад

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI
3:58

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Zhiyang (Frank) Dou

14,610 просмотров • 1 год назад

Yesterday we announced we are joining forces with Stereolabs to accelerate the development of a unified sensing and perception platform for Physical AI We're combining high-resolution digital lidar with stereo cameras and AI compute to offer the first end-to-end sensing and perception stack, enabling machines to sense, think, act, and learn faster and with more precision To the autonomy engineers building the next-generation on Physical AI: we're here to accelerate your development from prototype to production. Welcome to the next chapter of Ouster 🤖
0:37

Yesterday we announced we are joining forces with Stereolabs to accelerate the development of a unified sensing and perception platform for Physical AI We're combining high-resolution digital lidar with stereo cameras and AI compute to offer the first end-to-end sensing and perception stack, enabling machines to sense, think, act, and learn faster and with more precision To the autonomy engineers building the next-generation on Physical AI: we're here to accelerate your development from prototype to production. Welcome to the next chapter of Ouster 🤖

Ouster

14,291 просмотров • 6 месяцев назад

Onto Ultiverse's Next Chapter: On-Chain Procedural Content Generation (PCG)📢 We are entering a new phase with our AI On-Chain PCG framework, transitioning from a gaming platform to an infrastructure provider and ecosystem catalyst for the entire Web3 domain. Read more below🧵
0:21

Onto Ultiverse's Next Chapter: On-Chain Procedural Content Generation (PCG)📢 We are entering a new phase with our AI On-Chain PCG framework, transitioning from a gaming platform to an infrastructure provider and ecosystem catalyst for the entire Web3 domain. Read more below🧵

Ultiverse

141,530 просмотров • 2 лет назад

China is set to host the first humanoid robot arena competition in late May, focusing on direct physical confrontation between full-size bipedal robots. Participants will demonstrate real-time control, stability, and force capability under high-impact conditions—marking a new benchmark for physical AI in open competition.
0:36

China is set to host the first humanoid robot arena competition in late May, focusing on direct physical confrontation between full-size bipedal robots. Participants will demonstrate real-time control, stability, and force capability under high-impact conditions—marking a new benchmark for physical AI in open competition.

Reborn

93,073 просмотров • 1 год назад

Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬 We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. The key contributions of our work are: 🕹️ Enables the exploration of different training algorithms for AI Research Agents such as RL 🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions 🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness 🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning. 📈 Proposes a new evaluation metric for AI Research Agents MLGym makes it easy to: 1) Add new tasks 2) Evaluate new models 3) Integrate new agents Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩‍💻, experimentation 👩‍🔬, and iteration 🔄 to improve on ML tasks. Huge thanks to the exceptionally talented Deepak Nathani who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀
1:32

Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬 We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. The key contributions of our work are: 🕹️ Enables the exploration of different training algorithms for AI Research Agents such as RL 🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions 🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness 🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning. 📈 Proposes a new evaluation metric for AI Research Agents MLGym makes it easy to: 1) Add new tasks 2) Evaluate new models 3) Integrate new agents Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩‍💻, experimentation 👩‍🔬, and iteration 🔄 to improve on ML tasks. Huge thanks to the exceptionally talented Deepak Nathani who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀

Roberta Raileanu

105,084 просмотров • 1 год назад

AI Guidance for Physical Work David Lieb Much of the AI conversation focuses on which desk jobs will get replaced. But for physical work, AI can't yet act in the world. With multimodal models and ubiquitous hardware, there’s an opportunity to give workers live AI coaching that dramatically reduces training time and unlocks skilled labor at scale.
1:40

AI Guidance for Physical Work David Lieb Much of the AI conversation focuses on which desk jobs will get replaced. But for physical work, AI can't yet act in the world. With multimodal models and ubiquitous hardware, there’s an opportunity to give workers live AI coaching that dramatically reduces training time and unlocks skilled labor at scale.

Y Combinator

26,533 просмотров • 6 месяцев назад

You told us you’re running multiple AI agents and wanted a better UX. We listened and shipped it! Here’s what’s new in the latest Visual Studio Code release: 🗂️ Unified agent sessions workspace for local, background, and cloud agents 💻 Claude and Codex support for local and cloud agents 🔀 Parallel subagents 🌐 Integrated browser And more...
1:09

You told us you’re running multiple AI agents and wanted a better UX. We listened and shipped it! Here’s what’s new in the latest Visual Studio Code release: 🗂️ Unified agent sessions workspace for local, background, and cloud agents 💻 Claude and Codex support for local and cloud agents 🔀 Parallel subagents 🌐 Integrated browser And more...

Visual Studio Code

588,230 просмотров • 6 месяцев назад

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal
0:39

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,106 просмотров • 1 год назад

Traditional robots are like player pianos—they only play notes they’re given. Memo is more like a Master Chef who has watched hundreds of cooks in every type of kitchen. Our diverse dataset provides Memo with an “intuition” for grasping entirely new objects. This is a small step towards truly helpful robots that work in any home 🤖👨‍🍳
1:05

Traditional robots are like player pianos—they only play notes they’re given. Memo is more like a Master Chef who has watched hundreds of cooks in every type of kitchen. Our diverse dataset provides Memo with an “intuition” for grasping entirely new objects. This is a small step towards truly helpful robots that work in any home 🤖👨‍🍳

Sunday

197,619 просмотров • 8 месяцев назад

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️
1:01

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️

AI at Meta

20,244 просмотров • 1 год назад