Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

1️⃣ Ego-Exo4D A new foundational dataset + benchmark suite to support research on video learning & multimodal perception, co-developed with 14 university partners. Details ➡️ Core to the work is videos of skilled human activities, simultaneously capturing both first-person "egocentric" + multiple “exocentric” views.

AIatMeta's profile picture

AI at Meta

804,602 subscribers

133,666 Aufrufe • vor 2 Jahren •via X (Twitter)

BildungWissenschaft & TechnologieNachrichten & Politik
LIVE

Anya Rossi• Live Now

Private livecam show

5 Kommentare

Profilbild von AI at Meta
AI at Metavor 2 Jahren

10 years of FAIR. 10 years of advancing the state of the art in AI through open research. We're celebrating the 10th anniversary of Meta's Fundamental AI Research team and continuing that legacy by sharing our work on three exciting new research projects today. Details below 🧵

Profilbild von AI at Meta
AI at Metavor 2 Jahren

2️⃣ Seamless Communication A family of AI research models that enable more natural & authentic communication across languages. The models deliver vocal style & prosody preservation capabilities as well as near real-time streaming translations. Details ➡️

Profilbild von AI at Meta
AI at Metavor 2 Jahren

3️⃣ Audiobox A foundation model for audio generation. It unifies generation & editing capabilities for speech, SFX & soundscapes. It can generate voices & SFX with voice inputs + text prompts, to create custom audio for a range of use cases. Details ➡️

Profilbild von AI at Meta
AI at Metavor 2 Jahren

Today we're also sharing updates on three important categories of socially responsible AI research. This includes ROBBIE, a new tool to help provide a fuller picture of fairness in LLMs across 12 different demographic axes and five LLMs. Details ➡️

Profilbild von AI at Meta
AI at Metavor 2 Jahren

Beginning FAIR’s second decade, the organization continues to push forward with the same spirit of open science that jump-started the organization in 2013 — we’re as excited as ever about the work that lies ahead. Read more in the new post by @jpineau1 ⬇️

Ähnliche Videos

Together with the Ego4D consortium, we recently released Ego-Exo4D: A diverse, large-scale multi-modal, multi-view, video dataset and benchmark. Learn more about the work ➡️ Access the dataset ➡️ This work could help to advance AI models' understanding of complex human skills & enable new applications for AR systems, robotics & more.
0:49

Together with the Ego4D consortium, we recently released Ego-Exo4D: A diverse, large-scale multi-modal, multi-view, video dataset and benchmark. Learn more about the work ➡️ Access the dataset ➡️ This work could help to advance AI models' understanding of complex human skills & enable new applications for AR systems, robotics & more.

AI at Meta

78,070 Aufrufe • vor 2 Jahren

Today, we announce Ego-Exo4D, the largest and most diverse multi-view dataset, showing human experts around the world performing a core set of skilled activities, w/ unprecedented multi-modality, novel new video-language resources, and rich annotations
0:46

Today, we announce Ego-Exo4D, the largest and most diverse multi-view dataset, showing human experts around the world performing a core set of skilled activities, w/ unprecedented multi-modality, novel new video-language resources, and rich annotations

Ego4D

304,463 Aufrufe • vor 2 Jahren

Released by Reality Labs at Meta Research at #ECCV2024, Nymeria is a large-scale multimodal egocentric dataset for full-body motion understanding with potential applications in VR/MR headsets, smart glasses and more. More on this work + access to the dataset ➡️
0:25

Released by Reality Labs at Meta Research at #ECCV2024, Nymeria is a large-scale multimodal egocentric dataset for full-body motion understanding with potential applications in VR/MR headsets, smart glasses and more. More on this work + access to the dataset ➡️

AI at Meta

52,379 Aufrufe • vor 1 Jahr

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️
1:01

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️

AI at Meta

20,244 Aufrufe • vor 1 Jahr

Ropedia Xperience-10M is out on Hugging Face a large-scale egocentric multimodal dataset of human experience for embodied AI, robotics, world models, and spatial intelligence It contains 10 million experiences (interaction) and 10,000 hours of synchronized first-person recordings with six video streams, audio, stereo depth, camera pose, hand mocap, full-body mocap, IMU, and hierarchical language annotations dataset:
1:32

Ropedia Xperience-10M is out on Hugging Face a large-scale egocentric multimodal dataset of human experience for embodied AI, robotics, world models, and spatial intelligence It contains 10 million experiences (interaction) and 10,000 hours of synchronized first-person recordings with six video streams, audio, stereo depth, camera pose, hand mocap, full-body mocap, IMU, and hierarchical language annotations dataset:

AK

10,551 Aufrufe • vor 3 Monaten

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵
0:59

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵

Danfei Xu

287,105 Aufrufe • vor 3 Monaten

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji
1:24

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

VaderResearch

23,055 Aufrufe • vor 10 Monaten

Georgia Tech's research uses Meta’s Project Aria glasses to train humanoid robots with egocentric data. PhD student Simar Kareer developed an algorithm that leverages human data to enhance robot learning, achieving a 400% performance boost with just 90 minutes of recordings.
3:00

Georgia Tech's research uses Meta’s Project Aria glasses to train humanoid robots with egocentric data. PhD student Simar Kareer developed an algorithm that leverages human data to enhance robot learning, achieving a 400% performance boost with just 90 minutes of recordings.

The Humanoid Hub

65,618 Aufrufe • vor 1 Jahr

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)
0:27

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Ryan Hoque

114,089 Aufrufe • vor 1 Jahr

Today we’re announcing two new updates in our computer vision work — a new, expanded license for our DINOv2 model and the release of FACET, a comprehensive new benchmark dataset to help evaluate and improve fairness in vision models. More details ➡️ 🧵
0:27

Today we’re announcing two new updates in our computer vision work — a new, expanded license for our DINOv2 model and the release of FACET, a comprehensive new benchmark dataset to help evaluate and improve fairness in vision models. More details ➡️ 🧵

AI at Meta

453,971 Aufrufe • vor 2 Jahren

Meta PARTNR is a research framework supporting seamless human-robot collaboration. Building on our research with Habitat, we’re open sourcing a large-scale benchmark, dataset and large planning model that we hope will enable the community to effectively train social robots.
2:46

Meta PARTNR is a research framework supporting seamless human-robot collaboration. Building on our research with Habitat, we’re open sourcing a large-scale benchmark, dataset and large planning model that we hope will enable the community to effectively train social robots.

AI at Meta

106,951 Aufrufe • vor 1 Jahr

🛑📢 HD-EPIC: A Highly-Detailed Egocentric Video Dataset New collected videos 263 annotations/min: recipe, nutrition, actions, sounds, 3D object movement &fixture associations, masks. 26K VQA benchmark to challenge current VLMs 1/N
1:50

🛑📢 HD-EPIC: A Highly-Detailed Egocentric Video Dataset New collected videos 263 annotations/min: recipe, nutrition, actions, sounds, 3D object movement &fixture associations, masks. 26K VQA benchmark to challenge current VLMs 1/N

Dima Damen

20,952 Aufrufe • vor 1 Jahr

Our paper “Learning Situated Awareness in the Real World” has been accepted to #ICML2026 as a Spotlight (top 2.2%)! Congrats to Chuhan Li ✈️ICML and the team! We introduce SAW-Bench, a real-world egocentric benchmark for evaluating situated (first-person) spatial reasoning in multimodal models, uncovering a large gap between humans and current systems. All videos recorded via Meta Ray-Ban 2 glasses!
1:25

Our paper “Learning Situated Awareness in the Real World” has been accepted to #ICML2026 as a Spotlight (top 2.2%)! Congrats to Chuhan Li ✈️ICML and the team! We introduce SAW-Bench, a real-world egocentric benchmark for evaluating situated (first-person) spatial reasoning in multimodal models, uncovering a large gap between humans and current systems. All videos recorded via Meta Ray-Ban 2 glasses!

Xin Eric Wang

11,503 Aufrufe • vor 2 Monaten

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI
3:58

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Zhiyang (Frank) Dou

14,610 Aufrufe • vor 11 Monaten

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.
2:21

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.

AK

23,958 Aufrufe • vor 1 Jahr

Today, we're proud to announce a new partnership with University of Waterloo to explore the intersection of AI and the future of work and learning. Our partnership will fund a new Research Chair and hands-on learning labs, bringing together our expertise in AI with Waterloo’s innovative approach to education. We're excited to see what the next generation of leaders will build.
1:30

Today, we're proud to announce a new partnership with University of Waterloo to explore the intersection of AI and the future of work and learning. Our partnership will fund a new Research Chair and hands-on learning labs, bringing together our expertise in AI with Waterloo’s innovative approach to education. We're excited to see what the next generation of leaders will build.

Google Canada

18,550 Aufrufe • vor 9 Monaten

A closer look at the evolution of the research from Meta FAIR that led to our latest open source work for human-robot collaboration. Details on Meta PARTNR ➡️
1:37

A closer look at the evolution of the research from Meta FAIR that led to our latest open source work for human-robot collaboration. Details on Meta PARTNR ➡️

AI at Meta

41,228 Aufrufe • vor 1 Jahr

⚡️EgoVerse is a first-of-its-kind, collaborative ecosystem for human-to-robot learning. The consortium leverages Project Aria to capture high-fidelity, egocentric human data — including 3D hand and head poses — to train next-gen robot manipulation policies. With over 1,300 hours of data across 2,000+ tasks, EgoVerse is a prime example of how the Aria Research Kit is being used by our partners to accelerate the future of embodied AI. Learn more: 🔗 📰 Apply for the Aria Research Kit: #MachineLearning #Robotics #ProjectAria #EgoVerse #ComputerVision Simar Kareer , Ryan Punamiya , Roger Qiu, Xiongyi Cai , Alexey Gavryushin
0:35

⚡️EgoVerse is a first-of-its-kind, collaborative ecosystem for human-to-robot learning. The consortium leverages Project Aria to capture high-fidelity, egocentric human data — including 3D hand and head poses — to train next-gen robot manipulation policies. With over 1,300 hours of data across 2,000+ tasks, EgoVerse is a prime example of how the Aria Research Kit is being used by our partners to accelerate the future of embodied AI. Learn more: 🔗 📰 Apply for the Aria Research Kit: #MachineLearning #Robotics #ProjectAria #EgoVerse #ComputerVision Simar Kareer , Ryan Punamiya , Roger Qiu, Xiongyi Cai , Alexey Gavryushin

Project Aria @Meta

17,982 Aufrufe • vor 2 Monaten