Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Existing 3D human manipulation datasets are valuable, but are limited in scale and diversity. At #CVPR2025, we will introduce GigaHands👐 which, to our knowledge, is the most extensive 3D bimanual manipulation, interaction, and gesture dataset.🧵👇(1/9)

13,581 Aufrufe • vor 1 Jahr •via X (Twitter)

11 Kommentare

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

Some statistics about GigaHands 📊 🕒 34 hrs of manual activities 👥 56 participants 🧱 417 objects 🎞️ 14k 3D motion sequences 🖼️ 183M frames 📝 84k text annotations We hope the dataset opens new research directions in AI, robotics, computer vision, and animation! 🤖🎬🖥️ (2/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

🛠️ How did we build GigaHands? ✅ Markerless multi-camera motion capture with 51 cameras ✅ Procedurally-generated instructions ensuring comprehensive coverage of bimanual interactions ✅ Rich data captured without intrusive markers! (3/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

📑 It has rich annotations for: ✋ 3D hand shape & pose 📦 3D rigid object shape & pose; object scans 🧩 Hand-object segmentation masks 📍 2D/3D keypoints 🎥 Camera poses 📝 Detailed textual descriptions All captured using a markerless, multi-camera setup! 📸(4/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

GigaHands enables applications like text-conditioned motion generation, i.e., describe actions in text 👉 Generate realistic hand interactions! 🤝✨ (5/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

Or motion captioning to generate textual descriptions from hand motions—even across diverse datasets! 📖🔄 (6/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

Since GigaHands has dense cameras, we can reconstruct dynamic 3D hand-object interactions in great detail 🛠️🔍 (7/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

Tracking data from GigaHands can also enable motion retargeting to robotic and virtual hands. 🤖🖐️ (8/9)

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

This work was led by my PhD student, @RaoFu79761158 in collaboration with Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, and Daniel Ritchie @BrownVisualComp. Paper, data and code released: 🌐 #GigaHands #CVPR2025 #AI #ComputerVision #Robotics (9/9)

Profilbild von VistaShares
VistaSharesvor 1 Jahr

From semiconductors to data centers, AIS targets the critical components behind AI's exponential growth. Capture potential returns from this transformative technology sector.

Profilbild von Michael Black
Michael Blackvor 1 Jahr

Nice work! This will be a useful dataset!

Profilbild von Srinath Sridhar
Srinath Sridharvor 1 Jahr

Thanks, Michael!

Ähnliche Videos

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 Aufrufe • vor 3 Jahren