Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Concerto Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

AK

512,003 subscribers

11,213 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Supervised learning has held 3D Vision back for too long. Meet RayZer — a self-supervised 3D model trained with zero 3D labels: ❌ No supervision of camera & geometry ✅ Just RGB images And the wild part? RayZer outperforms supervised methods (as 3D labels from COLMAP is noisy) 🌐 Project: (1/4)

Supervised learning has held 3D Vision back for too long. Meet RayZer — a self-supervised 3D model trained with zero 3D labels: ❌ No supervision of camera & geometry ✅ Just RGB images And the wild part? RayZer outperforms supervised methods (as 3D labels from COLMAP is noisy) 🌐 Project: (1/4)

Hanwen Jiang

69,607 views • 1 year ago

Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals

Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals

AK

28,754 views • 1 year ago

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

Hila Chefer

68,791 views • 5 months ago

Pixie Fast and Generalizable Supervised Learning of 3D Physics from Pixels

Pixie Fast and Generalizable Supervised Learning of 3D Physics from Pixels

AK

21,434 views • 11 months ago

(1/N) Will this be the BERT/GPT moment for 3D vision？ Finally, unsupervised pre-training for 3D works. Led by Qitao Zhao , we present E-RayZer — a fully self-supervised 3D reconstruction model that: 🔥Matches or surpasses supervised methods like VGGT 👀Learns transferable 3D representations, outperforming CroCo, VideoMAE, and DINO 📈Scales with more unlabeled data A new recipe for scalable 3D foundation models.

(1/N) Will this be the BERT/GPT moment for 3D vision？ Finally, unsupervised pre-training for 3D works. Led by Qitao Zhao , we present E-RayZer — a fully self-supervised 3D reconstruction model that: 🔥Matches or surpasses supervised methods like VGGT 👀Learns transferable 3D representations, outperforming CroCo, VideoMAE, and DINO 📈Scales with more unlabeled data A new recipe for scalable 3D foundation models.

Hanwen Jiang

58,200 views • 8 months ago

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Sergey Levine

48,785 views • 1 year ago

Announcing the Video Editor in Spatial Media Toolkit. Effortlessly create spatial videos or photo slideshows with music, all privately on-device. Seamlessly integrates with our powerful 2D-to-3D conversion for stunning spatial storytelling.

Announcing the Video Editor in Spatial Media Toolkit. Effortlessly create spatial videos or photo slideshows with music, all privately on-device. Seamlessly integrates with our powerful 2D-to-3D conversion for stunning spatial storytelling.

Ryan Gordon

12,353 views • 1 year ago

Meet LA-Pose. Our latest model taking Wayve another step towards generalization at scale. LA-Pose employs large-scale self-supervised learning, building strong motion representations for 3D perception from 10.2 million unlabeled driving video snippets, unlike today's strongest approaches that often depend on expensive, carefully curated 3D supervision. With only a lightweight pose head and limited labelled data, LA-Pose achieves: 📷 State-of-the-art camera pose estimation 🌎 Strong zero-shot generalization across diverse driving scenarios 🏷️ Orders of magnitude less labelled data than fully supervised 3D approaches Our full blog post: Explore the full paper here:

Meet LA-Pose. Our latest model taking Wayve another step towards generalization at scale. LA-Pose employs large-scale self-supervised learning, building strong motion representations for 3D perception from 10.2 million unlabeled driving video snippets, unlike today's strongest approaches that often depend on expensive, carefully curated 3D supervision. With only a lightweight pose head and limited labelled data, LA-Pose achieves: 📷 State-of-the-art camera pose estimation 🌎 Strong zero-shot generalization across diverse driving scenarios 🏷️ Orders of magnitude less labelled data than fully supervised 3D approaches Our full blog post: Explore the full paper here:

Wayve

36,774 views • 3 months ago

Introducing “FlowMap”, the first self-supervised, differentiable structure-from-motion method that is competitive with conventional SfM like Colmap! IMO this solves a major missing piece for internet-scale training of 3D Deep Learning methods. 1/n

Introducing “FlowMap”, the first self-supervised, differentiable structure-from-motion method that is competitive with conventional SfM like Colmap! IMO this solves a major missing piece for internet-scale training of 3D Deep Learning methods. 1/n

Vincent Sitzmann

128,620 views • 2 years ago

📢 SHeaP: Self-Supervised Head Predictor Learned via 2D Gaussians 📢 Given a single input image, we predict accurate 3D head geometry, pose, and expression. Previous works (e.g. DECA, EMOCA) use differentiable mesh rasterization to learn a self-supervised head geometry predictor via a photometric reconstruction loss. We borrow these ideas, but our key insight is to replace the mesh rendering with 2D Gaussian Splatting. This leads to much higher accuracy of the underlying predicted geometry and thus more gradient signal during training. 🌍 🎥 Great work by Liam Schoneveld Davide Davoli Jiapeng Tang

📢 SHeaP: Self-Supervised Head Predictor Learned via 2D Gaussians 📢 Given a single input image, we predict accurate 3D head geometry, pose, and expression. Previous works (e.g. DECA, EMOCA) use differentiable mesh rasterization to learn a self-supervised head geometry predictor via a photometric reconstruction loss. We borrow these ideas, but our key insight is to replace the mesh rendering with 2D Gaussian Splatting. This leads to much higher accuracy of the underlying predicted geometry and thus more gradient signal during training. 🌍 🎥 Great work by Liam Schoneveld Davide Davoli Jiapeng Tang

Matthias Niessner

28,559 views • 1 year ago

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

Jiaman Li

15,886 views • 1 year ago

Professor Yi Ma (Yi Ma) discusses his new open-source book "Learning Deep Representations of Data Distributions" The thesis: Intelligence -- natural or artificial -- can be derived from just TWO principles: Parsimony and Self-Consistency.

Professor Yi Ma (Yi Ma) discusses his new open-source book "Learning Deep Representations of Data Distributions" The thesis: Intelligence -- natural or artificial -- can be derived from just TWO principles: Parsimony and Self-Consistency.

Machine Learning Street Talk

58,270 views • 8 months ago

✨ICLR 2024 Spotlight Still hand-labeling phases or tuning GANs with your trajectories? We propose FLD, a self-supervised representation method that extracts spatial-temporal relationships in high-dimensional trajectories. RL policies informed by FLD show extended generality.🧵

✨ICLR 2024 Spotlight Still hand-labeling phases or tuning GANs with your trajectories? We propose FLD, a self-supervised representation method that extracts spatial-temporal relationships in high-dimensional trajectories. RL policies informed by FLD show extended generality.🧵

Chenhao Li

81,933 views • 2 years ago

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

AK

85,570 views • 2 years ago

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 views • 1 year ago

Excited to present UnScene3D coming to #CVPR24. David Rozenberszki introduces the first fully unsupervised learned 3D instance segmentation! Using graph cuts over self-supervised features, and self-training for robust & dense masks. by David Rozenberszki, w/ Or Litany

Excited to present UnScene3D coming to #CVPR24. David Rozenberszki introduces the first fully unsupervised learned 3D instance segmentation! Using graph cuts over self-supervised features, and self-training for robust & dense masks. by David Rozenberszki, w/ Or Litany

Angela Dai

11,586 views • 2 years ago

Combine the powers of 3D and 2D to kick-off your projects 💁🏽 ⚡️: learn how to do previs with Story Tools! #b3d #training #BlenderStudio #learning #previs #storyboard #course

Combine the powers of 3D and 2D to kick-off your projects 💁🏽 ⚡️: learn how to do previs with Story Tools! #b3d #training #BlenderStudio #learning #previs #storyboard #course

Blender Studio 🔶

12,699 views • 4 months ago

Introducing Gen 2 spatial motion: 3D motion capture, full-body dynamics, joint torques, ground reaction forces, advanced motion retargeting, motion prediction. Works on AI video, phones, cinema cams, stadia. Launched at Movement Day, BAFTA London.

Introducing Gen 2 spatial motion: 3D motion capture, full-body dynamics, joint torques, ground reaction forces, advanced motion retargeting, motion prediction. Works on AI video, phones, cinema cams, stadia. Launched at Movement Day, BAFTA London.

Move AI

68,714 views • 1 year ago