正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

We release Action100M, the hero behind VL-JEPA. It is a large dataset with O(100 million) dense action annotations on HowTo100M procedural videos. We hope it serves as a robust data foundation to advance physical world modeling research.

Delong Chen (陈德龙)

1,510 subscribers

103,753 次观看 • 8 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Meta PARTNR is a research framework supporting seamless human-robot collaboration. Building on our research with Habitat, we’re open sourcing a large-scale benchmark, dataset and large planning model that we hope will enable the community to effectively train social robots.

Meta PARTNR is a research framework supporting seamless human-robot collaboration. Building on our research with Habitat, we’re open sourcing a large-scale benchmark, dataset and large planning model that we hope will enable the community to effectively train social robots.

AI at Meta

107,046 次观看 • 1 年前

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

AK

122,567 次观看 • 3 年前

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

Julen Urain

42,451 次观看 • 2 年前

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Pascale Fung

90,144 次观看 • 9 个月前

VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relevant dynamics. During training, the VLA leverages V-JEPA2 by conditioning its predictor. This clever trick adds a world modeling objective to the training, which also allows pretraining on human videos. At inference, the world model is dropped entirely, keeping only a standard VLA architecture: Qwen backbone and action head. The demo here was only fine-tuned on 13 examples, showing great pretraining capability and running in real time on NVIDIA Robotics DGX Spark! VLA-JEPA is the first world model to be ported to LeRobot, and I feel like it won't be the last 🚀 Thomas Wolf clem 🤗

VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relevant dynamics. During training, the VLA leverages V-JEPA2 by conditioning its predictor. This clever trick adds a world modeling objective to the training, which also allows pretraining on human videos. At inference, the world model is dropped entirely, keeping only a standard VLA architecture: Qwen backbone and action head. The demo here was only fine-tuned on 13 examples, showing great pretraining capability and running in real time on NVIDIA Robotics DGX Spark! VLA-JEPA is the first world model to be ported to LeRobot, and I feel like it won't be the last 🚀 Thomas Wolf clem 🤗

LeRobot

320,328 次观看 • 3 个月前

Training world models needs egocentric video and dense action signals, synchronized. That data is genuinely hard to find. We built it from Counter-Strike 2 demos. CS2-10k: 600K+ player-round videos, 10K+ hours, per-frame annotations — keyboard state, mouse delta, 3D position, camera yaw/pitch. All paired to the visual stream. Why CS2 demos? Matches are deterministic replays. We can reconstruct first-person video and extract the exact control inputs that caused every visual change. No labeling, no estimation. We're also releasing cs2-dem-renderer — the open-source pipeline we used to build it. Give it a .dem file, it outputs .mp4 + .parquet. Run your own dataset at whatever scale you need.

Training world models needs egocentric video and dense action signals, synchronized. That data is genuinely hard to find. We built it from Counter-Strike 2 demos. CS2-10k: 600K+ player-round videos, 10K+ hours, per-frame annotations — keyboard state, mouse delta, 3D position, camera yaw/pitch. All paired to the visual stream. Why CS2 demos? Matches are deterministic replays. We can reconstruct first-person video and extract the exact control inputs that caused every visual change. No labeling, no estimation. We're also releasing cs2-dem-renderer — the open-source pipeline we used to build it. Give it a .dem file, it outputs .mp4 + .parquet. Run your own dataset at whatever scale you need.

Reka

95,438 次观看 • 2 个月前

We are happy to report that Chronicles of the Wolf is running superbly on the #NintendoSwitch with load times that are virtually undetectable! Here's a little glimpse of it in action. We hope to release it within the next few months! 🐺 #screenshotsaturday #metroidvania #indiedev

We are happy to report that Chronicles of the Wolf is running superbly on the #NintendoSwitch with load times that are virtually undetectable! Here's a little glimpse of it in action. We hope to release it within the next few months! 🐺 #screenshotsaturday #metroidvania #indiedev

Migami Games

37,295 次观看 • 2 年前

Ropedia Xperience-10M is out on Hugging Face a large-scale egocentric multimodal dataset of human experience for embodied AI, robotics, world models, and spatial intelligence It contains 10 million experiences (interaction) and 10,000 hours of synchronized first-person recordings with six video streams, audio, stereo depth, camera pose, hand mocap, full-body mocap, IMU, and hierarchical language annotations dataset:

Ropedia Xperience-10M is out on Hugging Face a large-scale egocentric multimodal dataset of human experience for embodied AI, robotics, world models, and spatial intelligence It contains 10 million experiences (interaction) and 10,000 hours of synchronized first-person recordings with six video streams, audio, stereo depth, camera pose, hand mocap, full-body mocap, IMU, and hierarchical language annotations dataset:

AK

10,621 次观看 • 6 个月前

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Dyna Robotics

1,789,475 次观看 • 1 个月前

$AI agents are about to redefine the internet. The mistake we made with Large Language Models? We let a handful of corporations capture all the value. Action Model is building a different path. By training through our extension, users gain fractional ownership in the Large Action Model, giving them a real stake in the future of AI. When LLMs emerged, the upside flowed to Big Tech. This time, it doesn’t have to. They’re building AI on our data, and keeping the upside for themselves. Community-owned Large Action Model is how we take it back.$

AI agents are about to redefine the internet. The mistake we made with Large Language Models? We let a handful of corporations capture all the value. Action Model is building a different path. By training through our extension, users gain fractional ownership in the Large Action Model, giving them a real stake in the future of AI. When LLMs emerged, the upside flowed to Big Tech. This time, it doesn’t have to. They’re building AI on our data, and keeping the upside for themselves. Community-owned Large Action Model is how we take it back.

Action Model

77,676 次观看 • 6 个月前

Can 1,000 real-world 3D scenes make a state-of-the-art model better? At OVER Research, we tested it by fine-tuning R3, a state-of-the-art streaming 3D reconstruction model, on roughly 1,000 scenes from the OVER dataset. The result: 25% lower trajectory error on our real-world environments compared to the deployed public checkpoint. And we did it using only 1,000 scenes, the same open dataset anyone can download for free, out of the 285,000+ real-world 3D scenes OVER has today. Read the full OVER Research publication: Download the 1,000-scene dataset on Hugging Face:

Can 1,000 real-world 3D scenes make a state-of-the-art model better? At OVER Research, we tested it by fine-tuning R3, a state-of-the-art streaming 3D reconstruction model, on roughly 1,000 scenes from the OVER dataset. The result: 25% lower trajectory error on our real-world environments compared to the deployed public checkpoint. And we did it using only 1,000 scenes, the same open dataset anyone can download for free, out of the 285,000+ real-world 3D scenes OVER has today. Read the full OVER Research publication: Download the 1,000-scene dataset on Hugging Face:

Over the Reality 🌐

34,522 次观看 • 21 天前

The largest fully annotated open egocentric human dataset. Today we're open-sourcing EgoSuite-Open100K with Hugging Face : 100,000 hours of egocentric human data, free for anyone to use. Physical AI has a scaling law of its own now, and its input is human data. That part is settled. What the field still needs is a shared foundation to build on. 15,000+ tasks. 15,000+ real scenes, from factory floors to retail backrooms. Hand pose, body pose and subtask-level semantics, with wrist camera coverage on part of the set. Licensed for commercial training, not research-only. We build the data infrastructure for Physical AI. Open100K is the first public layer of it. The first 10,000 hours are live now, with the rest rolling out in stages.

The largest fully annotated open egocentric human dataset. Today we're open-sourcing EgoSuite-Open100K with Hugging Face : 100,000 hours of egocentric human data, free for anyone to use. Physical AI has a scaling law of its own now, and its input is human data. That part is settled. What the field still needs is a shared foundation to build on. 15,000+ tasks. 15,000+ real scenes, from factory floors to retail backrooms. Hand pose, body pose and subtask-level semantics, with wrist camera coverage on part of the set. Licensed for commercial training, not research-only. We build the data infrastructure for Physical AI. Open100K is the first public layer of it. The first 10,000 hours are live now, with the rest rolling out in stages.

Lightwheel

73,678 次观看 • 24 天前

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

Xueyan Zou

39,142 次观看 • 3 个月前

As we enter 2026, we begin the year with a message from the late Dr. Jane Goodall at UNESCO: “We need countries and individuals to work together to make this a better world.” A reminder that hope is built through collective action.

As we enter 2026, we begin the year with a message from the late Dr. Jane Goodall at UNESCO: “We need countries and individuals to work together to make this a better world.” A reminder that hope is built through collective action.

UNESCO

10,069 次观看 • 8 个月前

VCE is proud to collaborate on the release of the RARE Power Dataset—an open-source, high-res tool for solar & wind energy modeling. It fills a key gap for grid planners, ensuring more reliable clean power systems. Use the QR code to go directly to the data!

VCE is proud to collaborate on the release of the RARE Power Dataset—an open-source, high-res tool for solar & wind energy modeling. It fills a key gap for grid planners, ensuring more reliable clean power systems. Use the QR code to go directly to the data!

Vibrant Clean Energy

12,609 次观看 • 1 年前

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,141 次观看 • 1 年前

Harrison "But it was the Prime Minister that said we have to deal with the world as it is. This is the world as it is." "So what is Canada meaningfully doing to advance our interests in this time rather than griping from the sidelines?" Kate Harrison

Harrison "But it was the Prime Minister that said we have to deal with the world as it is. This is the world as it is." "So what is Canada meaningfully doing to advance our interests in this time rather than griping from the sidelines?" Kate Harrison

cbcwatcher

96,456 次观看 • 3 个月前