正在加载视频...

视频加载失败

At Axis, every trajectory submitted by our community undergoes a strict replay validation process. We run each submission through checker to verify whether the target task was successfully completed. To see how strict it is, check this demo (Task: Place The Toy Train On The Board Game Box). Real...

30,772 次观看 • 3 个月前 •via X (Twitter)

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Jim Fan

332,199 次观看 • 2 年前

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 次观看 • 1 年前

After 8 months of building in stealth and testing our infrastructure on 10000+ hours of real-world data and hundreds of unique environments, we're bringing FPV Labs into the open today. FPV Labs started with the following bet - if human data proves to be the underlying factor that determines scaling laws in general-purpose robotics, it will trigger the largest economic transformation in human history, and the underlying infrastructure that captures that data will determine how fast we get there. We will achieve this by building the full-stack infrastructure for capturing, processing, transferring, and evaluating human experience into spatial, temporal, and semantic knowledge for machines. Despite all the research novelty behind ChatGPT, its success can be attributed to one foundational fact - the scaling law of transformers. We believe the same dynamics have made their way into robotics. Recent studies showed task completion rates jumping from 30% to 70% when human demonstration data scaled from 1,000 to 20,000 hours, a log-linear trend that mirrors exactly what we saw in language and vision. Seeing these emergent signs of scaling law curves in robotics, we believe we are entering the era of general-purpose robotics policies, which makes the next few years the most exciting time in the history of this field. But the library of physical interactions required to train general-purpose robot policies does not exist yet. Over the last 8 months, we've seen dozens of companies emerge in this space. We were really happy to see new companies pushing this space forward, but we also saw the same pattern repeat: every egocentric data company was making some tradeoffs between quality, scale, and diversity. We have built FPV labs on the core principle that high-quality data is orders of magnitude more valuable than sheer volume. Case in point, self-driving cars collect thousands of hours of data per day, but only a small fraction of that data is actually useful for training better models. Several studies, like RT-2, have shown that as little as 1% of data improves as much as 25% on task success. The quality and diversity of data matter a lot more than scale, so there is clearly a power law curve in the downstream impact of data. We've spent months obsessing over data quality by building our stack, discarding it, rebuilding it, and iterating until we found a formula that doesn't compromise downstream quality at scale. We believe the downstream impact here is far more profound than most people realize. Workers globally are paid around $60 trillion per year in aggregate, and a lion's share of that compensation goes to physical labor - tasks that require navigating real spaces, manipulating real objects, and negotiating the infinite variability of the physical world. Human-to-robot transfer will be one of the most important infrastructures that will shape our society in the near future, and if it works, the economic impact will dwarf every technology transition that came before it in an exponential manner and lead to the creation of goods and services we can’t imagine today. Our mission is to lay the groundwork for us to transition into this future - the future of abundance. We are deeply grateful to our earliest believers, Paras Chopra and Lossfunk, who played a critical role in shaping our thinking.

Abhishek Anand

81,285 次观看 • 3 个月前

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,380 次观看 • 1 年前

Which one is our priority for securing a digital future for Zimbabwe? The Honourable Speaker's position while providing a clear, logical, and data-informed counter-argument we appreciate his efforts and submission. Let us have a conversation on our priorities in the current Zimbabwean context which we presented yesterday: This is a critical national conversation on our digital future. The proposition of a Google data centre is undoubtedly appealing. Building a centre of Technological Excellence that houses multiple Tech companies and a Start-up Ecosystem is also prudent, though we are calling it a Technology Park. However, I was of the following view which I am willing to be criticised and guided on. The strategic prioritisation of a national ICT Park ecosystem currently is the foundational path to sustainable, sovereign growth and capacity for Zimbabwe. We are reaching out to every province through the Digital Centres, Innovation Hubs at Universities and Colleges. Infrastructure which we need to capacitate better. This is a decentralised system utilising and expanding the infrastructure we have. The central question is not if we want major players like Google; that is not questionable, but when, how and on what terms. Why are we building our own foundation, house, techno park and mini ICT Parks in the provinces and districts? We absolutely need at least a one hyperscale data centre, but centralisation in the current and future environment can be restrictive. This was our plan: 1.The Foundation Before the Skyscraper: Capacity and Traffic We need to increase our national data volumes and internet traffic . Our urgent priority should be upgrading our national backbone, specifically the Optical Fibre on Power Transmission Lines (e.g., Powertel's network from Insukamini to Johannesburg), to peer efficiently with global giants. This builds the foundational "digital highway" we manage. 2.Generating Traffic Through Digitalization A data centre is a response to demand, not a creator of it. We must first drive digitalization aggressively through digital payments, process automation, and e-governance to generate the significant local traffic that would make a data centre viable. Furthermore, we must incorporate Edge Computing strategies to process data closer to the source, a more efficient model for our current needs and in compliance to our Cyber and Data Protection Act which we can amend if need be. Being a regional internet and cybersecurity hub is crucial as the Speaker alluded to me mentioning it to SADC Parliamentary Forum, this is the ultimate goal. 3.PPPs are key to lessen the burden to the fiscus and we are ready to accommodate organisations willing to assist government.(Cont)

Hon Tatenda.A. Mavetera

490,311 次观看 • 8 个月前

Kled Version 3 is coming. Over $20M+ in rewards will be paid directly to users from leading AI labs across robotics, legal services, image and video generation, world modeling, and more. In the last seven days, we’ve received inbound data requests from several decacorn AI labs and enterprises for datasets our human data marketplace is uniquely positioned to provide. Since receiving the specs for these requests, we now have a much better picture and understanding of how to reshape the systems that collect this data, so here’s what’s coming: 1. A fully redesigned home experience: The home feed is being rebuilt to surface the highest-value, most relevant tasks for each user, similar to how Uber Eats surfaces top restaurants. The goal is to turn every user into their most effective version as a data contributor. 2. Automated quality enforcement at scale: New ML systems are being built to evaluate task-specific requirements in real time. For example, if a task requires “two hands visible on camera at all times,” any video that fails that spec will be automatically rejected. This logic will apply across thousands of tasks and specifications using a general ML. 3. Kled Shop: Some tasks require better capture hardware. We’re introducing Kled Shop, where users can redeem points or tokens for equipment like Meta glasses, drones, and other tools. Points and tokens can be converted directly from payouts. 4. Partner-run data labeling and evaluation work: Some of our partners operate high-paying data labeling and model evaluation programs. We’re integrating their workflows directly into Kled so qualified users can access these roles in one place. These jobs are owned and managed by our partners. Kled’s role is to route the right people to the right work. Some opportunities pay $50–$1,000 per hour depending on expertise. 5. Global payouts and localization: We’re partnering with a major payment processor to enable cashouts in users’ native currencies. This unlocks broader global participation. Multi-language support is also coming to accelerate user growth. This full suite of tools will be rolling out soon, directly to Kled users. Top earners are currently making ~$7,000 per month. With this update, we should see the first ~$10,000 per month earner.

Avi Patel

124,728 次观看 • 5 个月前