Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Most robot datasets store one sentence per episode: the task. Real problems need more: a plan, a memory, room to answer questions and change course. We just shipped full language support in the LeRobot dataset format. Now lerobot-annotate points a vision-language model at each episode and writes the subtasks,... show more

LeRobot

20,060 subscribers

13,451 views • 2 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

7 Comments

Nick Venturi2 months ago

now robots can explain why they ignored you

Lex2 months ago

Good to see richer annotations getting easier and cheaper. The auto-generated subtasks and Q&A could be really useful for training.

Sam W2 months ago

Is the nvidia dgx spark good for training and sims for a robotic arm to cook food ?

antonio_machine_learning2 months ago

just tell us the command to run a demo that shows off this feature, most of the documentation is out of sync and nothing works.

起岚26 days ago

这条说到点子上了。我们采集系统每条 episode 只存一句 task，回看数据经常后悔没多记几笔。episode 元数据这块全行业都是欠账。

Molty1 month ago

Plans and memory at $1.20 per 100 episodes. Does annotated data improve policy success?

Ebokify1 month ago

Robot Modeling and Control (2 ed)

Related Videos

How do you know your robot actually succeeded? In v0.6.0, you ask a VLM 🤖 LeRobot ships a new unified reward models API with two zero-shot additions. Robometer is a general-purpose reward model trained on 1M+ trajectories across 21 robot embodiments: point it at any LeRobot dataset and it scores task progress and success straight from video and a language instruction, no task-specific training needed. TOPReward goes even further: no reward weights at all. It wraps an off-the-shelf VLM and reads the log-probability of the token "True" given the video and the instruction. Both come with labeling scripts that write per-frame progress curves into your dataset, ready for reward-aware behavior cloning. More on how they work:

How do you know your robot actually succeeded? In v0.6.0, you ask a VLM 🤖 LeRobot ships a new unified reward models API with two zero-shot additions. Robometer is a general-purpose reward model trained on 1M+ trajectories across 21 robot embodiments: point it at any LeRobot dataset and it scores task progress and success straight from video and a language instruction, no task-specific training needed. TOPReward goes even further: no reward weights at all. It wraps an off-the-shelf VLM and reads the log-probability of the token "True" given the video and the instruction. Both come with labeling scripts that write per-frame progress curves into your dataset, ready for reward-aware behavior cloning. More on how they work:

LeRobot

22,513 views • 2 months ago

Introducing Memory 101 🧠 Every ai tool you use is a goldfish. brilliant for one chat, close the tab and it forgets you completely. memory is the most underbuilt layer in ai right now. so we're explaining how it actually works, one episode at a time. coming up: → rag vs memory, why most "ai memory" is just search → the 5 things real memory needs: facts, profiles, graphs, time, forgetting → why good memory has to forget things follow along!

Introducing Memory 101 🧠 Every ai tool you use is a goldfish. brilliant for one chat, close the tab and it forgets you completely. memory is the most underbuilt layer in ai right now. so we're explaining how it actually works, one episode at a time. coming up: → rag vs memory, why most "ai memory" is just search → the 5 things real memory needs: facts, profiles, graphs, time, forgetting → why good memory has to forget things follow along!

supermemory

17,235 views • 3 months ago

ANTHROPIC KILLED THE VECTOR DATABASE AND REPLACED IT WITH A FOLDER no embeddings. no black-box retrieval. no special memory tools. the agent just reads and writes a folder the insight: your agent already knows how to use a filesystem. so Anthropic made long-term memory a directory of small markdown files, mounted straight into the agent's sandbox at /mnt/memory the agent manages it with the tools it already has, bash, read, write, edit, grep, and Anthropic auto-injects a note into the system prompt so the agent always knows the memory is there and where to look the result: > every change is an immutable version: full audit trail, point-in-time recovery, redaction > read-only or read-write per store, so many agents can safely share one memory > workspace-scoped and persistent, memory that survives every session end most teams bolt on a vector DB and pray retrieval works. Anthropic went the opposite way: compiled knowledge as plain, human-readable files you can open, diff, and edit yourself and there's a second layer. Anthropic's "Dreaming" runs offline, reviews past sessions and the store, and rewrites it into a cleaner, higher-signal version. the agent literally consolidates memory in its sleep. bookmark this. the most production-grade agent memory right now is just a well-organized folder, and Anthropic proved it.

ANTHROPIC KILLED THE VECTOR DATABASE AND REPLACED IT WITH A FOLDER no embeddings. no black-box retrieval. no special memory tools. the agent just reads and writes a folder the insight: your agent already knows how to use a filesystem. so Anthropic made long-term memory a directory of small markdown files, mounted straight into the agent's sandbox at /mnt/memory the agent manages it with the tools it already has, bash, read, write, edit, grep, and Anthropic auto-injects a note into the system prompt so the agent always knows the memory is there and where to look the result: > every change is an immutable version: full audit trail, point-in-time recovery, redaction > read-only or read-write per store, so many agents can safely share one memory > workspace-scoped and persistent, memory that survives every session end most teams bolt on a vector DB and pray retrieval works. Anthropic went the opposite way: compiled knowledge as plain, human-readable files you can open, diff, and edit yourself and there's a second layer. Anthropic's "Dreaming" runs offline, reviews past sessions and the store, and rewrites it into a cleaner, higher-signal version. the agent literally consolidates memory in its sleep. bookmark this. the most production-grade agent memory right now is just a well-organized folder, and Anthropic proved it.

NO1ennn

13,359 views • 1 month ago

MolmoAct2 runs zero-shot on an SO-ARM101, no training required 🦾 Ai2 open Action Reasoning Model is now in LeRobot with the full lifecycle covered: fine-tuning (full or LoRA), evaluation, and real-robot deployment. It pairs a Molmo2-ER vision-language backbone with a flow-matching action expert to turn images, language, and proprioceptive state into action chunks. The best part: ready-made checkpoints ship with calibration correction baked in, so you can point it at your SO-100/101 and just watch it work. Inference fits in ~12GB at bf16, and LoRA fine-tuning runs on a single 24GB GPU. Big thanks to the Ai2 team for building this in the open.

MolmoAct2 runs zero-shot on an SO-ARM101, no training required 🦾 Ai2 open Action Reasoning Model is now in LeRobot with the full lifecycle covered: fine-tuning (full or LoRA), evaluation, and real-robot deployment. It pairs a Molmo2-ER vision-language backbone with a flow-matching action expert to turn images, language, and proprioceptive state into action chunks. The best part: ready-made checkpoints ship with calibration correction baked in, so you can point it at your SO-100/101 and just watch it work. Inference fits in ~12GB at bf16, and LoRA fine-tuning runs on a single 24GB GPU. Big thanks to the Ai2 team for building this in the open.

LeRobot

22,200 views • 2 months ago

Can 1,000 real-world 3D scenes make a state-of-the-art model better? At OVER Research, we tested it by fine-tuning R3, a state-of-the-art streaming 3D reconstruction model, on roughly 1,000 scenes from the OVER dataset. The result: 25% lower trajectory error on our real-world environments compared to the deployed public checkpoint. And we did it using only 1,000 scenes, the same open dataset anyone can download for free, out of the 285,000+ real-world 3D scenes OVER has today. Read the full OVER Research publication: Download the 1,000-scene dataset on Hugging Face:

Can 1,000 real-world 3D scenes make a state-of-the-art model better? At OVER Research, we tested it by fine-tuning R3, a state-of-the-art streaming 3D reconstruction model, on roughly 1,000 scenes from the OVER dataset. The result: 25% lower trajectory error on our real-world environments compared to the deployed public checkpoint. And we did it using only 1,000 scenes, the same open dataset anyone can download for free, out of the 285,000+ real-world 3D scenes OVER has today. Read the full OVER Research publication: Download the 1,000-scene dataset on Hugging Face:

Over the Reality 🌐

34,651 views • 1 month ago

Most C Programmers don't have a clue as to how the CPU works. Here is a lecture from my course on the C Language that introduces how to visualise the CPU as a C (or general software) Programmer.... More Details are at: Being able to model the behaviour of CPU provides a solid base to reason about your code. The summary is - - The CPU is a state machine which maintains it's states in registers. - It has busses to memory for requesting data/instruction read/writes. - Interrupts to attend to events. I discuss this and more in my course on the C Language.

Most C Programmers don't have a clue as to how the CPU works. Here is a lecture from my course on the C Language that introduces how to visualise the CPU as a C (or general software) Programmer.... More Details are at: Being able to model the behaviour of CPU provides a solid base to reason about your code. The summary is - - The CPU is a state machine which maintains it's states in registers. - It has busses to memory for requesting data/instruction read/writes. - Interrupts to attend to events. I discuss this and more in my course on the C Language.

Piyush Itankar

28,019 views • 1 year ago

We're releasing XDOF s ABC-130k dataset converted to RRD on a Hugging Face bucket, along with the conversion code. ABC-130k is the largest open-source bi-manual teleoperation dataset published to date. With Rerun, you can view all the episodes, query for insights, and use them in training workflows. One interesting finding is that annotators use varying levels of granularity for subtask labeling. Take the chess task in this video, for example: One episode uses language like 'place the chess piece', while another uses a fine-grained description like 'place the queen/pawn at E8'. This distinction is clearly visualized in Rerun through the colors in the state view! Our repo also includes a "mistake" query example and other interesting observations about the dataset, complementing the official ABC-130k dataset card. Download a subset and explore it yourself!! What other open datasets would you like to see in RRD?

We're releasing XDOF s ABC-130k dataset converted to RRD on a Hugging Face bucket, along with the conversion code. ABC-130k is the largest open-source bi-manual teleoperation dataset published to date. With Rerun, you can view all the episodes, query for insights, and use them in training workflows. One interesting finding is that annotators use varying levels of granularity for subtask labeling. Take the chess task in this video, for example: One episode uses language like 'place the chess piece', while another uses a fine-grained description like 'place the queen/pawn at E8'. This distinction is clearly visualized in Rerun through the colors in the state view! Our repo also includes a "mistake" query example and other interesting observations about the dataset, complementing the official ABC-130k dataset card. Download a subset and explore it yourself!! What other open datasets would you like to see in RRD?

Rerun

21,234 views • 1 month ago

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

NO1ennn

227,121 views • 1 month ago

Your robot can now see in depth 👀📏 LeRobot v0.6.0 adds real end-to-end depth support. Plug in an Intel RealSense, set use_depth: true, and depth maps join your RGB observations automatically: captured in millimeters, compressed into compact 12-bit depth video alongside your normal camera streams, and decoded back to physical units at training time. Depth renders live while you record and in the dataset viewer, and it works across SO-100/101, Koch, OpenArm, reBot, Unitree G1 and more. You also get full control over how everything gets encoded, RGB or depth, codec, quality, presets, all of it. More on how it works:

Your robot can now see in depth 👀📏 LeRobot v0.6.0 adds real end-to-end depth support. Plug in an Intel RealSense, set use_depth: true, and depth maps join your RGB observations automatically: captured in millimeters, compressed into compact 12-bit depth video alongside your normal camera streams, and decoded back to physical units at training time. Depth renders live while you record and in the dataset viewer, and it works across SO-100/101, Koch, OpenArm, reBot, Unitree G1 and more. You also get full control over how everything gets encoded, RGB or depth, codec, quality, presets, all of it. More on how it works:

LeRobot

31,020 views • 2 months ago

STANFORD BUILT AN LLM MEMORY THAT WORKS LIKE A BRAIN AND IT MADE STANDARD RAG LOOK ANCIENT no fine-tuning. no bigger model. just a smarter way to remember the insight: the brain doesn't store memories as isolated notes. it indexes them and links them - so one cue pulls back the whole memory so HippoRAG stops treating documents as separate chunks. it builds a knowledge graph, then spreads activation across it (Personalized PageRank) to pull a full answer from a single cue - the brain calls this "pattern completion" the result: > up to 20% better on hard multi-hop questions > 10-30x cheaper and 6-13x faster than iterative retrieval > connects facts split across documents in a single step most RAG still encodes every passage in isolation - so it breaks the moment the answer is scattered more retrieval was never the answer. brain-like memory is bookmark this. the next leap in AI agents is how they remember, not how big they get

STANFORD BUILT AN LLM MEMORY THAT WORKS LIKE A BRAIN AND IT MADE STANDARD RAG LOOK ANCIENT no fine-tuning. no bigger model. just a smarter way to remember the insight: the brain doesn't store memories as isolated notes. it indexes them and links them - so one cue pulls back the whole memory so HippoRAG stops treating documents as separate chunks. it builds a knowledge graph, then spreads activation across it (Personalized PageRank) to pull a full answer from a single cue - the brain calls this "pattern completion" the result: > up to 20% better on hard multi-hop questions > 10-30x cheaper and 6-13x faster than iterative retrieval > connects facts split across documents in a single step most RAG still encodes every passage in isolation - so it breaks the moment the answer is scattered more retrieval was never the answer. brain-like memory is bookmark this. the next leap in AI agents is how they remember, not how big they get

NO1ennn

32,454 views • 1 month ago

THIS GUY TURNED ONE OBSIDIAN VAULT INTO A SHARED BRAIN FOR 5 CLAUDE AGENTS BUILT TO HANDLE 80% OF A SOLO DEVELOPER’S WORK. 00:03 he zooms out and hundreds of connected notes appear on screen. it is the perfect visual for Claude working from one shared project memory instead of starting from zero every session. most developers give Claude 1 file and 1 prompt. this setup gives it the architecture, coding rules, tests, client briefs, and previous decisions before it writes anything. obsidian becomes the memory layer, storing everything in markdown files Claude can read alongside the codebase. every agent works from the same context instead of repeating the same questions. then 5 specialists split the job. one researches, one plans, one codes, one reviews, and one deploys while the developer only controls the important decisions. one vault, 5 agents, 2 model tiers, and a $50 to $150 monthly stack can turn Claude from a chatbot into a small development team.

THIS GUY TURNED ONE OBSIDIAN VAULT INTO A SHARED BRAIN FOR 5 CLAUDE AGENTS BUILT TO HANDLE 80% OF A SOLO DEVELOPER’S WORK. 00:03 he zooms out and hundreds of connected notes appear on screen. it is the perfect visual for Claude working from one shared project memory instead of starting from zero every session. most developers give Claude 1 file and 1 prompt. this setup gives it the architecture, coding rules, tests, client briefs, and previous decisions before it writes anything. obsidian becomes the memory layer, storing everything in markdown files Claude can read alongside the codebase. every agent works from the same context instead of repeating the same questions. then 5 specialists split the job. one researches, one plans, one codes, one reviews, and one deploys while the developer only controls the important decisions. one vault, 5 agents, 2 model tiers, and a $50 to $150 monthly stack can turn Claude from a chatbot into a small development team.

Gipp 🦅

10,799 views • 2 months ago

Before Claude Fable 5 got banned, I turned all my fine-tuning research and experiments into a product: (A CLI for data generation to fine-tune AI models). Finetuner uses Codex 5.5 or Opus 4.8 as the orchestrator and Chinese models (DeepSeek v4 Pro, Kimi K2.7, MiMo v2.5, etc.) to generate dataset rows. This system is unique because these are not Python-script-paraphrased datasets. Each and every row is handcrafted by Chinese models based on the orchestrator model's instructions. Now anyone can generate datasets to fine-tune small language models (1B to 30B models). I achieved 10x lower costs and 5x better dataset quality. Releasing the product in a few days. (I'll open-source the skills.)

Before Claude Fable 5 got banned, I turned all my fine-tuning research and experiments into a product: (A CLI for data generation to fine-tune AI models). Finetuner uses Codex 5.5 or Opus 4.8 as the orchestrator and Chinese models (DeepSeek v4 Pro, Kimi K2.7, MiMo v2.5, etc.) to generate dataset rows. This system is unique because these are not Python-script-paraphrased datasets. Each and every row is handcrafted by Chinese models based on the orchestrator model's instructions. Now anyone can generate datasets to fine-tune small language models (1B to 30B models). I achieved 10x lower costs and 5x better dataset quality. Releasing the product in a few days. (I'll open-source the skills.)

CJ Zafir

40,211 views • 3 months ago

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 views • 4 months ago

I find it wild that Physical Intelligence robotics models can work at all given that they only have a 1 second context length. As the robot is moving around a room and cleaning up a table, it only sees the current timestamp of images from the cameras (plus the text description of its task). I asked Sergey Levine was how this model can work at all. If a human had to execute a many minutes long plan in order to accomplish a physical task, but had her memory reset every second, there's no way she could pull it off But somehow just knowing what state the world is literally in this moment is all you need to know to keep a reasonably complication plan going? Sergey's answer calls on Morevac's Paradox, and I thought it was quite interesting. Full episode out tomorrow!

I find it wild that Physical Intelligence robotics models can work at all given that they only have a 1 second context length. As the robot is moving around a room and cleaning up a table, it only sees the current timestamp of images from the cameras (plus the text description of its task). I asked Sergey Levine was how this model can work at all. If a human had to execute a many minutes long plan in order to accomplish a physical task, but had her memory reset every second, there's no way she could pull it off But somehow just knowing what state the world is literally in this moment is all you need to know to keep a reasonably complication plan going? Sergey's answer calls on Morevac's Paradox, and I thought it was quite interesting. Full episode out tomorrow!

Dwarkesh Patel

126,193 views • 1 year ago

MICROSOFT TRIED TO BUY IT. HARVARD AND THREE US UNIVERSITIES RELEASED IT FOR FREE a team from Harvard, Michigan State, Minnesota, and Georgia ran the first hard study on how agent memory behaves over time. the finding is uncomfortab they call it the "experience-following property": when a new task looks like a memory the agent pulls up, its answer copies that memory's answer almost 1:1 (correlation up to 0.95). the agent isn't reasoning - it's imitating that breaks in two ways: > error propagation - one wrong memory repeats on every similar task after it > misaligned replay - "successful" past runs turn out low-value or misleading when reused so "just add more memory" quietly makes agents worse. what fixed it: > store only high-quality experiences, not everything > use future task outcomes as free labels to delete low-value memories > pair selective adding with utility-based deletion the payoff: ~10% absolute accuracy gains, a cleaner memory bank, and agents stable under distribution shift no company owns this. they dropped the whole thing open-source, for free bookmark this. the next edge in AI agents isn't more memory - it's cleaner memory

MICROSOFT TRIED TO BUY IT. HARVARD AND THREE US UNIVERSITIES RELEASED IT FOR FREE a team from Harvard, Michigan State, Minnesota, and Georgia ran the first hard study on how agent memory behaves over time. the finding is uncomfortab they call it the "experience-following property": when a new task looks like a memory the agent pulls up, its answer copies that memory's answer almost 1:1 (correlation up to 0.95). the agent isn't reasoning - it's imitating that breaks in two ways: > error propagation - one wrong memory repeats on every similar task after it > misaligned replay - "successful" past runs turn out low-value or misleading when reused so "just add more memory" quietly makes agents worse. what fixed it: > store only high-quality experiences, not everything > use future task outcomes as free labels to delete low-value memories > pair selective adding with utility-based deletion the payoff: ~10% absolute accuracy gains, a cleaner memory bank, and agents stable under distribution shift no company owns this. they dropped the whole thing open-source, for free bookmark this. the next edge in AI agents isn't more memory - it's cleaner memory

NO1ennn

27,826 views • 1 month ago

$QVAC SDK will support in 0.9.0 (gonna be release in ~10 days) LoRA fine-tuning directly on-device, letting developers customize LLMs with their own data without sending anything to the cloud. You just load a base model, point it at your training dataset, and get a lightweight LoRA adapter back — all running locally. The fine-tuned model can then be used for inference immediately, with no extra setup. Why it matters: LoRA (Low-Rank Adaptation) fine-tuning lets you specialize a general-purpose language model for your specific use case — like matching a brand's tone, mastering domain terminology, or following a particular output format — using a fraction of the compute a full fine-tune would require. QVAC handles the entire workflow locally: dataset preparation, training with configurable hyperparameters, checkpoint saving, and seamless inference with the resulting adapter. Your data never leaves the device. The developer experience: Fine-tuning with QVAC is as simple as calling "sdk.finetune()" with your dataset and a few hyperparameters. Training runs entirely on your local hardware, produces a compact LoRA adapter file, and supports pause/resume so you can stop a job and pick it back up without losing progress. The result plugs straight into QVAC's inference pipeline — no model conversion, no deployment step, just immediate local completions with your fine-tuned model.$

QVAC SDK will support in 0.9.0 (gonna be release in ~10 days) LoRA fine-tuning directly on-device, letting developers customize LLMs with their own data without sending anything to the cloud. You just load a base model, point it at your training dataset, and get a lightweight LoRA adapter back — all running locally. The fine-tuned model can then be used for inference immediately, with no extra setup. Why it matters: LoRA (Low-Rank Adaptation) fine-tuning lets you specialize a general-purpose language model for your specific use case — like matching a brand's tone, mastering domain terminology, or following a particular output format — using a fraction of the compute a full fine-tune would require. QVAC handles the entire workflow locally: dataset preparation, training with configurable hyperparameters, checkpoint saving, and seamless inference with the resulting adapter. Your data never leaves the device. The developer experience: Fine-tuning with QVAC is as simple as calling "sdk.finetune()" with your dataset and a few hyperparameters. Training runs entirely on your local hardware, produces a compact LoRA adapter file, and supports pause/resume so you can stop a job and pick it back up without losing progress. The result plugs straight into QVAC's inference pipeline — no model conversion, no deployment step, just immediate local completions with your fine-tuned model.

Paolo Ardoino 🤖

42,413 views • 5 months ago

The 1000th episode of the No Laying Up podcast is available now in your feeds and we take a look back at the winding road we took to get here. Favorite episodes, hardest you’ve laughed, most stressful moments and more - this one is a fun trip down memory lane and a great reminder of how NLU has grown and changed over the years. No Laying Up Episode 1000 is available now wherever you get your podcasts, we hope you enjoy it.

The 1000th episode of the No Laying Up podcast is available now in your feeds and we take a look back at the winding road we took to get here. Favorite episodes, hardest you’ve laughed, most stressful moments and more - this one is a fun trip down memory lane and a great reminder of how NLU has grown and changed over the years. No Laying Up Episode 1000 is available now wherever you get your podcasts, we hope you enjoy it.

No Laying Up

101,065 views • 1 year ago

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

467,237 views • 1 year ago

researchers gave a tiny local model human-style memory and its context limit basically stopped existing a team from MBZUAI, Princeton and Weizmann took a 1B model and rebuilt how it reads. instead of attending to everything at once, the model reads in 1,024 token chunks and passes the important stuff forward through an associative memory, the same way you carry the plot of a book between chapters without rereading them. the design mirrors human memory on purpose. full attention inside a chunk works as short-term memory. the module that carries information between chunks works as long-term memory. they even trained it like a person, starting with short easy texts and raising the difficulty gradually, because memory thrown into the deep end learns nothing. the numbers back it up. the normal model burns 40GB of GPU memory on a long document and collapses hard past its limit, dropping from 0.86 to 0.32 accuracy. the memory version holds 0.71 at double that length while using a flat 12GB no matter how long the input gets. it also needs about 30% fewer FLOPs. the part i keep thinking about is that nobody scaled anything here. they didn't build a bigger model, didn't stretch the window, didn't add compute. they looked at how a brain handles a long day and copied the architecture. a model small enough to run on a consumer gpu now survives documents its own architecture used to choke on. we keep treating intelligence as a compute problem. sometimes it's a memory problem.

researchers gave a tiny local model human-style memory and its context limit basically stopped existing a team from MBZUAI, Princeton and Weizmann took a 1B model and rebuilt how it reads. instead of attending to everything at once, the model reads in 1,024 token chunks and passes the important stuff forward through an associative memory, the same way you carry the plot of a book between chapters without rereading them. the design mirrors human memory on purpose. full attention inside a chunk works as short-term memory. the module that carries information between chunks works as long-term memory. they even trained it like a person, starting with short easy texts and raising the difficulty gradually, because memory thrown into the deep end learns nothing. the numbers back it up. the normal model burns 40GB of GPU memory on a long document and collapses hard past its limit, dropping from 0.86 to 0.32 accuracy. the memory version holds 0.71 at double that length while using a flat 12GB no matter how long the input gets. it also needs about 30% fewer FLOPs. the part i keep thinking about is that nobody scaled anything here. they didn't build a bigger model, didn't stretch the window, didn't add compute. they looked at how a brain handles a long day and copied the architecture. a model small enough to run on a consumer gpu now survives documents its own architecture used to choke on. we keep treating intelligence as a compute problem. sometimes it's a memory problem.

Alex Veremeyenko

16,147 views • 2 months ago

Apple built a large foundation model and fine-tuned it on multiple tasks. But they are doing something very clever: They load a single model in memory and use different adapters to specialize the model on the fly. I recorded a video to show you how to write the code to do the same thing Apple is doing. I explain everything step by step. Here is what I'll show you in the video: 1. We'll load two datasets 2. Then load a large model 3. Then, we'll fine-tune the model on both datasets I'll use LoRA to fine-tune the model. This process creates two small adapters, each specializing in solving one of the datasets. The base model's original parameters will remain unchanged. From here: 4. We'll generate a list of tasks 5. We'll load the correct adapter to solve each task The large model I'm using needs 346 MB of memory, but I only need to load it once. Each adapter is only 2.7 MB. I only need to load the base model once and pair it with any of the fine-tuned adapters. Minimum memory footprint and I can solve multiple tasks. Hope this helps!

Apple built a large foundation model and fine-tuned it on multiple tasks. But they are doing something very clever: They load a single model in memory and use different adapters to specialize the model on the fly. I recorded a video to show you how to write the code to do the same thing Apple is doing. I explain everything step by step. Here is what I'll show you in the video: 1. We'll load two datasets 2. Then load a large model 3. Then, we'll fine-tune the model on both datasets I'll use LoRA to fine-tune the model. This process creates two small adapters, each specializing in solving one of the datasets. The base model's original parameters will remain unchanged. From here: 4. We'll generate a list of tasks 5. We'll load the correct adapter to solve each task The large model I'm using needs 346 MB of memory, but I only need to load it once. Each adapter is only 2.7 MB. I only need to load the base model once and pair it with any of the fine-tuned adapters. Minimum memory footprint and I can solve multiple tasks. Hope this helps!

Santiago

84,747 views • 2 years ago