Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

✨ Introducing Keypoint Action Tokens. 🤖 We translate visual observations and robot actions into a "language" that off-the-shelf LLMs can ingest and output. This transforms LLMs into in-context, low-level imitation learning machines. 🚀 Let me explain. 👇🧵

Norman Di Palo

1,522 subscribers

23,097 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & Technologie Bildung

Anya Rossi• Live Now

Private livecam show

11 Kommentare

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

We represent visual inputs as keypoint tokens, and robot poses as action tokens. Keypoints are extracted via DINO, projected in 3D, and represented as text. The gripper pose is represented as a triplet of 3D points in the same euclidean space as the keypoints. There's more👇🧵

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

Recording a demonstration means storing keypoint tokens from the scene and the action tokens of the trajectory. The numbers are represented as text (tokens), and can therefore go into an LLM prompt. This is where things get interesting... 👇🧵

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

With the Keypoint Action Tokens "language", we unlock the pattern extraction and completion abilities of LLMs, from language to robotics. Given keypoint tokens from a new scene, we demonstrate that LLMs can output an effective trajectory of action tokens! 👇🧵

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

Our experiments demonstrate that: 1) KAT surpasses SOTA methods like Diffusion Policies (DP) 2) the KAT representation can boost DP performance as well, improving on the end-to-end version KAT is also robust to distractors, as we show in the third execution of the video. 👇🧵

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

A fascinating result is that KAT's performance improves as the underlying LLMs improve. Using GPT-4 Turbo outperforms GPT-3.5 Turbo. This suggests that the improvement and scaling of LLMs can lead to free improvement to robotics, using LLMs as imitation learning machines. 👇🧵

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

In-context learning also means *zero training time*, and the ability to deploy and test skills immediately, or store them as text and retrieve them later. All our pipeline relies on pre-trained Foundation Models and can be deployed immediately. 👇🧵

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

Therefore, we show one of the most interesting uses of LLMs in robotics is *without any natural language*. Surprising! 💡 Supervised by @Ed__Johns. Paper: Website with videos:

Profilbild von XXXin

XXXinvor 1 Jahr

Does it mean the user gives no prompt but the debit will determine which task to perform based on the initial scene configuration? What comes next?

Profilbild von Norman Di Palo

Norman Di Palovor 1 Jahr

This method was trained on single tasks in isolation. Extending it to receive language commands has been done eg here

Profilbild von ऋषिक तिवारी (Rishik Tiwari)

ऋषिक तिवारी (Rishik Tiwari)vor 2 Jahren

Really interesting, I lately have been trying to do something similar but with egocentric cam and affordance maps (inspired from VoxPoser). While KAT may work for topdown RGBD, ego views will be difficult as it gets obstructed after grasping the obj.

Profilbild von Norman Di Palo

Norman Di Palovor 2 Jahren

Interesting! I worked quite a bit with wrist-cams and they offer advantages and disadvantages. I am currently working on some wrist-cam based extensions.

Ähnliche Videos

Very excited to announce: Keypoint Action Tokens! We found that LLMs can be repurposed as "imitation learning engines" for robots, by representing both observations & actions as 3D keypoints, and feeding into an LLM for in-context learning. See: More 👇

Very excited to announce: Keypoint Action Tokens! We found that LLMs can be repurposed as "imitation learning engines" for robots, by representing both observations & actions as 3D keypoints, and feeding into an LLM for in-context learning. See: More 👇

Edward Johns

32,569 Aufrufe • vor 2 Jahren

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Kaustubh Sridhar

52,158 Aufrufe • vor 11 Monaten

With OpenAI, Figure 01 can now have full conversations with people -OpenAI models provide high-level visual and language intelligence -Figure neural networks deliver fast, low-level, dexterous robot actions Everything in this video is a neural network:

With OpenAI, Figure 01 can now have full conversations with people -OpenAI models provide high-level visual and language intelligence -Figure neural networks deliver fast, low-level, dexterous robot actions Everything in this video is a neural network:

Figure

5,101,798 Aufrufe • vor 2 Jahren

Yann LeCun says language isn’t intelligence. Predicting text doesn’t mean understanding reality. The real world is messy, physical, and causal and today’s LLMs barely touch that. The next leap is Physical AI: world models, cause and effect, real planning. Do you think LLMs can evolve into this, or do we need a completely new architecture?

Yann LeCun says language isn’t intelligence. Predicting text doesn’t mean understanding reality. The real world is messy, physical, and causal and today’s LLMs barely touch that. The next leap is Physical AI: world models, cause and effect, real planning. Do you think LLMs can evolve into this, or do we need a completely new architecture?

VraserX e/acc

76,178 Aufrufe • vor 5 Monaten

Your bimanual manipulators might need a Robot Neck 🤖🦒 Introducing Vision in Action: Learning Active Perception from Human Demonstrations ViA learns task-specific, active perceptual strategies—such as searching, tracking, and focusing—directly from human demos, enabling robust visuomotor policies under visual occlusions. 🧵👇

Your bimanual manipulators might need a Robot Neck 🤖🦒 Introducing Vision in Action: Learning Active Perception from Human Demonstrations ViA learns task-specific, active perceptual strategies—such as searching, tracking, and focusing—directly from human demos, enabling robust visuomotor policies under visual occlusions. 🧵👇

Haoyu Xiong

123,129 Aufrufe • vor 1 Jahr

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,708 Aufrufe • vor 3 Jahren

Andrej Karpathy beautifully explains the fundamental difference of learning between a human and an LLM. > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” many of the best minds in this field think LLMs are not really learning anything, and therefore are incapable of surpassing human-level intelligence.

Andrej Karpathy beautifully explains the fundamental difference of learning between a human and an LLM. > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” many of the best minds in this field think LLMs are not really learning anything, and therefore are incapable of surpassing human-level intelligence.

ℏεsam

895,122 Aufrufe • vor 9 Monaten

Turn any GitHub repository into LLM-ready text! Simply replace "hub" with "ingest" in a GitHub URL and receive a prompt-friendly text ingest for LLMs. Gitingest is 100% open-source and provides: - Directory structure - A brief summary of the project - The entire content as LLM-ready text Plus, it comes with a nice python package and you can run the UI locally! Stay tuned, I'm working on something really cool with this!✨ Link to the GitHub repo in the next tweet! _____ Find me → Akshay 🚀 ✔️ For more insights and tutorials on ML and AI Engineering!

Turn any GitHub repository into LLM-ready text! Simply replace "hub" with "ingest" in a GitHub URL and receive a prompt-friendly text ingest for LLMs. Gitingest is 100% open-source and provides: - Directory structure - A brief summary of the project - The entire content as LLM-ready text Plus, it comes with a nice python package and you can run the UI locally! Stay tuned, I'm working on something really cool with this!✨ Link to the GitHub repo in the next tweet! _____ Find me → Akshay 🚀 ✔️ For more insights and tutorials on ML and AI Engineering!

Akshay 🚀

191,382 Aufrufe • vor 1 Jahr

In the era of long-context LLMs, I needed a way to quickly convert a GitHub repo into a text file to feed as a prompt. Introducing RepoToTextForLLMs 📖 Not only does it return the full repo in one file, but it also appends a super prompt for analysis and understanding 👇

In the era of long-context LLMs, I needed a way to quickly convert a GitHub repo into a text file to feed as a prompt. Introducing RepoToTextForLLMs 📖 Not only does it return the full repo in one file, but it also appends a super prompt for analysis and understanding 👇

Pietro Schirano

104,906 Aufrufe • vor 2 Jahren

.Richard Sutton, father of reinforcement learning, doesn’t think LLMs are bitter-lesson-pilled. My steel man of Richard’s position: we need some new architecture to enable continual (on-the-job) learning. And if we have continual learning, we don't need a special training phase - the agent just learns on-the-fly - like all humans, and indeed, like all animals. This new paradigm will render our current approach with LLMs obsolete. I did my best to represent the view that LLMs will function as the foundation on which this experiential learning can happen. Some sparks flew. 0:00:00 – Are LLMs a dead-end? 0:13:51 – Do humans do imitation learning? 0:23:57 – The Era of Experience 0:34:25 – Current architectures generalize poorly out of distribution 0:42:17 – Surprises in the AI field 0:47:28 – Will The Bitter Lesson still apply after AGI? 0:54:35 – Succession to AI

.Richard Sutton, father of reinforcement learning, doesn’t think LLMs are bitter-lesson-pilled. My steel man of Richard’s position: we need some new architecture to enable continual (on-the-job) learning. And if we have continual learning, we don't need a special training phase - the agent just learns on-the-fly - like all humans, and indeed, like all animals. This new paradigm will render our current approach with LLMs obsolete. I did my best to represent the view that LLMs will function as the foundation on which this experiential learning can happen. Some sparks flew. 0:00:00 – Are LLMs a dead-end? 0:13:51 – Do humans do imitation learning? 0:23:57 – The Era of Experience 0:34:25 – Current architectures generalize poorly out of distribution 0:42:17 – Surprises in the AI field 0:47:28 – Will The Bitter Lesson still apply after AGI? 0:54:35 – Succession to AI

Dwarkesh Patel

3,081,359 Aufrufe • vor 9 Monaten

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

AI web agents like Operator and Anthropic’s Computer Use can operate a browser, but the LLMs inside are brittle, and you can’t trust what’s on the web. In this 🧵, I’ll show how adversaries can fool Anthropic’s web agent into sending phishing emails or revealing credit card info.

Micah Goldblum

42,969 Aufrufe • vor 1 Jahr

Can we build a generalist robotic policy that doesn’t just memorize training data and regurgitate it during test time, but instead remembers past actions as memory and conditions its decisions on them?🤖💡 Introducing SAM2Act—a multi-view robotic transformer-based policy that integrates a visual foundation model with a memory architecture for robotic manipulation. Project page: 🧵👇

Can we build a generalist robotic policy that doesn’t just memorize training data and regurgitate it during test time, but instead remembers past actions as memory and conditions its decisions on them?🤖💡 Introducing SAM2Act—a multi-view robotic transformer-based policy that integrates a visual foundation model with a memory architecture for robotic manipulation. Project page: 🧵👇

Jiafei Duan

87,573 Aufrufe • vor 1 Jahr

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Announcing How Transformer LLMs Work, created with Jay Alammar and Maarten Grootendorst, co-authors of the beautifully illustrated book, “Hands-On Large Language Models.” This course offers a deep dive into the inner workings of the transformer architecture that powers large language models (LLMs). The transformer architecture revolutionized generative AI; in fact, the "GPT" in ChatGPT stands for "Generative Pre-Trained Transformer." Originally introduced in the Google Brain team's groundbreaking 2017 paper "Attention Is All You Need," by Vaswani and others, transformers were a highly scalable model for machine translation tasks. Variants of this architecture now power today’s LLMs such as those from OpenAI, Google, Meta, Cohere, Anthropic and DeepSeek. In this course, you’ll learn in detail how LLMs process text. You'll also work through code examples that illustrate that transformer's individual components. In details, you’ll learn: - How the representation of language has evolved, from Bag-of-Words to Word2Vec embeddings to the transformer architecture that captures a word's meanings taking into account the context of other words in the input. - How inputs are broken down into tokens before they are sent to the language model. - The details of a transformer's main stages: Tokenization and embedding, the stack of transformer blocks, and the language model head. - The inner workings of the transformer block, including attention, which calculates relevance scores, and the feedforward layer, which incorporates stored information learned in training. - How cached calculations make transformers faster. - Some of the most recent ideas in the latest models such as Mixture-of-Experts (MoE) which uses multiple sub-models and a router on each layer to improve the quality of LLMs. By the end of this course, you’ll have a deep understanding of how LLMs actually process text and be able to read through papers describing the latest models and understand the details. Gaining this intuition will improve your approach to building LLM applications. Please sign up here:

Andrew Ng

253,812 Aufrufe • vor 1 Jahr

Introducing MindOS: a playful art experiment in building a mind from the inside out. Turning memory, sensation, personality, and the subconscious into a living, generative interface. Using Google’s Google Antigravity tech to give the inner world a visual home. Several locally run LLMs, multimodal models, and Stable Diffusion. Thanks Robert Scoble for the prompt to create this! More to come

Introducing MindOS: a playful art experiment in building a mind from the inside out. Turning memory, sensation, personality, and the subconscious into a living, generative interface. Using Google’s Google Antigravity tech to give the inner world a visual home. Several locally run LLMs, multimodal models, and Stable Diffusion. Thanks Robert Scoble for the prompt to create this! More to come

Brad Necyk

69,414 Aufrufe • vor 7 Monaten

Learning from robot data? Standard. Direct Video-Action Models (DVA) is different: treat robot control as video generation, then translate the generated video into actions. Built by , the system pre-trains causal video models from scratch and can run complex production tasks for hours using only ~10 hours of robot data. • hundreds of frames of visual context • real-time control via causal video prediction More: The team behind it just exited 18 months of stealth with a $450M Series A at a $1.7B valuation. Founded by Jagdeep Singh (ex-QuantumScape) with a Stanford-heavy science team: CSO Eric Ryan Chan (ex-WorldLabs) and Prof. Gordon Wetzstein. Already running in large-scale automotive production environments. Backed by Vinod Khosla Ventures, Temasek, Premji Invest, and John Doerr. Thanks for sharing, Tongzhou Mu 🤖🦾🦿 👋

Learning from robot data? Standard. Direct Video-Action Models (DVA) is different: treat robot control as video generation, then translate the generated video into actions. Built by , the system pre-trains causal video models from scratch and can run complex production tasks for hours using only ~10 hours of robot data. • hundreds of frames of visual context • real-time control via causal video prediction More: The team behind it just exited 18 months of stealth with a $450M Series A at a $1.7B valuation. Founded by Jagdeep Singh (ex-QuantumScape) with a Stanford-heavy science team: CSO Eric Ryan Chan (ex-WorldLabs) and Prof. Gordon Wetzstein. Already running in large-scale automotive production environments. Backed by Vinod Khosla Ventures, Temasek, Premji Invest, and John Doerr. Thanks for sharing, Tongzhou Mu 🤖🦾🦿 👋

Ilir Aliu

26,209 Aufrufe • vor 4 Monaten

What if your thoughts aren’t your own? Professors Elan Barenholtz and William Hahn propose that language is not a tool we use but a self-generating organism that uses us. In this mind-bending live discussion at the University of Toronto, they argue that language installs itself in our minds like software, runs autonomously, and shapes behavior at a deeper level than we realize. Drawing on LLMs, autoregression, and cognitive science, they suggest your brain may function like a predictive engine and that “memory,” “self,” and even “God” may just be tokens in an informational system.

What if your thoughts aren’t your own? Professors Elan Barenholtz and William Hahn propose that language is not a tool we use but a self-generating organism that uses us. In this mind-bending live discussion at the University of Toronto, they argue that language installs itself in our minds like software, runs autonomously, and shapes behavior at a deeper level than we realize. Drawing on LLMs, autoregression, and cognitive science, they suggest your brain may function like a predictive engine and that “memory,” “self,” and even “God” may just be tokens in an informational system.

Curt Jaimungal

52,269 Aufrufe • vor 1 Jahr

AMI Labs founder Yann LeCun on why LLMs are fooling us the same way AI has for decades: He argues that every generation of AI scientists has made the same mistake: confusing task performance with real intelligence. LeCun's core challenge to the current hype: "We're fooled into thinking those machines are intelligent because they can manipulate language. And we're used to the fact that people who can manipulate language very well are implicitly smart." He's clear that LLMs are useful, but being a useful tool and being intelligent are two very different things. The real insight is the historical pattern he's lived through. Since the 1950s, wave after wave of AI researchers have claimed their breakthrough was the path to human-level intelligence. Marvin Minsky. Herbert Simon. Frank Rosenblatt — who invented the perceptron, the first learning machine, in the 1950s — all predicted machines as smart as humans within a decade. "They were all wrong." LeCun has personally witnessed three of these cycles of hype and disappointment. And his verdict on the current one is blunt: "This generation with LLMs is also wrong. It's just another example of being fooled." The pattern: A new technique emerges → machines get good at specific tasks → we assume general intelligence The question worth asking: are we impressed by these tools because they're intelligent, or because they sound like they are?

AMI Labs founder Yann LeCun on why LLMs are fooling us the same way AI has for decades: He argues that every generation of AI scientists has made the same mistake: confusing task performance with real intelligence. LeCun's core challenge to the current hype: "We're fooled into thinking those machines are intelligent because they can manipulate language. And we're used to the fact that people who can manipulate language very well are implicitly smart." He's clear that LLMs are useful, but being a useful tool and being intelligent are two very different things. The real insight is the historical pattern he's lived through. Since the 1950s, wave after wave of AI researchers have claimed their breakthrough was the path to human-level intelligence. Marvin Minsky. Herbert Simon. Frank Rosenblatt — who invented the perceptron, the first learning machine, in the 1950s — all predicted machines as smart as humans within a decade. "They were all wrong." LeCun has personally witnessed three of these cycles of hype and disappointment. And his verdict on the current one is blunt: "This generation with LLMs is also wrong. It's just another example of being fooled." The pattern: A new technique emerges → machines get good at specific tasks → we assume general intelligence The question worth asking: are we impressed by these tools because they're intelligent, or because they sound like they are?

Big Brain AI

126,482 Aufrufe • vor 5 Monaten