Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Exploration is key for robots to generalize, especially in open-ended environments with vague goals and sparse rewards. BUT, how do we go beyond random poking? Wouldn't it be great to have a robot that explores an environment just like a kid? Introducing Imagine, Verify, Execute (IVE)! IVE leverages Vision-Language... show more

Jia-Bin Huang

79,355 subscribers

45,352 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & Teknoloji Eğitim

Anya Rossi• Live Now

Private livecam show

5 Yorum

Jia-Bin Huang profil fotoğrafı

Jia-Bin Huang1 yıl önce

Brought to you by the amazing @umdcs students Seungjae Lee @JayLEE_0301, Daniel Ekpo (@daniekpo7), Haowen Liu, and my colleagues @furongh and @abhi2610 Check out the project page for more visual results!

The Rundown AI profil fotoğrafı

The Rundown AI2 yıl önce

AI won't replace you, but a person using AI will. Join 500,000+ readers and learn how to use AI in just 5 minutes a day (for free).

Wenhu Chen profil fotoğrafı

Wenhu Chen1 yıl önce

Great work! Congrats!

Jia-Bin Huang profil fotoğrafı

Jia-Bin Huang1 yıl önce

Thanks, @WenhuChen !

Roei Herzig profil fotoğrafı

Roei Herzig1 yıl önce

Very cool!

Benzer Videolar

💡Can robots autonomously design their own tools and figure out how to use them? We present VLMgineer 🛠️, a framework that leverages Vision Language Models with Evolutionary Search to automatically generate and refine physical tool designs alongside corresponding robot action plans. ✨ VLMgineer can fully automate tool and action design with AI-driven physical creativity. No human intervention. No pre-defined templates or few-shot examples. ✨ VLMgineer outperforms human-specified designs and existing everyday tools. ✨ We let the VLM fully decide how to evolve designs. Deep dive with me: 🧵

💡Can robots autonomously design their own tools and figure out how to use them? We present VLMgineer 🛠️, a framework that leverages Vision Language Models with Evolutionary Search to automatically generate and refine physical tool designs alongside corresponding robot action plans. ✨ VLMgineer can fully automate tool and action design with AI-driven physical creativity. No human intervention. No pre-defined templates or few-shot examples. ✨ VLMgineer outperforms human-specified designs and existing everyday tools. ✨ We let the VLM fully decide how to evolve designs. Deep dive with me: 🧵

Junyao Shi

29,764 görüntüleme • 1 yıl önce

Jony Ive: I refuse to speculate on how Steve Jobs would react to this Ferrari. For a decade, the tech world waited for an Apple Car that never arrived. Instead, its spiritual successor just rolled out of Maranello. Jony Ive designed the interior of Ferrari's first electric vehicle, the Luce. And his first strategic move is a massive rejection. Apple inadvertently ruined modern car interiors. The iPad trained every automaker to glue a giant screen to the dashboard and kill physical controls. Now, Ive is reversing that trend. For the Ferrari Luce, he rejected the giant touchscreen in favor of tactile, mechanical buttons and advanced glass. When asked how Steve Jobs would react to the car, Ive refused to guess. But he admitted he wouldn't even be in the room without his history with Jobs. I love the irony here. The mythical Apple Car finally exists inside a Ferrari. And instead of pushing more screens into the cabin, Ive is using the design foundation he built with Jobs to prove that the future of luxury is tactile. Source: Cleo Abram

Jony Ive: I refuse to speculate on how Steve Jobs would react to this Ferrari. For a decade, the tech world waited for an Apple Car that never arrived. Instead, its spiritual successor just rolled out of Maranello. Jony Ive designed the interior of Ferrari's first electric vehicle, the Luce. And his first strategic move is a massive rejection. Apple inadvertently ruined modern car interiors. The iPad trained every automaker to glue a giant screen to the dashboard and kill physical controls. Now, Ive is reversing that trend. For the Ferrari Luce, he rejected the giant touchscreen in favor of tactile, mechanical buttons and advanced glass. When asked how Steve Jobs would react to the car, Ive refused to guess. But he admitted he wouldn't even be in the room without his history with Jobs. I love the irony here. The mythical Apple Car finally exists inside a Ferrari. And instead of pushing more screens into the cabin, Ive is using the design foundation he built with Jobs to prove that the future of luxury is tactile. Source: Cleo Abram

Tony Jacob | FindaClip.com

105,109 görüntüleme • 1 ay önce

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

465,968 görüntüleme • 1 yıl önce

LARP: Language-Agent Role Play for Open-World Games paper page: Language agents have shown impressive problem-solving skills within defined settings and brief timelines. Yet, with the ever-evolving complexities of open-world simulations, there's a pressing need for agents that can flexibly adapt to complex environments and consistently maintain a long-term memory to ensure coherent actions. To bridge the gap between language agents and open-world games, we introduce Language Agent for Role-Playing (LARP), which includes a cognitive architecture that encompasses memory processing and a decision-making assistant, an environment interaction module with a feedback-driven learnable action space, and a postprocessing method that promotes the alignment of various personalities. The LARP framework refines interactions between users and agents, predefined with unique backgrounds and personalities, ultimately enhancing the gaming experience in open-world contexts. Furthermore, it highlights the diverse uses of language models in a range of areas such as entertainment, education, and various simulation scenarios.

LARP: Language-Agent Role Play for Open-World Games paper page: Language agents have shown impressive problem-solving skills within defined settings and brief timelines. Yet, with the ever-evolving complexities of open-world simulations, there's a pressing need for agents that can flexibly adapt to complex environments and consistently maintain a long-term memory to ensure coherent actions. To bridge the gap between language agents and open-world games, we introduce Language Agent for Role-Playing (LARP), which includes a cognitive architecture that encompasses memory processing and a decision-making assistant, an environment interaction module with a feedback-driven learnable action space, and a postprocessing method that promotes the alignment of various personalities. The LARP framework refines interactions between users and agents, predefined with unique backgrounds and personalities, ultimately enhancing the gaming experience in open-world contexts. Furthermore, it highlights the diverse uses of language models in a range of areas such as entertainment, education, and various simulation scenarios.

AK

143,974 görüntüleme • 2 yıl önce

Yann LeCun (Yann LeCun) just revealed why he left Meta, Why are LLMs an extremely narrow field, and how "World model" is the way to build really meaningful agentic future. 🎯 Beautiful and simple in 1.5 mints. "I can’t imagine we can build agentic systems without those systems having the ability to predict, in advance, what the consequences of their actions are going to be. The way we act in the world is that we can predict the consequences of our actions, and that’s what allows us to plan. So what is a world model? Given the state of an environment, a system you want to control at time t, and given an action or intervention you imagine taking, can you predict the state of the world (or the system) at time t + 1? If you can, that’s a world model. You don’t do this at a pixel level, if it’s video. You do this in an abstract representation space, and that’s a crucial key insight." --- From 'AI House Davos" YT channel (full link in comment)

Yann LeCun (Yann LeCun) just revealed why he left Meta, Why are LLMs an extremely narrow field, and how "World model" is the way to build really meaningful agentic future. 🎯 Beautiful and simple in 1.5 mints. "I can’t imagine we can build agentic systems without those systems having the ability to predict, in advance, what the consequences of their actions are going to be. The way we act in the world is that we can predict the consequences of our actions, and that’s what allows us to plan. So what is a world model? Given the state of an environment, a system you want to control at time t, and given an action or intervention you imagine taking, can you predict the state of the world (or the system) at time t + 1? If you can, that’s a world model. You don’t do this at a pixel level, if it’s video. You do this in an abstract representation space, and that’s a crucial key insight." --- From 'AI House Davos" YT channel (full link in comment)

Rohan Paul

383,162 görüntüleme • 5 ay önce

This is THE moment of Physical AI! We are officially announcing Cosmos 3: Omnimodal World Models for Physical AI 🚀 - Cosmos 3 is an omnimodal world model: within a unified architecture, it can understand and generate language, images, video, audio, and actions. - It is not just a VLM, not just a video generator, not just an audio-visual generative model, and not just a physics simulator / world-action model. It can understand images and videos, generate images, videos, and audio, simulate future worlds, predict actions, and generate robot policies—enabling models to truly begin to “touch the world.” - Cosmos 3 is the #1 open-weight reasoner / T2I / I2V / robot policy across many benchmarks. Huge thanks to every teammate who fought side by side on this journey—from architecture, data, training, infra, serving, and evaluation to post-training. Every part of this project carries an incredible amount of hard work. This was my first time leading a project as Tech Lead, and I feel truly fortunate. The future of Physical AI needs models that can not only “see” and “describe” the world, but also “imagine,” “simulate,” and “act”—and eventually close the loop with the real world. I hope Cosmos 3 can become an important starting point for this direction, and I’m excited to push Physical AI into its next stage together with the open-source community. Welcome to the era of Physical AI. HuggingFace: Project Website: Code:

This is THE moment of Physical AI! We are officially announcing Cosmos 3: Omnimodal World Models for Physical AI 🚀 - Cosmos 3 is an omnimodal world model: within a unified architecture, it can understand and generate language, images, video, audio, and actions. - It is not just a VLM, not just a video generator, not just an audio-visual generative model, and not just a physics simulator / world-action model. It can understand images and videos, generate images, videos, and audio, simulate future worlds, predict actions, and generate robot policies—enabling models to truly begin to “touch the world.” - Cosmos 3 is the #1 open-weight reasoner / T2I / I2V / robot policy across many benchmarks. Huge thanks to every teammate who fought side by side on this journey—from architecture, data, training, infra, serving, and evaluation to post-training. Every part of this project carries an incredible amount of hard work. This was my first time leading a project as Tech Lead, and I feel truly fortunate. The future of Physical AI needs models that can not only “see” and “describe” the world, but also “imagine,” “simulate,” and “act”—and eventually close the loop with the real world. I hope Cosmos 3 can become an important starting point for this direction, and I’m excited to push Physical AI into its next stage together with the open-source community. Welcome to the era of Physical AI. HuggingFace: Project Website: Code:

Max Zhaoshuo Li 李赵硕 ✈️ RSS

1,077,988 görüntüleme • 1 ay önce

We all have dreams and aspirations, but it’s important to have a VISION that goes beyond your personal goals. It’s not just about setting goals, but it’s about having a purpose that’s bigger than yourself. it’s about making a difference in your family, community, nation and leaving a legacy. How do we develop a vision for our lives? It starts with prayer and seeking God’s guidance. We also need to take action and be willing to step out in faith. Lastly, we need to surround ourselves with people who support and encourage our vision. When we have a community of like-minded individuals, we can draw strength and inspiration from one another.

We all have dreams and aspirations, but it’s important to have a VISION that goes beyond your personal goals. It’s not just about setting goals, but it’s about having a purpose that’s bigger than yourself. it’s about making a difference in your family, community, nation and leaving a legacy. How do we develop a vision for our lives? It starts with prayer and seeking God’s guidance. We also need to take action and be willing to step out in faith. Lastly, we need to surround ourselves with people who support and encourage our vision. When we have a community of like-minded individuals, we can draw strength and inspiration from one another.

Pastor Tom Deuschle

19,426 görüntüleme • 3 yıl önce

Dr Fei-Fei-Li explains with a simple example how everyday household chores are so extremely difficult for Robots. "If you tell a robot to open the top drawer and watch out for the vase, this is actually a really hard task for robots." because the robot must ground language into the real world. Words like "top", "drawer", and "vase" are abstract. The system has to map them to 3D locations, objects, and relations in a noisy scene. This requires robust perception, object recognition, and spatial reasoning under uncertainty. The robot also lacks human commonsense. "Watch out" implies predicting consequences, estimating clearances, and understanding that vases are fragile. Encoding such priors, like how heavy a drawer is or how a vase might tip, is very complex and difficult without rich world knowledge. Learning the behavior from rewards is tough. The success signal is very sparse here, so naive exploration almost never stumbles on a full success sequence. This makes policy learning sample inefficient and brittle, especially when the environment changes between training and deployment. A sparse reward situation is when the agent only gets a success signal at the very end, and gets little or no feedback along the way. If a robot must open a drawer without hitting a vase, it might get reward only if the drawer ends up open and the vase is intact. Every partial try before that looks the same to the learner, reward equals 0. --- From "DSAI by Dr. Osbert Tay" YT channel

Dr Fei-Fei-Li explains with a simple example how everyday household chores are so extremely difficult for Robots. "If you tell a robot to open the top drawer and watch out for the vase, this is actually a really hard task for robots." because the robot must ground language into the real world. Words like "top", "drawer", and "vase" are abstract. The system has to map them to 3D locations, objects, and relations in a noisy scene. This requires robust perception, object recognition, and spatial reasoning under uncertainty. The robot also lacks human commonsense. "Watch out" implies predicting consequences, estimating clearances, and understanding that vases are fragile. Encoding such priors, like how heavy a drawer is or how a vase might tip, is very complex and difficult without rich world knowledge. Learning the behavior from rewards is tough. The success signal is very sparse here, so naive exploration almost never stumbles on a full success sequence. This makes policy learning sample inefficient and brittle, especially when the environment changes between training and deployment. A sparse reward situation is when the agent only gets a success signal at the very end, and gets little or no feedback along the way. If a robot must open a drawer without hitting a vase, it might get reward only if the drawer ends up open and the vase is intact. Every partial try before that looks the same to the learner, reward equals 0. --- From "DSAI by Dr. Osbert Tay" YT channel

Rohan Paul

342,553 görüntüleme • 8 ay önce

ELON: TESLA IS THE WORLD’S BIGGEST ROBOT MAKER AND AI SAFETY LEADER "Tesla is arguably the world's biggest robot maker, because we have basically semi-autonomous cars that will ultimately be fully autonomous. And we are building a humanoid robot that will be basically like the car, but with legs. I kind of held open doing that for a while because I certainly don't want to hasten the AI apocalypse. But clearly, when you look at Boston Dynamics and these humanoid robots, they're going to happen with or without Tesla. So with Tesla, I've got a lot more ability to ensure robotics safety and AI, and I'll try my best to do that." Source: ELON CLIPS

ELON: TESLA IS THE WORLD’S BIGGEST ROBOT MAKER AND AI SAFETY LEADER "Tesla is arguably the world's biggest robot maker, because we have basically semi-autonomous cars that will ultimately be fully autonomous. And we are building a humanoid robot that will be basically like the car, but with legs. I kind of held open doing that for a while because I certainly don't want to hasten the AI apocalypse. But clearly, when you look at Boston Dynamics and these humanoid robots, they're going to happen with or without Tesla. So with Tesla, I've got a lot more ability to ensure robotics safety and AI, and I'll try my best to do that." Source: ELON CLIPS

Mario Nawfal

62,954 görüntüleme • 8 ay önce

Thank you for the recognition Paul Azunre WE 3Farmate SPENT YEARS BUILDING A MACHINE THAT CAN REVOLUTIONIZE FARMING IN AFRICA AND BEYOND THROUGH ADVANCED AI and AUTOMATION THAT CAN WORK IN DIVERSE FARM ENVIRONMENTS, turning everyday tasks into seamless workflows while enabling produce growers to focus on what matters most. Beyond the mere automation of agriculture, our robot integrates several AI technologies to make farming intelligent, accurate, and productive but it seems a lot of people do not fully grasp what it takes to program a fully functioning agricultural robot, which is the first of its kind in the history of our country. It has taken us years to master the intricate and highly technical processes of teaching a robot how to see and understand its environment. Teaching a robot to spray a weed without touching a crop is not easy. It takes pixel-level precision and a staggering, painstaking number of years of complex annotation to get it right. We have failed many more times trying to get it right than we have trying to convince people that, with the right funding, we cannot just build farm machines but also have the capacity to engineer advanced technologies that can be applied across multiple industries. Come to think of it—how many industrial machines are we manufacturing locally and producing for export? With the engineering of our farm robot (FAMA), which is market-ready for deployment on farmlands, we are demonstrating that Ghanaian engineers have what it takes to transform our industry and make Africa a competitive, self-sustaining hub of innovation and manufacturing. There is no country that has truly developed in its fullest sense without a strong foundation in engineering. We are leveraging computer vision, edge computing, and real-time data processing to drive sustainable farming. A lot of people who sat on the sidelines of our history, waiting to see how young people like us would push our brand visibility, have done themselves a great disservice. Those who reposted, liked, and shared our content have helped amplify our story and have contributed to shaping the future of agriculture through AI. THANK YOU GHANA THANK YOU AFRICA GHANAIANS DID IT. AS AN AFRICAN, YOU CAN ALSO DO IT. We thank everyone who has truly supported us in bringing what we have achieved to the public. Thank you for believing in US. “The future is here.” 🇬🇭 🇬🇭 🇬🇭

MARK OFORIQUAYE

30,137 görüntüleme • 3 ay önce

Sanders: But there is another vision out there as well—a darker vision. It is a vision which says that we must give up on democracy, that we are too stupid and inept to govern ourselves, and that we must put more and more power into the hands of one man. It is a vision that says we should accept an economy in which a handful of oligarchs have unbelievable wealth while the vast majority of our people struggle to put food on the table. It is a vision that says the only thing that matters in life is the accumulation of money and power—and that it is OK if we lie, cheat, and steal to achieve those goals. It is a vision which says that we must hate each other because of where we were born, hate each other because of a different language someone speaks, hate each other because of differences in the color of our skin, or our religion, or our sexual orientation. It is a vision that foments hatred, division, and more division. It is an Orwellian vision, which says that we must live in a constant state of fear—that we must always have an enemy and that we must always be at war. It is a vision which says that we have unlimited amounts of money for bombs and guns and killing, but never enough money to feed our children, provide affordable housing, or enable our parents to retire with dignity.

Sanders: But there is another vision out there as well—a darker vision. It is a vision which says that we must give up on democracy, that we are too stupid and inept to govern ourselves, and that we must put more and more power into the hands of one man. It is a vision that says we should accept an economy in which a handful of oligarchs have unbelievable wealth while the vast majority of our people struggle to put food on the table. It is a vision that says the only thing that matters in life is the accumulation of money and power—and that it is OK if we lie, cheat, and steal to achieve those goals. It is a vision which says that we must hate each other because of where we were born, hate each other because of a different language someone speaks, hate each other because of differences in the color of our skin, or our religion, or our sexual orientation. It is a vision that foments hatred, division, and more division. It is an Orwellian vision, which says that we must live in a constant state of fear—that we must always have an enemy and that we must always be at war. It is a vision which says that we have unlimited amounts of money for bombs and guns and killing, but never enough money to feed our children, provide affordable housing, or enable our parents to retire with dignity.

Acyn

60,780 görüntüleme • 3 ay önce

We have seen a lot of legged robots doing navigation in the wild. But how about mobile manipulation in the wild? I have been pushing the direction of learning a unified, efficient, and dynamic 3D representation of scenes (for navigation) and objects (for manipulation) for the past two years. And now we have GeFF --- our large-scale, generalizable feature field, that combines the speed of a feed-forward neural network with the rich semantics from Foundation Models, to handle dynamically changing scenes, and enable open-ended, language-grounded scene and object understanding.

We have seen a lot of legged robots doing navigation in the wild. But how about mobile manipulation in the wild? I have been pushing the direction of learning a unified, efficient, and dynamic 3D representation of scenes (for navigation) and objects (for manipulation) for the past two years. And now we have GeFF --- our large-scale, generalizable feature field, that combines the speed of a feed-forward neural network with the rich semantics from Foundation Models, to handle dynamically changing scenes, and enable open-ended, language-grounded scene and object understanding.

Xiaolong Wang

42,767 görüntüleme • 2 yıl önce

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 görüntüleme • 3 yıl önce

Reinforcement on robots is highly limited by our ability to design good reward functions; this means that designing strong, generalizable reward functions is a key enabler to progress on real-world reinforcement learning. But we already have a very general class of models: VLMs. Wouldn’t it be great if you could just use a VLM to generate rewards, then? TOPReward directly generates rewards from the probability of the “True” token of a VLM question-answering response; this makes it easy to implement, incredibly general, and surprisingly powerful. We talked to Shirui and Cole Harrison to learn more. Watch Episode#75 of RoboPapers now to learn more, with Chris Paxton and Jiafei Duan!

Reinforcement on robots is highly limited by our ability to design good reward functions; this means that designing strong, generalizable reward functions is a key enabler to progress on real-world reinforcement learning. But we already have a very general class of models: VLMs. Wouldn’t it be great if you could just use a VLM to generate rewards, then? TOPReward directly generates rewards from the probability of the “True” token of a VLM question-answering response; this makes it easy to implement, incredibly general, and surprisingly powerful. We talked to Shirui and Cole Harrison to learn more. Watch Episode#75 of RoboPapers now to learn more, with Chris Paxton and Jiafei Duan!

RoboPapers

13,185 görüntüleme • 2 ay önce

Ashes of Creation Fans I'm sorry to hear the game you have been really looking forward to continuing is getting shut down. I know you'll find other games to love, and although Luminary isn't an MMO, we are co-op multiplayer and may be able to fill that void for some of you. Action RPG with real time combat, great exploration, and a community that is quickly growing. If that sounds like something you may be interested in, we'd love to have you :)

Ashes of Creation Fans I'm sorry to hear the game you have been really looking forward to continuing is getting shut down. I know you'll find other games to love, and although Luminary isn't an MMO, we are co-op multiplayer and may be able to fill that void for some of you. Action RPG with real time combat, great exploration, and a community that is quickly growing. If that sounds like something you may be interested in, we'd love to have you :)

Luminary | Refractive

146,612 görüntüleme • 5 ay önce

ELON: EVERYONE ON EARTH WILL WANT THEIR OWN ROBOT “My prediction is the robots will actually make so many robots and AI that they will actually saturate all human needs, meaning you won't be able to even think of something to ask the robot for at a certain point. There'll be more robots than people. I think everyone on Earth is going to have one and going to want one, because who wouldn't want a robot to, assuming it's very safe, watch over your kids, take care of your pet. If you have elderly parents, it's very difficult to take care of them. It's expensive, and there just aren't enough young people to take care of the old people. If you had a robot that could take care of and protect an elderly parent, I think that would be great. That would be an amazing thing to have. And I think we will have those things.” Source: Elon Musk Interview at WEF, January 2026

ELON: EVERYONE ON EARTH WILL WANT THEIR OWN ROBOT “My prediction is the robots will actually make so many robots and AI that they will actually saturate all human needs, meaning you won't be able to even think of something to ask the robot for at a certain point. There'll be more robots than people. I think everyone on Earth is going to have one and going to want one, because who wouldn't want a robot to, assuming it's very safe, watch over your kids, take care of your pet. If you have elderly parents, it's very difficult to take care of them. It's expensive, and there just aren't enough young people to take care of the old people. If you had a robot that could take care of and protect an elderly parent, I think that would be great. That would be an amazing thing to have. And I think we will have those things.” Source: Elon Musk Interview at WEF, January 2026

Mario Nawfal

47,723 görüntüleme • 5 ay önce

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI Breakfast

33,722 görüntüleme • 3 yıl önce

Important announcement (with job opportunities!): I’m thrilled to share that I just joined Lila Sciences as SVP of Open-Endedness! Lila is a new name in the AI space, but one you will be hearing a lot from. Their unique mission to pursue Scientific Superintelligence could not align better with my interest in open-ended creativity. Lila is about the entire scientific enterprise, not just a single constrained domain like drug discovery. And of course, the history of scientific progress is an unmistakable precedent for open-ended discovery and Why Greatness Cannot Be Planned. For AI itself to join this epic quest beyond just a supporting role, to create new magic that the human mind has yet to imagine, we will need to take creativity seriously well beyond the conventional pillars of more data, more compute, and more time on inference. That may be where the frontier labs currently are, but if you know about open-endedness, you know that open-ended creativity is not about taking a test and getting a good score. Lila understands the difference and is willing to invest to help me create the best open-endedness team in the world (as part of their overall AI effort) to make it happen. So I want to invite you, if you think this opportunity sounds as exciting as I do, to reach out if you’re interested in joining my team. I’m looking for a diverse range of expertise: pre-training, fine-tuning, RLHF, distillation, mechanistic interpretability, and yes - quality diversity techniques! We’re going to do things on my team that nobody else is doing. This will not be the usual roadmap. Compensation will be industry competitive and our team will be based in San Francisco with a hybrid work schedule. 1/n

Important announcement (with job opportunities!): I’m thrilled to share that I just joined Lila Sciences as SVP of Open-Endedness! Lila is a new name in the AI space, but one you will be hearing a lot from. Their unique mission to pursue Scientific Superintelligence could not align better with my interest in open-ended creativity. Lila is about the entire scientific enterprise, not just a single constrained domain like drug discovery. And of course, the history of scientific progress is an unmistakable precedent for open-ended discovery and Why Greatness Cannot Be Planned. For AI itself to join this epic quest beyond just a supporting role, to create new magic that the human mind has yet to imagine, we will need to take creativity seriously well beyond the conventional pillars of more data, more compute, and more time on inference. That may be where the frontier labs currently are, but if you know about open-endedness, you know that open-ended creativity is not about taking a test and getting a good score. Lila understands the difference and is willing to invest to help me create the best open-endedness team in the world (as part of their overall AI effort) to make it happen. So I want to invite you, if you think this opportunity sounds as exciting as I do, to reach out if you’re interested in joining my team. I’m looking for a diverse range of expertise: pre-training, fine-tuning, RLHF, distillation, mechanistic interpretability, and yes - quality diversity techniques! We’re going to do things on my team that nobody else is doing. This will not be the usual roadmap. Compensation will be industry competitive and our team will be based in San Francisco with a hybrid work schedule. 1/n

Kenneth Stanley

63,111 görüntüleme • 1 yıl önce

Robot simulation is getting a big upgrade! 🔥 Normally, building a simulation environment, a kitchen, warehouse, office takes days or weeks. You have to model the geometry, add textures, set lighting, generate collision meshes… all before a robot can even interact with it. Marble by World Labs removes most of that work. You type a text prompt or give it an image, and it generates a full 3D scene with depth, lighting, and a collider mesh you can export into engines like MuJoCo. That means researchers can quickly create lots of different environments to train robots on, which is exactly what domain randomization needs. If we want robots to learn faster, we need to create worlds faster. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robot simulation is getting a big upgrade! 🔥 Normally, building a simulation environment, a kitchen, warehouse, office takes days or weeks. You have to model the geometry, add textures, set lighting, generate collision meshes… all before a robot can even interact with it. Marble by World Labs removes most of that work. You type a text prompt or give it an image, and it generates a full 3D scene with depth, lighting, and a collider mesh you can export into engines like MuJoCo. That means researchers can quickly create lots of different environments to train robots on, which is exactly what domain randomization needs. If we want robots to learn faster, we need to create worlds faster. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

15,498 görüntüleme • 7 ay önce

Elon Musk: We can have a future where nobody's in need. “A future of abundance for all sounds impossible. But what I'm here to tell you is that it will indeed be the case. What's key to that is robotics and AI: Self-driving cars and autonomous humanoid robots that can be your own personal robot buddy [...] and enable the production of goods and services, basically with no limit. When you combine that with sustainable energy from the sun and batteries, we can at the same time also maintain a great environment. I think that's the future that we want. A future where nobody's in need, you can have what you want. But we still have the beautiful parts of nature that we like. And also space travel—let's not forget that.” Tesla All-Hands Meeting, March 20, 2025

Elon Musk: We can have a future where nobody's in need. “A future of abundance for all sounds impossible. But what I'm here to tell you is that it will indeed be the case. What's key to that is robotics and AI: Self-driving cars and autonomous humanoid robots that can be your own personal robot buddy [...] and enable the production of goods and services, basically with no limit. When you combine that with sustainable energy from the sun and batteries, we can at the same time also maintain a great environment. I think that's the future that we want. A future where nobody's in need, you can have what you want. But we still have the beautiful parts of nature that we like. And also space travel—let's not forget that.” Tesla All-Hands Meeting, March 20, 2025

ELON CLIPS

1,330,056 görüntüleme • 1 yıl önce