正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

My team is hiring in multimodal representation learning. We are working on tactile representations to equip robots with a physical understanding of the world. Bored of scraping disembodied internet data and looking for a new challenge? Apply here:

Mustafa Mukadam

2,056 subscribers

56,704 次观看 • 2 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

3 条评论

Mustafa Mukadam 的头像

Mustafa Mukadam2 年前

Some recent works from my team along these lines - Tactile to Depth: - Tactile Diffusion: - Tactile-VLM: - Tactile Seq-to-Seq SSM:

Alexander Soare 的头像

Alexander Soare2 年前

I'll apply if PhD is not a hard requirement. LI for professional creds GH for whatever personal projects I can fit in

Ram Koppu 的头像

Ram Koppu2 年前

@chris_j_paxton Are there any end to end open source foundation models for Robotics?

相关视频

My team at FAIR AI at Meta is hiring postdocs. We are working in the space of dexterous manipulation and tactile sensing. If you are interested reach out / apply:

My team at FAIR AI at Meta is hiring postdocs. We are working in the space of dexterous manipulation and tactile sensing. If you are interested reach out / apply:

Mustafa Mukadam

40,587 次观看 • 2 年前

Does GPT understand the world? Here is what Ilya Sutskever, co-founder of OpenAI, says during a discussion with Jensen Huang, CEO of Nvidia: (1) When we train a large neural network to accurately predict the next word in lots of different texts from the internet, the AI is learning a world model. (2) On the surface, it may look like learning correlations in text, but it turns out that to 'just learn' statistical correlations in text, to compress information really well, what the neural network learns is some representation of the process that produced the text. (3) This text is a projection of the world...what the neural network is learning is aspects of the world, of people, of the human conditions, their hopes, dreams, motivations, their interactions...the situations we are in. The neural network learns a compressed, abstract, usable representation." Do you think learning representations = understanding? Are large language models simply stochastic parrots, or are they much more?

Does GPT understand the world? Here is what Ilya Sutskever, co-founder of OpenAI, says during a discussion with Jensen Huang, CEO of Nvidia: (1) When we train a large neural network to accurately predict the next word in lots of different texts from the internet, the AI is learning a world model. (2) On the surface, it may look like learning correlations in text, but it turns out that to 'just learn' statistical correlations in text, to compress information really well, what the neural network learns is some representation of the process that produced the text. (3) This text is a projection of the world...what the neural network is learning is aspects of the world, of people, of the human conditions, their hopes, dreams, motivations, their interactions...the situations we are in. The neural network learns a compressed, abstract, usable representation." Do you think learning representations = understanding? Are large language models simply stochastic parrots, or are they much more?

Alex Ker 🔭

1,367,117 次观看 • 2 年前

🤖ROBOTS ARE GETTING SMARTER AT TOUCHING THE REAL WORLD Researchers from UC Berkeley, NVIDIA and Stanford introduced T-Rex, a framework that combines vision, language and tactile sensing. Instead of relying on cameras alone, robots can now respond to physical contact in real time. Robots are no longer just seeing objects. They’re learning how to FEEL them.

🤖ROBOTS ARE GETTING SMARTER AT TOUCHING THE REAL WORLD Researchers from UC Berkeley, NVIDIA and Stanford introduced T-Rex, a framework that combines vision, language and tactile sensing. Instead of relying on cameras alone, robots can now respond to physical contact in real time. Robots are no longer just seeing objects. They’re learning how to FEEL them.

Coin Bureau

28,275 次观看 • 2 个月前

Language models process data as one-dimensional sequences of tokens — ideal for text, but limited for understanding the physical world. In contrast, spatial intelligence embraces a 3D representation, unlocking a deeper understanding of space and time. More from martin_casado's interview with Fei-Fei Li and Justin Johnson:

Language models process data as one-dimensional sequences of tokens — ideal for text, but limited for understanding the physical world. In contrast, spatial intelligence embraces a 3D representation, unlocking a deeper understanding of space and time. More from martin_casado's interview with Fei-Fei Li and Justin Johnson:

a16z

64,662 次观看 • 1 年前

We are delighted to announce the publication of the Physical Literacy Consensus Statement for England! 📢 It's been developed to facilitate a shared understanding of physical literacy for those working in the sport, education, physical activity, health and youth sectors.

We are delighted to announce the publication of the Physical Literacy Consensus Statement for England! 📢 It's been developed to facilitate a shared understanding of physical literacy for those working in the sport, education, physical activity, health and youth sectors.

Sport England

122,931 次观看 • 2 年前

Bernt Børnich says teleoperation data and synthetic data are like a crutch to bootstrap your way to robots learning in the real world.

Bernt Børnich says teleoperation data and synthetic data are like a crutch to bootstrap your way to robots learning in the real world.

The Humanoid Hub

27,977 次观看 • 1 年前

.travis kalanick just raised $1.7B for atoms, led by a16z - and now he's hiring hundreds of people. after 8 years in stealth, Travis is "back" and looking for builders who want to solve hard problems and digitize the physical world. here's a glimpse into our podcast interview with him, where he talks about what he's looking for in new team-members... and a list of roles that are open right now! link in thread.

.travis kalanick just raised $1.7B for atoms, led by a16z - and now he's hiring hundreds of people. after 8 years in stealth, Travis is "back" and looking for builders who want to solve hard problems and digitize the physical world. here's a glimpse into our podcast interview with him, where he talks about what he's looking for in new team-members... and a list of roles that are open right now! link in thread.

Katie Kirsch

43,038 次观看 • 1 个月前

Introducing ESM Cambrian. Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world. We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

Introducing ESM Cambrian. Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world. We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

Alex Rives

207,283 次观看 • 1 年前

Keith Pelley on what he is looking for with the possibility of hiring both a president and GM for the Leafs. "They have to be data centric."

Keith Pelley on what he is looking for with the possibility of hiring both a president and GM for the Leafs. "They have to be data centric."

TSN

37,543 次观看 • 4 个月前

Watch our co-founder, , explain how the future of machine learning will lead us to a continuously updated digital representation of the world on the internet, and why this requires shifts in the way we access and allocate computational resources.

Watch our co-founder, , explain how the future of machine learning will lead us to a continuously updated digital representation of the world on the internet, and why this requires shifts in the way we access and allocate computational resources.

gensyn

29,885 次观看 • 1 年前

Introducing a new look, for a new world. At Verse, we’re on a mission to redefine digital environments. We aim to bridge the gap between the physical and digital worlds by providing our users with an experience that closely mimics real-life interactions. This is a world built for players, brands and communities to thrive in a new era of experiential gameplay and engagement. Welcome to the Internet of Reality. Welcome to Verse World.

Introducing a new look, for a new world. At Verse, we’re on a mission to redefine digital environments. We aim to bridge the gap between the physical and digital worlds by providing our users with an experience that closely mimics real-life interactions. This is a world built for players, brands and communities to thrive in a new era of experiential gameplay and engagement. Welcome to the Internet of Reality. Welcome to Verse World.

Verse World

64,571 次观看 • 1 年前

Haven't uploaded an updated Showreel in a while, so here we go! Still on the lookout for a new challenge and new team to join, so if you're looking for a gameplay animator with plenty of experience digging around in blueprints, state machines and sequencer, please reach out! 🌟

Haven't uploaded an updated Showreel in a while, so here we go! Still on the lookout for a new challenge and new team to join, so if you're looking for a gameplay animator with plenty of experience digging around in blueprints, state machines and sequencer, please reach out! 🌟

🌟 Lukas Lenniger 🌟

13,009 次观看 • 1 年前

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

stash

228,452 次观看 • 4 个月前

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

75,232 次观看 • 1 个月前

Muse Spark 1.2 supports a broad range of multimodal tasks, from turning visuals into working code to translating perception into physical action. It also brings robust audio-visual understanding to enable video-heavy workflows common in real-world enterprise use. Today, we’re sharing new evals and demos that illustrate the breadth of the model’s visual understanding and reasoning capabilities. Let’s start with a demo that shows how Muse Spark parses multimodal observations and calls tools to guide a robot to navigate in an unstructured environment to find a rubber duck. 🧵👇

Muse Spark 1.2 supports a broad range of multimodal tasks, from turning visuals into working code to translating perception into physical action. It also brings robust audio-visual understanding to enable video-heavy workflows common in real-world enterprise use. Today, we’re sharing new evals and demos that illustrate the breadth of the model’s visual understanding and reasoning capabilities. Let’s start with a demo that shows how Muse Spark parses multimodal observations and calls tools to guide a robot to navigate in an unstructured environment to find a rubber duck. 🧵👇

AI at Meta

48,320 次观看 • 7 天前

Over 6,000 STEAM Labs are being established in government schools across Punjab, empowering students with hands-on learning in robotics, coding, and data analytics. Here are real moments from Govt. High School for Girls, Jallo Morh, where young minds are building robots and exploring the world of tech. Punjab is the trailblazer.

Over 6,000 STEAM Labs are being established in government schools across Punjab, empowering students with hands-on learning in robotics, coding, and data analytics. Here are real moments from Govt. High School for Girls, Jallo Morh, where young minds are building robots and exploring the world of tech. Punjab is the trailblazer.

Maryam Nawaz Sharif

25,235 次观看 • 11 个月前

🚨 Is the timeline for physical AGI shrinking? Speaking on the The All-In Podcast with @jason 1X CEO Bernt Børnich dropped a massive prediction: we are less than a decade away from a physical AI "hard takeoff." His personal bet? Just 3 years. What his 3-year hard takeoff actually looks like: ▪️ Robots manufacturing robots ▪️ Autonomous chip fabs & data centers ▪️ Self-sufficient mining & refining ▪️ A true abundance of physical labor 1X's strategy to get there relies on an open ecosystem and training the NEO humanoid on internet-scale video.

🚨 Is the timeline for physical AGI shrinking? Speaking on the The All-In Podcast with @jason 1X CEO Bernt Børnich dropped a massive prediction: we are less than a decade away from a physical AI "hard takeoff." His personal bet? Just 3 years. What his 3-year hard takeoff actually looks like: ▪️ Robots manufacturing robots ▪️ Autonomous chip fabs & data centers ▪️ Self-sufficient mining & refining ▪️ A true abundance of physical labor 1X's strategy to get there relies on an open ecosystem and training the NEO humanoid on internet-scale video.

Humanoids daily

19,106 次观看 • 28 天前

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.

Neuracore

40,796 次观看 • 9 个月前

Experiments in progress. The one on the right has been learning for ~3 hours, the one in the middle for ~1 hour, and the one on the left just started a few minutes ago. The initial motivation for making the physical Atari was just to commit ourselves to a subset of algorithms that can make progress in this setup. This commitment rules out algorithms that require billions of samples to learn (or worse, require multiple environments running in parallel). Atari games are simple enough that we should be able to show learning on them in a short amount of time with no prior knowledge. Since then, I've realized that this setup is also a good way to compare different paradigms in robotics in a principled way. These paradigms are sim2real, learning from tele-operated data, and learning directly on the robots. So far, I have observed that getting sim2real to work reliably is hard. It requires tweaks that don't scale. Policies that can play perfectly in simulation fall apart because of latencies and the messiness of the real world. These aspects could be modeled to improve the simulation, but not without sinking significant human engineering hours. I have higher hopes for learning from tele-operated data, but that requires a human to learn the task first. These experiments are on my to-do list. I have to learn to play some of the games well through the robot. I’m half-decent at playing Pong and Ms Pacman now. Learning directly on robots is looking like the most promising approach. This approach takes away pesky distribution shifts and makes it possible to have algorithms that continually improve with more data and time without any human intervention. It feels great to let experiments run overnight and wake up to find improved policies. With learning on robots, I should, in principle, be able to go on a long vacation and come back to find better policies for complex tasks beyond Atari games. Whether that is possible with current learning algorithms is a different question.

Experiments in progress. The one on the right has been learning for ~3 hours, the one in the middle for ~1 hour, and the one on the left just started a few minutes ago. The initial motivation for making the physical Atari was just to commit ourselves to a subset of algorithms that can make progress in this setup. This commitment rules out algorithms that require billions of samples to learn (or worse, require multiple environments running in parallel). Atari games are simple enough that we should be able to show learning on them in a short amount of time with no prior knowledge. Since then, I've realized that this setup is also a good way to compare different paradigms in robotics in a principled way. These paradigms are sim2real, learning from tele-operated data, and learning directly on the robots. So far, I have observed that getting sim2real to work reliably is hard. It requires tweaks that don't scale. Policies that can play perfectly in simulation fall apart because of latencies and the messiness of the real world. These aspects could be modeled to improve the simulation, but not without sinking significant human engineering hours. I have higher hopes for learning from tele-operated data, but that requires a human to learn the task first. These experiments are on my to-do list. I have to learn to play some of the games well through the robot. I’m half-decent at playing Pong and Ms Pacman now. Learning directly on robots is looking like the most promising approach. This approach takes away pesky distribution shifts and makes it possible to have algorithms that continually improve with more data and time without any human intervention. It feels great to let experiments run overnight and wake up to find improved policies. With learning on robots, I should, in principle, be able to go on a long vacation and come back to find better policies for complex tasks beyond Atari games. Whether that is possible with current learning algorithms is a different question.

Khurram Javed

52,110 次观看 • 8 个月前