Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

How can agents understand the world from diverse language? 🌎 Excited to introduce Dynalang, an agent that learns to understand language by 𝙢𝙖𝙠𝙞𝙣𝙜 𝙥𝙧𝙚𝙙𝙞𝙘𝙩𝙞𝙤𝙣𝙨 𝙖𝙗𝙤𝙪𝙩 𝙩𝙝𝙚 𝙛𝙪𝙩𝙪𝙧𝙚 with a multimodal world model!

Jessy Lin

5,278 subscribers

107,491 views • 3 years ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

π0.7 handles diverse prompts that don't just say what to do, but also how to do it, including rich language and multimodal information, such as visual subgoal images. At test time, these images can be produced by a lightweight world model.

π0.7 handles diverse prompts that don't just say what to do, but also how to do it, including rich language and multimodal information, such as visual subgoal images. At test time, these images can be produced by a lightweight world model.

Physical Intelligence

33,748 views • 3 months ago

LARP: Language-Agent Role Play for Open-World Games paper page: Language agents have shown impressive problem-solving skills within defined settings and brief timelines. Yet, with the ever-evolving complexities of open-world simulations, there's a pressing need for agents that can flexibly adapt to complex environments and consistently maintain a long-term memory to ensure coherent actions. To bridge the gap between language agents and open-world games, we introduce Language Agent for Role-Playing (LARP), which includes a cognitive architecture that encompasses memory processing and a decision-making assistant, an environment interaction module with a feedback-driven learnable action space, and a postprocessing method that promotes the alignment of various personalities. The LARP framework refines interactions between users and agents, predefined with unique backgrounds and personalities, ultimately enhancing the gaming experience in open-world contexts. Furthermore, it highlights the diverse uses of language models in a range of areas such as entertainment, education, and various simulation scenarios.

LARP: Language-Agent Role Play for Open-World Games paper page: Language agents have shown impressive problem-solving skills within defined settings and brief timelines. Yet, with the ever-evolving complexities of open-world simulations, there's a pressing need for agents that can flexibly adapt to complex environments and consistently maintain a long-term memory to ensure coherent actions. To bridge the gap between language agents and open-world games, we introduce Language Agent for Role-Playing (LARP), which includes a cognitive architecture that encompasses memory processing and a decision-making assistant, an environment interaction module with a feedback-driven learnable action space, and a postprocessing method that promotes the alignment of various personalities. The LARP framework refines interactions between users and agents, predefined with unique backgrounds and personalities, ultimately enhancing the gaming experience in open-world contexts. Furthermore, it highlights the diverse uses of language models in a range of areas such as entertainment, education, and various simulation scenarios.

AK

143,974 views • 2 years ago

Excited to introduce SketchAgent!👩‍🎨 We leverage the prior of pretrained multimodal LLMs for language-driven, sequential sketch generation and human-agent collaborative sketching! ✨ Try our fun interface here:

Excited to introduce SketchAgent!👩‍🎨 We leverage the prior of pretrained multimodal LLMs for language-driven, sequential sketch generation and human-agent collaborative sketching! ✨ Try our fun interface here:

Yael Vinker

31,956 views • 1 year ago

We are excited to announce the 1st version of our multimodal assistant, Yasa-1, a language assistant with visual and auditory sensors that can take actions via code execution 🪄. Yasa-1 can understand text, images, videos, sounds & more! 🚀 Check out more details below👇

We are excited to announce the 1st version of our multimodal assistant, Yasa-1, a language assistant with visual and auditory sensors that can take actions via code execution 🪄. Yasa-1 can understand text, images, videos, sounds & more! 🚀 Check out more details below👇

Reka

814,303 views • 2 years ago

NEWS: Nvidia has just introduced "Cosmos," a world foundation model created to understand the physical world. The model can generate synthetic data to train robotics.

NEWS: Nvidia has just introduced "Cosmos," a world foundation model created to understand the physical world. The model can generate synthetic data to train robotics.

Sawyer Merritt

254,296 views • 1 year ago

Eric Schmidt: the point at which AI agents can talk to each other in a language we can't understand, we should unplug the computers

Eric Schmidt: the point at which AI agents can talk to each other in a language we can't understand, we should unplug the computers

Tsarathustra

154,517 views • 2 years ago

🚀 We introduce Neural Theorizer (NEO) — a new type of world model that learns to theorize the world from observation, without language or LLM supervision. Selected as an ICML 2026 oral presentation — 0.7% of submitted papers. The paper asks: "What does it mean to understand the world and build a world model?" Today’s world models are often trained to predict the future: the next frame, next latent state, or next observation. But is prediction enough? We argue that a world model should be a theory-building system: one that discovers reusable primitives, composes them into executable explanations, and transfers those explanations to novel phenomena. NEO is our first step toward this vision — a World Theory Model that learns explicit, compositional theories from raw observation. This work was led by my wonderful students: Doojin Baek*(Doojin Baek), Gyubin Lee* (GyuBin Lee), Junyeob Baek (Junyeob Baek), and Hosung Lee (Hosung Lee). For more details, take a look at the paper — and if you’re attending ICML, let’s talk there! 📄 arXiv: 🌐 Project page:

🚀 We introduce Neural Theorizer (NEO) — a new type of world model that learns to theorize the world from observation, without language or LLM supervision. Selected as an ICML 2026 oral presentation — 0.7% of submitted papers. The paper asks: "What does it mean to understand the world and build a world model?" Today’s world models are often trained to predict the future: the next frame, next latent state, or next observation. But is prediction enough? We argue that a world model should be a theory-building system: one that discovers reusable primitives, composes them into executable explanations, and transfers those explanations to novel phenomena. NEO is our first step toward this vision — a World Theory Model that learns explicit, compositional theories from raw observation. This work was led by my wonderful students: Doojin Baek(Doojin Baek), Gyubin Lee (GyuBin Lee), Junyeob Baek (Junyeob Baek), and Hosung Lee (Hosung Lee). For more details, take a look at the paper — and if you’re attending ICML, let’s talk there! 📄 arXiv: 🌐 Project page:

Sungjin Ahn

98,596 views • 1 month ago

Openclaw / Agents can now command ANY robot to grasp objects via language Integrate dimensional with your personal agent to unlock the physical world Fully open source. Join the hundreds of developers already vibecoding atoms

Openclaw / Agents can now command ANY robot to grasp objects via language Integrate dimensional with your personal agent to unlock the physical world Fully open source. Join the hundreds of developers already vibecoding atoms

stash

19,677 views • 4 months ago

Embeddings (and how to create them) are, perhaps, the most interesting idea behind Large Language Models. I built a simple model to help you understand embeddings from scratch. Here is a step-by-step video explanation:

Embeddings (and how to create them) are, perhaps, the most interesting idea behind Large Language Models. I built a simple model to help you understand embeddings from scratch. Here is a step-by-step video explanation:

Santiago

44,107 views • 1 year ago

The only way for Americans to understand the true impact of Trump tariffs is to speak to them in a language they can understand 👇

The only way for Americans to understand the true impact of Trump tariffs is to speak to them in a language they can understand 👇

Cyrus Janssen

117,507 views • 1 year ago

Perceptron’s platform is here — built for Physical AI Developers can now use Isaac-0.1 or Qwen3VL 235B via: Perceptron API — fast, reliable multimodal intelligence Python SDK — simple, grounded prompting for vision + language Build apps that see and understand the world.

Perceptron’s platform is here — built for Physical AI Developers can now use Isaac-0.1 or Qwen3VL 235B via: Perceptron API — fast, reliable multimodal intelligence Python SDK — simple, grounded prompting for vision + language Build apps that see and understand the world.

Perceptron AI

36,095 views • 9 months ago

If you understand that the common language of the world is humanity, you will understand the importance of this message. The zio regime is still butchering Palestinians. Please go to part 2 #humanity #truth

If you understand that the common language of the world is humanity, you will understand the importance of this message. The zio regime is still butchering Palestinians. Please go to part 2 #humanity #truth

Nicole Jenes

33,134 views • 1 year ago

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,440 views • 2 years ago

1- I want to introduce you to Palestinian heritage 🧵 This song is from the resistance heritage. If you don't understand the song, it's not because you don't know Arabic. Arabs don't understand it either because it's a coded language that Palestinians invented.

1- I want to introduce you to Palestinian heritage 🧵 This song is from the resistance heritage. If you don't understand the song, it's not because you don't know Arabic. Arabs don't understand it either because it's a coded language that Palestinians invented.

Muhammad ( Taqi al-Din )

121,681 views • 2 years ago

I built a Multimodal AI Agent using Gemini 2.0 that can understand videos and search the internet at the same time. 100% Opensource Code with step-by-step tutorial.

Sensitive content

I built a Multimodal AI Agent using Gemini 2.0 that can understand videos and search the internet at the same time. 100% Opensource Code with step-by-step tutorial.

Shubham Saboo

97,816 views • 1 year ago

The only way world can survive this community .. It's to follow China 🇨🇳 way .. Sooner the world 🌎 can understand this .. Better it would be ..

The only way world can survive this community .. It's to follow China 🇨🇳 way .. Sooner the world 🌎 can understand this .. Better it would be ..

AstroCounselKK 🇮🇳

20,974 views • 7 months ago

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

AK

85,570 views • 2 years ago

Warren Buffett: "You have to understand accounting. It's the language [of business]. It's like being in a foreign country without knowing the language if you're in business and you don't understand accounting." "You want to get as comfortable with [accounting] as you are with the English language."

Warren Buffett: "You have to understand accounting. It's the language [of business]. It's like being in a foreign country without knowing the language if you're in business and you don't understand accounting." "You want to get as comfortable with [accounting] as you are with the English language."

Kevin Carpenter

36,031 views • 1 month ago

Four Maps to Understand Iran's Kurdish Provinces Here's how four layers of data, from geography to religion to language, explain the region:

Four Maps to Understand Iran's Kurdish Provinces Here's how four layers of data, from geography to religion to language, explain the region:

The National Context

45,597 views • 6 months ago