正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Muse Spark 1.2 supports a broad range of multimodal tasks, from turning visuals into working code to translating perception into physical action. It also brings robust audio-visual understanding to enable video-heavy workflows common in real-world enterprise use. Today, we’re sharing new evals and demos that illustrate the breadth of... show more

AI at Meta

838,715 subscribers

36,540 次观看 • 1 天前 •via X (Twitter)

科学技术教育游戏

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Muse Spark 1.1 also excels in perception and multimodal reasoning, inspecting visual and audio inputs, preserving details across long workflows, and acting on them in real execution environments. It shows particular strengths in visual-to-code generation, rich image/video captioning, and agentic computer use. In this demo, using video shot from a smartphone, Muse Spark 1.1 extracts useful photos and reasons about the product to operate a user's browser and make a Facebook Marketplace listing on the user's behalf.

Muse Spark 1.1 also excels in perception and multimodal reasoning, inspecting visual and audio inputs, preserving details across long workflows, and acting on them in real execution environments. It shows particular strengths in visual-to-code generation, rich image/video captioning, and agentic computer use. In this demo, using video shot from a smartphone, Muse Spark 1.1 extracts useful photos and reasons about the product to operate a user's browser and make a Facebook Marketplace listing on the user's behalf.

AI at Meta

73,367 次观看 • 1 个月前

Here’s an example of Muse Code’s multimodal visual coding capabilities. In this demo, the user inputs a fly-through video of a home into the terminal as an mp4 file. Muse Code interprets the video and produces a visually rich website with booking capabilities.

Here’s an example of Muse Code’s multimodal visual coding capabilities. In this demo, the user inputs a fly-through video of a home into the terminal as an mp4 file. Muse Code interprets the video and produces a visually rich website with booking capabilities.

AI at Meta

21,570 次观看 • 16 天前

Muse Spark 1.1 is here! Available today via API and on Of all the agentic use cases, I’m most excited about this one: going from a single video shot straight to task completion. 🚀 We’ve also polished and launched Computer Use and Video-Audio Perception for Muse Spark 1.1. Read more:

Muse Spark 1.1 is here! Available today via API and on Of all the agentic use cases, I’m most excited about this one: going from a single video shot straight to task completion. 🚀 We’ve also polished and launched Computer Use and Video-Audio Perception for Muse Spark 1.1. Read more:

Jiahui Yu

119,469 次观看 • 1 个月前

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Gemini 3 Pro is the best model in the world for multimodal understanding. One of its most exciting capabilities is document understanding and reasoning. This means you can convert information in any format and into the medium that works best for you. Gemini 3 also has leading multilingual capabilities, enabling it to process, reason and even capture cultural relevance across a variety of languages. For example, here Gemini 3 is translating handwritten recipes in Korean and English to build a digital family cookbook in different languages.

Google AI

36,696 次观看 • 9 个月前

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

AI at Meta

310,374 次观看 • 1 年前

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,962 次观看 • 9 个月前

Most AI demos stop at text prompts. I wanted to see what Qwen3.7-Plus could do beyond chat, so I tested it through the API using an Airbnb interface screenshot. The model analyzed the UI, identified key components, explained the visual hierarchy, highlighted usability strengths, suggested UX improvements, and even generated a React + Tailwind structure to recreate the interface. A few things that stood out: 👁️ Multimodal understanding from a single screenshot 🧠 Strong reasoning around UX and product design decisions 💻 Able to bridge visual inputs and code generation ⚡ OpenAI-compatible API that was easy to integrate 💰 Impressive capability-to-cost ratio Qwen3.7-Plus isn't just a chatbot. It's a multimodal agent model designed for workflows that combine vision, reasoning, coding, and automation. Try Qwen3.7-Plus with limited time 20% off:

Most AI demos stop at text prompts. I wanted to see what Qwen3.7-Plus could do beyond chat, so I tested it through the API using an Airbnb interface screenshot. The model analyzed the UI, identified key components, explained the visual hierarchy, highlighted usability strengths, suggested UX improvements, and even generated a React + Tailwind structure to recreate the interface. A few things that stood out: 👁️ Multimodal understanding from a single screenshot 🧠 Strong reasoning around UX and product design decisions 💻 Able to bridge visual inputs and code generation ⚡ OpenAI-compatible API that was easy to integrate 💰 Impressive capability-to-cost ratio Qwen3.7-Plus isn't just a chatbot. It's a multimodal agent model designed for workflows that combine vision, reasoning, coding, and automation. Try Qwen3.7-Plus with limited time 20% off:

Kuria | AI

17,044 次观看 • 2 个月前

My team is hiring in multimodal representation learning. We are working on tactile representations to equip robots with a physical understanding of the world. Bored of scraping disembodied internet data and looking for a new challenge? Apply here:

My team is hiring in multimodal representation learning. We are working on tactile representations to equip robots with a physical understanding of the world. Bored of scraping disembodied internet data and looking for a new challenge? Apply here:

Mustafa Mukadam

56,704 次观看 • 2 年前

Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow? Introducing 🚀MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding. Key takeaways: 🌟WebUI-trained models show major gains in visual web understanding and agent tasks. 💻 🌟Models also generalize well to non-UI tasks like DocVQA/OCR. 📄 How it works: We generate multimodal instructions with a text LLM using structured text from webpage accessibility trees. We then pair them with UI screenshots, to train multimodal models. Homepage: Paper: Dataset: Model: Congrats to the student lead Junpeng Liu and the team Tianyue Ou Yifan Song Yuxiao Qu Chenyan Xiong Wenhu Chen Graham Neubig ! More details are in the following threads ⬇️

Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow? Introducing 🚀MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding. Key takeaways: 🌟WebUI-trained models show major gains in visual web understanding and agent tasks. 💻 🌟Models also generalize well to non-UI tasks like DocVQA/OCR. 📄 How it works: We generate multimodal instructions with a text LLM using structured text from webpage accessibility trees. We then pair them with UI screenshots, to train multimodal models. Homepage: Paper: Dataset: Model: Congrats to the student lead Junpeng Liu and the team Tianyue Ou Yifan Song Yuxiao Qu Chenyan Xiong Wenhu Chen Graham Neubig ! More details are in the following threads ⬇️

Xiang Yue

57,699 次观看 • 1 年前

🆕 Anthropic's Claude Opus 4.8 is now generally available and rolling out in GitHub Copilot. Early testing shows: • It demonstrates a clear step forward in code understanding and generation across a range of real-world coding tasks. • It handles complex problem-solving and large-codebase navigation with notable improvement to previous versions. Try it out in Visual Studio Code or Copilot CLI.

🆕 Anthropic's Claude Opus 4.8 is now generally available and rolling out in GitHub Copilot. Early testing shows: • It demonstrates a clear step forward in code understanding and generation across a range of real-world coding tasks. • It handles complex problem-solving and large-codebase navigation with notable improvement to previous versions. Try it out in Visual Studio Code or Copilot CLI.

GitHub

209,313 次观看 • 2 个月前

Breaking news: Cosmos 3 is here. They are attempting to do something completely new 🤯 Why is Physical AI much harder than building a chatbot? Understanding the world is not enough, robots need to predict it and act inside it. That's the idea behind NVIDIA Cosmos 3: → Reasoning model understands what's happening from video, images, text, and actions. → World model generates future states of the environment. → Action model generates the actions needed to achieve a goal. Previous systems often stitched these capabilities together using separate models. Cosmos 3 combines them into a single architecture with two components: ▪️ Reasoner Tower Analyzes observations and builds an understanding of objects, motion, interactions, and physical context. ▪️ Generator Tower Uses that understanding to generate future videos and action sequences that obey physical constraints. So Cosmos 3 moves from: Perception → Model A Prediction → Model B Actions → Model C to: Perception + Prediction + Actions → One unified system. The goal is to make robots, autonomous vehicles, and smart environments better at answering three questions: 1. What is happening? 2. What will happen next? 3. What should I do? That's a big shift from today's AI, which mostly focuses on generating text. And check the benchmarks! Physical AI needs to generate decisions that survive contact with the real world. 🚗🤖

Breaking news: Cosmos 3 is here. They are attempting to do something completely new 🤯 Why is Physical AI much harder than building a chatbot? Understanding the world is not enough, robots need to predict it and act inside it. That's the idea behind NVIDIA Cosmos 3: → Reasoning model understands what's happening from video, images, text, and actions. → World model generates future states of the environment. → Action model generates the actions needed to achieve a goal. Previous systems often stitched these capabilities together using separate models. Cosmos 3 combines them into a single architecture with two components: ▪️ Reasoner Tower Analyzes observations and builds an understanding of objects, motion, interactions, and physical context. ▪️ Generator Tower Uses that understanding to generate future videos and action sequences that obey physical constraints. So Cosmos 3 moves from: Perception → Model A Prediction → Model B Actions → Model C to: Perception + Prediction + Actions → One unified system. The goal is to make robots, autonomous vehicles, and smart environments better at answering three questions: 1. What is happening? 2. What will happen next? 3. What should I do? That's a big shift from today's AI, which mostly focuses on generating text. And check the benchmarks! Physical AI needs to generate decisions that survive contact with the real world. 🚗🤖

Turing Post

13,836 次观看 • 2 个月前

MiniMax H3 is native in ComfyUI, day zero with the weights. Model highlights: → Text-to-video, prompt only → Image-to-video → First-and-last-frame - control the opening frame, the closing frame, or both → Reference-to-video - carry a subject, a motion, or a voice through the clip → In-place editing - modify a shot you already have instead of regenerating it The capability MiniMax (official) leads with is multimodal context understanding, and it's what collapses those five tasks into one model. H3 takes images, audio, and video together and resolves them against a prompt that explains how they relate. Download the workflows & learn more in our blog 👇

MiniMax H3 is native in ComfyUI, day zero with the weights. Model highlights: → Text-to-video, prompt only → Image-to-video → First-and-last-frame - control the opening frame, the closing frame, or both → Reference-to-video - carry a subject, a motion, or a voice through the clip → In-place editing - modify a shot you already have instead of regenerating it The capability MiniMax (official) leads with is multimodal context understanding, and it's what collapses those five tasks into one model. H3 takes images, audio, and video together and resolves them against a prompt that explains how they relate. Download the workflows & learn more in our blog 👇

ComfyUI

232,033 次观看 • 18 天前

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

Chuang Gan

19,963 次观看 • 1 个月前

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

Fei-Fei Li

25,199 次观看 • 24 天前

“ xAI team is currently working heavily on coding models. Right now, the main focus is training a specialized coding model that will be both fast and smart. I believe we’ll share it with you guys in a few weeks. That’s exciting. Second ,after coding, we all see that the main weakness of Grok 4 is its multimodal capabilities. In fact, it was so bad that Grok was effectively looking at the world while squinting through glass trying to see blurry features and make sense of them. The most immediate improvement we’re stepping on with the next generation pre trained model is huge gains in image understanding, video understanding, and audio. Right now, the model can hear and see the world just like any of you. And with all the tools and other agents it can talk to, we’re going to see a huge unlock for many different application layers once multimodal agents arrive.” — xAI Team

“ xAI team is currently working heavily on coding models. Right now, the main focus is training a specialized coding model that will be both fast and smart. I believe we’ll share it with you guys in a few weeks. That’s exciting. Second ,after coding, we all see that the main weakness of Grok 4 is its multimodal capabilities. In fact, it was so bad that Grok was effectively looking at the world while squinting through glass trying to see blurry features and make sense of them. The most immediate improvement we’re stepping on with the next generation pre trained model is huge gains in image understanding, video understanding, and audio. Right now, the model can hear and see the world just like any of you. And with all the tools and other agents it can talk to, we’re going to see a huge unlock for many different application layers once multimodal agents arrive.” — xAI Team

Apurv Kochara

88,142 次观看 • 6 个月前

Got the vibe but not the words? Starting today, you can simply show or tell AI Mode about what you're looking for and get rich, visual results to explore. ✨ This experience works by bringing together Search’s world-class visual understanding from Lens and Image search with Gemini 2.5’s advanced multimodal capabilities. When you show AI Mode what you’re searching for it uses our new “visual search fan-out” technique to get a deeper understanding of what’s in an image (including subtle details and secondary objects), and runs multiple queries in the background. This helps it understand the visual context and nuance of your request to deliver relevant visual responses. You can also tell AI Mode what vibe you’re going for (no image needed). For example, ask AI Mode for bedroom design inspo and go back and forth to refine your vision. We’re rolling out this experience in English in the U.S. starting this week. Try it out and let us know what you think!

Got the vibe but not the words? Starting today, you can simply show or tell AI Mode about what you're looking for and get rich, visual results to explore. ✨ This experience works by bringing together Search’s world-class visual understanding from Lens and Image search with Gemini 2.5’s advanced multimodal capabilities. When you show AI Mode what you’re searching for it uses our new “visual search fan-out” technique to get a deeper understanding of what’s in an image (including subtle details and secondary objects), and runs multiple queries in the background. This helps it understand the visual context and nuance of your request to deliver relevant visual responses. You can also tell AI Mode what vibe you’re going for (no image needed). For example, ask AI Mode for bedroom design inspo and go back and forth to refine your vision. We’re rolling out this experience in English in the U.S. starting this week. Try it out and let us know what you think!

Google AI

68,285 次观看 • 10 个月前

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,440 次观看 • 2 年前

New generative UI experiences in the Google Gemini and AI Mode in Search use Gemini 3’s multimodal understanding capabilities to make entire user interfaces for you from scratch — dynamically delivering what you need in a visual and interactive way. Google Research software engineer Yaniv Leviathan shares more about the process of building generative UI (hint: so much fun that the team couldn't stop playing with it) and how he's been using it so far.

New generative UI experiences in the Google Gemini and AI Mode in Search use Gemini 3’s multimodal understanding capabilities to make entire user interfaces for you from scratch — dynamically delivering what you need in a visual and interactive way. Google Research software engineer Yaniv Leviathan shares more about the process of building generative UI (hint: so much fun that the team couldn't stop playing with it) and how he's been using it so far.

Google

138,435 次观看 • 9 个月前

Today, Box is announcing major new AI agent capabilities to let customers tap into the full value of their unstructured data. First, we’re announcing all new updates to the Box AI Studio to make it even easier to build AI agents that tap into your enterprise content for any job function, business process, or industry specific use case. We are also expanding our set of foundational agents that customers will be able to use to work with their enterprise content, including new features like search and research on unstructured data. Next, we’re announcing Box Extract to enable customers to use AI agents seamlessly for complex data extraction from any type of document or content. This makes it easier than ever to pull out data from contracts, invoices, research data, marketing assets, medical charts, and more. Finally, we’re introducing Box Automate, a new workflow automation solution within Box that lets you deploy AI agents across enterprise content-centric workflows. With Box Automate, you can design your business process in a simple drag and drop builder and then drop in AI agents at any step in the process. This ensures agents execute tasks at the right steps in a workflow every time. Best of all, our AI agents and workflow tools are designed to work across any system our customers work within, whether it’s leveraging pre-built integrations, Box APIs, or the new Box MCP Server. Ultimately, all of these capabilities come together to transform how companies can work with their enterprise content. Software has historically only been good at automating work that deals with structured data, which is why ERP, CRM, and HR systems have been mainstays of enterprise software for so long. The data in these systems fits neatly into a database, and the workflows are very ripe for automation. But it turns out most of the work in the world deals with unstructured data. It’s ideating through research documents, working with a client on contracts, reviewing details for a new product launch, looking at a patient’s healthcare record to make a diagnosis, working through due diligence documents for an M&A deal, and so on. For the first time ever, we can begin to bring all new insights and automation to this work with AI agents. At Box, we’re incredibly excited to be on this journey to help customers transform how they work with their most important data.

Today, Box is announcing major new AI agent capabilities to let customers tap into the full value of their unstructured data. First, we’re announcing all new updates to the Box AI Studio to make it even easier to build AI agents that tap into your enterprise content for any job function, business process, or industry specific use case. We are also expanding our set of foundational agents that customers will be able to use to work with their enterprise content, including new features like search and research on unstructured data. Next, we’re announcing Box Extract to enable customers to use AI agents seamlessly for complex data extraction from any type of document or content. This makes it easier than ever to pull out data from contracts, invoices, research data, marketing assets, medical charts, and more. Finally, we’re introducing Box Automate, a new workflow automation solution within Box that lets you deploy AI agents across enterprise content-centric workflows. With Box Automate, you can design your business process in a simple drag and drop builder and then drop in AI agents at any step in the process. This ensures agents execute tasks at the right steps in a workflow every time. Best of all, our AI agents and workflow tools are designed to work across any system our customers work within, whether it’s leveraging pre-built integrations, Box APIs, or the new Box MCP Server. Ultimately, all of these capabilities come together to transform how companies can work with their enterprise content. Software has historically only been good at automating work that deals with structured data, which is why ERP, CRM, and HR systems have been mainstays of enterprise software for so long. The data in these systems fits neatly into a database, and the workflows are very ripe for automation. But it turns out most of the work in the world deals with unstructured data. It’s ideating through research documents, working with a client on contracts, reviewing details for a new product launch, looking at a patient’s healthcare record to make a diagnosis, working through due diligence documents for an M&A deal, and so on. For the first time ever, we can begin to bring all new insights and automation to this work with AI agents. At Box, we’re incredibly excited to be on this journey to help customers transform how they work with their most important data.

Aaron Levie

91,863 次观看 • 11 个月前