Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

AI only has pre-schooler level capabilities for visual tasks. "The frontier models—the best of them—still reason around the age of a preschooler, and any elementary school kid in that benchmark was able to beat all the frontier models on these visual tasks." "These are tasks that are not just... show more

The Information

1,588,972 subscribers

26,250 views • 3 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 Comments

Vanar3 months ago

The gap isn’t seeing the world. It’s consistently reasoning and acting within it under uncertainty.

Yao888pi Shen3 months ago

You can't even define text logic, yet you expect images to teach AI physics? AI is just text vectors + probability. Without clear evaluation functions, escaping to an undefined domain isn't 'progress'—it's pure waste. VOID"

Rupert Davies3 months ago

A child builds mental models of space. An LLM predicts tokens. These are not the same thing.

Vlady3 months ago

funny that an actual elementary schooler beats every frontier model on those visual tasks, benchmarks don't lie.

anjali kushwaha3 months ago

T

เอ็ม 🤖 | AI & Cats3 months ago

AI progress isn't just about scaling models. Understanding where they still struggle is just as important.

Related Videos

Open-source models are already taking the easier tasks. The bull case for frontier labs rests on one question. "A lot of software engineering, white-collar work in general, does not need Fable 5.1 or Astra 6 level intelligence." "A lot of businesses, especially lower gross margin businesses, are very rationally making the decision to start offloading some of the easier tasks to these increasingly capable open-source models." "So whether the frontier labs are a durable business model really comes down to your belief on whether the TAM of new things you can do with increasingly smarter frontier intelligence will outgrow the portion of tasks being offloaded to these cheaper models." "If you were to magically spawn 100 million superintelligent PhDs who are experts in every field and don't need to eat or sleep, do you think the economy would be able to absorb that relatively quickly and still get high ROI?"

Open-source models are already taking the easier tasks. The bull case for frontier labs rests on one question. "A lot of software engineering, white-collar work in general, does not need Fable 5.1 or Astra 6 level intelligence." "A lot of businesses, especially lower gross margin businesses, are very rationally making the decision to start offloading some of the easier tasks to these increasingly capable open-source models." "So whether the frontier labs are a durable business model really comes down to your belief on whether the TAM of new things you can do with increasingly smarter frontier intelligence will outgrow the portion of tasks being offloaded to these cheaper models." "If you were to magically spawn 100 million superintelligent PhDs who are experts in every field and don't need to eat or sleep, do you think the economy would be able to absorb that relatively quickly and still get high ROI?"

SemiAnalysis

39,409 views • 19 days ago

Reasoning models tailored for diverse AI tasks. 🚀 Meet Amazon Nova 2 foundation models, supporting fast, cost-effective reasoning to multimodal capabilities. Power versatile tasks like AI agents, code-generation, and Conversational AI. Choose the perfect match for your workload.

Reasoning models tailored for diverse AI tasks. 🚀 Meet Amazon Nova 2 foundation models, supporting fast, cost-effective reasoning to multimodal capabilities. Power versatile tasks like AI agents, code-generation, and Conversational AI. Choose the perfect match for your workload.

Amazon Web Services

2,074,736 views • 10 months ago

BREAKING: Only ~1% of Bending Spoons’ AI requests & tokens goes to frontier models. 99% runs on self-hosted open-weight models. CEO Luca Ferrari on the shift to Sovereign AI: “We end up using open-weight models we self-host, so these are basically free. I mean, there is a little bit of cloud cost, but for 99% of the requests and tokens.” “And the frontier models, generally closed weight, through APIs for maybe 1% of the requests, only for the most complex tasks.” “Sometimes we use them automatically to check the work that got done by the slightly less intelligent models, as a more senior engineer would with a more junior engineer.”

BREAKING: Only ~1% of Bending Spoons’ AI requests & tokens goes to frontier models. 99% runs on self-hosted open-weight models. CEO Luca Ferrari on the shift to Sovereign AI: “We end up using open-weight models we self-host, so these are basically free. I mean, there is a little bit of cloud cost, but for 99% of the requests and tokens.” “And the frontier models, generally closed weight, through APIs for maybe 1% of the requests, only for the most complex tasks.” “Sometimes we use them automatically to check the work that got done by the slightly less intelligent models, as a more senior engineer would with a more junior engineer.”

Molly O’Shea

243,526 views • 1 month ago

Open models are closing the intelligence gap, says ollama's Jeffrey Morgan. That opens up a new opportunity for founders: coordinating fast, cheap models to solve more complex problems. “We’re maybe less than three months behind between the frontier closed models and the open models. But the next problem to solve is extreme efficiency.” “They’re good enough for 80% of the tasks, they’re really fast and they’re ultra cheap.” “That class of models, in my mind, will be the first ones that come down to this idea of the unlimited tokens.” “Being able to coordinate these flash models together to do different tasks can also yield great results that a bigger model can.”

Open models are closing the intelligence gap, says ollama's Jeffrey Morgan. That opens up a new opportunity for founders: coordinating fast, cheap models to solve more complex problems. “We’re maybe less than three months behind between the frontier closed models and the open models. But the next problem to solve is extreme efficiency.” “They’re good enough for 80% of the tasks, they’re really fast and they’re ultra cheap.” “That class of models, in my mind, will be the first ones that come down to this idea of the unlimited tokens.” “Being able to coordinate these flash models together to do different tasks can also yield great results that a bigger model can.”

Y Combinator

58,362 views • 1 month ago

Axis Robotics Point System - Free Airdrop Guide: Axis Robotics {Robotics Sector} Currently, I think this is the one of the free airdrop you should focus on because: - Raised $12M (led by Hack VC) - Building on AI + Robotics (robotics could be the next hot sector after AI) - Cost free tasks (only requires effort) - Low competition due to the difficult tasks + negative sentiment around airdrops STEPS 👇🏻 🔗 • Create an account • Complete all the available tasks. There are many live tasks to complete. It might feel hard when you start but once you adapt, it gets easier. • The more tasks you complete, the more points you'll earn • Sign all the tasks you complete (you can use bulk signing) Point distribution: Based on: • Volume of tasks • Difficulty level • Variety of tasks completed Points are distributed every 2 weeks (MONDAY) Snapshot taken on Friday. Special tasks: There are some special tasks that give higher points, but not everyone can access them (requires an access code). Will update if I get any. For special tasks, click on Alliance and complete the tasks there. Still very early, as the point system went live only a few days ago. You can contribute now and try to climb the leaderboard. Added a video showing how the tasks are completed by controlling the robots👇🏻

Axis Robotics Point System - Free Airdrop Guide: Axis Robotics {Robotics Sector} Currently, I think this is the one of the free airdrop you should focus on because: - Raised $12M (led by Hack VC) - Building on AI + Robotics (robotics could be the next hot sector after AI) - Cost free tasks (only requires effort) - Low competition due to the difficult tasks + negative sentiment around airdrops STEPS 👇🏻 🔗 • Create an account • Complete all the available tasks. There are many live tasks to complete. It might feel hard when you start but once you adapt, it gets easier. • The more tasks you complete, the more points you'll earn • Sign all the tasks you complete (you can use bulk signing) Point distribution: Based on: • Volume of tasks • Difficulty level • Variety of tasks completed Points are distributed every 2 weeks (MONDAY) Snapshot taken on Friday. Special tasks: There are some special tasks that give higher points, but not everyone can access them (requires an access code). Will update if I get any. For special tasks, click on Alliance and complete the tasks there. Still very early, as the point system went live only a few days ago. You can contribute now and try to climb the leaderboard. Added a video showing how the tasks are completed by controlling the robots👇🏻

Abhi.eth

20,968 views • 2 months ago

New AI models can now perform many of the tasks that once defined entry-level jobs. In another year, imagine where we are at.

New AI models can now perform many of the tasks that once defined entry-level jobs. In another year, imagine where we are at.

Financial Dystopia

22,716 views • 7 months ago

What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.

What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.

Epoch AI

93,515 views • 4 months ago

The first real evidence that the days of LLM Scaling laws are over. Introducing SCOPE: the world's most efficient Neural Planner. 🔍📊 We tested SCOPE vs Frontier LLMs for planning tasks on TextCraft (text version of Minecraft) and here are the results: ⤵️ - SCOPE Runs 55x faster than GPT 3.5 (3 seconds vs 164 seconds) - SCOPE is 160,000 smaller than GPT 4o (11M parameters vs 1.8T parameters) - SCOPE is more accurate on Planning tasks (56%) than frontier LLM models The age of efficient AI models starts now. 🔗📌 Read the full write up here:

The first real evidence that the days of LLM Scaling laws are over. Introducing SCOPE: the world's most efficient Neural Planner. 🔍📊 We tested SCOPE vs Frontier LLMs for planning tasks on TextCraft (text version of Minecraft) and here are the results: ⤵️ - SCOPE Runs 55x faster than GPT 3.5 (3 seconds vs 164 seconds) - SCOPE is 160,000 smaller than GPT 4o (11M parameters vs 1.8T parameters) - SCOPE is more accurate on Planning tasks (56%) than frontier LLM models The age of efficient AI models starts now. 🔗📌 Read the full write up here:

Skyfall AI

49,170 views • 9 months ago

JARVIS-VLA just dropped on Hugging Face Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse obtain VLA models in Minecraft that can follow human instructions on over 1k different atomic tasks, including crafting, smelting, cooking, mining, and killing. experiments demonstrate that post-training on non-trajectory tasks leads to a significant 40% improvement over the best agent baseline on a diverse set of atomic tasks. Furthermore, demonstrate that approach surpasses traditional imitation learning-based policies in Minecraft, achieving state-of-the-art performance.

JARVIS-VLA just dropped on Hugging Face Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse obtain VLA models in Minecraft that can follow human instructions on over 1k different atomic tasks, including crafting, smelting, cooking, mining, and killing. experiments demonstrate that post-training on non-trajectory tasks leads to a significant 40% improvement over the best agent baseline on a diverse set of atomic tasks. Furthermore, demonstrate that approach surpasses traditional imitation learning-based policies in Minecraft, achieving state-of-the-art performance.

AK

60,243 views • 1 year ago

🚨 Frontier Model Router - Mix And Match All the Frontier LLMs Our most popular custom router is our frontier-model router that combines the best LLMs for all your tasks GPT 5.6 sol - data analysis Fable / Opus 4.8 - coding Flash 3.5 - summarization Grok 4.5 - real-time Pick your favorite LLMs for all your tasks. As of today the custom routers can be used in ANY harness - Abacus AI, Claude, Codex, Open-Code....

🚨 Frontier Model Router - Mix And Match All the Frontier LLMs Our most popular custom router is our frontier-model router that combines the best LLMs for all your tasks GPT 5.6 sol - data analysis Fable / Opus 4.8 - coding Flash 3.5 - summarization Grok 4.5 - real-time Pick your favorite LLMs for all your tasks. As of today the custom routers can be used in ANY harness - Abacus AI, Claude, Codex, Open-Code....

Bindu Reddy

4,592,609 views • 3 months ago

LLMs are either not capable or not affordable for visual reasoning tasks at scale. The Elorian team - one of Jeff Dean’s best reports who led Gemini, an Apple LLM lead, xAI lead, and a Harvard professor - just launched an Astra-level visual reasoning model, Glimpse, for 17x cheaper. Watch it break down the flying Tesla video physics here. Extremely critical for video inputs in manufacturing, robotics, gaming, content moderation and more.

LLMs are either not capable or not affordable for visual reasoning tasks at scale. The Elorian team - one of Jeff Dean’s best reports who led Gemini, an Apple LLM lead, xAI lead, and a Harvard professor - just launched an Astra-level visual reasoning model, Glimpse, for 17x cheaper. Watch it break down the flying Tesla video physics here. Extremely critical for video inputs in manufacturing, robotics, gaming, content moderation and more.

Deedy

164,252 views • 10 days ago

one of the most challenging tasks for frontier models is being able to extract thousands of values from extremely dense tables in documents. our new Extract v2.5 agents are able to get 93%-96%+ on long-list extraction, including cells that fall in between pages. in contrast, astra gets ~30% accuracy over this benchmark. in these cases, frontier VLMs can stop early or miss repeated records. they also struggle to attribute each value back to the source. we are able to attribute every single extracted value back to the source. check out the video below, the blog post, and llamaparse! blog: llamaparse:

one of the most challenging tasks for frontier models is being able to extract thousands of values from extremely dense tables in documents. our new Extract v2.5 agents are able to get 93%-96%+ on long-list extraction, including cells that fall in between pages. in contrast, astra gets ~30% accuracy over this benchmark. in these cases, frontier VLMs can stop early or miss repeated records. they also struggle to attribute each value back to the source. we are able to attribute every single extracted value back to the source. check out the video below, the blog post, and llamaparse! blog: llamaparse:

Jerry Liu

13,305 views • 25 days ago

For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation

For the first time in human history, we are teaching a Foundation Model to master the diverse tasks of medicinal chemists, biologists, and computational scientists all in one place. In our latest collaboration with Liquid AI, we are moving away from fragmented, specialized tools toward a single, super-intelligent model. What surprised me most? This model isn't just performing at reasonable levels—it has started outperforming specialist models across physics-based tasks, imaging, and longitudinal data. Why this changes everything: -Synergy over Specialization: Fine-tuning on specific tasks has unlocked unexpected capabilities in synergetic areas, opening a new frontier in multimodal AI research. -Zero-Shot Potential: We are building a model that can perform out-of-scope tasks, moving us closer to an "AI deity" for drug discovery. -Quality First: The goal isn't just to bypass regulations to save time; it’s about using these synergies to develop better, more effective drugs. We are no longer just looking at linear regression or simple text; we are looking at the future of how humanity fights disease. #LiquidAI #InsilicoMedicine #GenerativeAI #DrugDiscovery #DeepTech #BiotechInnovation

Alex Zhavoronkov, PhD (aka Aleksandrs Zavoronkovs)

10,544 views • 7 months ago

The most effective AI systems don't rely on a single model. Frontier models provide state-of-the-art performance for complex tasks, while routers automatically select lightweight, open-source models for simpler jobs to optimize accuracy, latency, and cost. Learn more:

The most effective AI systems don't rely on a single model. Frontier models provide state-of-the-art performance for complex tasks, while routers automatically select lightweight, open-source models for simpler jobs to optimize accuracy, latency, and cost. Learn more:

NVIDIA AI

12,152 views • 7 months ago

Bill Gates says scaling AI systems will work for two more iterations and after that the next big frontier is meta-cognition where AI can reason about its tasks

Bill Gates says scaling AI systems will work for two more iterations and after that the next big frontier is meta-cognition where AI can reason about its tasks

Tsarathustra

160,597 views • 2 years ago

We benchmarked leading multimodal foundation models (GPT-4o, Claude 3.5 Sonnet, Gemini, Llama, etc.) on standard computer vision tasks—from segmentation to surface normal estimation—using standard datasets like COCO and ImageNet. These models have made remarkable progress; however, it is unclear exactly where they stand in terms of understanding vision in detail. Especially when it comes to tasks beyond question-answering. How well do they understand an object's segments or geometry? Our analyses yield an assessment that is quantitatively and qualitatively detailed and is compatible with evaluations developed in the field of computer vision over the past decades. Observed trends: 🔹 The foundation models consistently underperform task-specific SOTA models across all tasks. However, they are respectable generalists, which is remarkable as they are presumably trained primarily on image-text-based tasks. 🔹 They perform semantic tasks notably better than geometric ones. 🔹 GPT-4o performs the best among non-reasoning models, getting the top position in 4 out of 6 tasks. 🔹 Reasoning models, e.g., o3, show improvements in geometric tasks. 🔹 The 'image generation' models, e.g., GPT-40 Image Generation, which have been natively trained multimodally, exhibit quirks. E.g., hallucinated objects, misalignment between the input and output, etc. 🔹 While the prompting techniques affect performance, better models exhibit less sensitivity to variations in prompts. We control for the variance introduced by the prompting methods in our experiments. 🌐 Detailed analyses, visualizations: ⌨️ code: 🧵 1/n

We benchmarked leading multimodal foundation models (GPT-4o, Claude 3.5 Sonnet, Gemini, Llama, etc.) on standard computer vision tasks—from segmentation to surface normal estimation—using standard datasets like COCO and ImageNet. These models have made remarkable progress; however, it is unclear exactly where they stand in terms of understanding vision in detail. Especially when it comes to tasks beyond question-answering. How well do they understand an object's segments or geometry? Our analyses yield an assessment that is quantitatively and qualitatively detailed and is compatible with evaluations developed in the field of computer vision over the past decades. Observed trends: 🔹 The foundation models consistently underperform task-specific SOTA models across all tasks. However, they are respectable generalists, which is remarkable as they are presumably trained primarily on image-text-based tasks. 🔹 They perform semantic tasks notably better than geometric ones. 🔹 GPT-4o performs the best among non-reasoning models, getting the top position in 4 out of 6 tasks. 🔹 Reasoning models, e.g., o3, show improvements in geometric tasks. 🔹 The 'image generation' models, e.g., GPT-40 Image Generation, which have been natively trained multimodally, exhibit quirks. E.g., hallucinated objects, misalignment between the input and output, etc. 🔹 While the prompting techniques affect performance, better models exhibit less sensitivity to variations in prompts. We control for the variance introduced by the prompting methods in our experiments. 🌐 Detailed analyses, visualizations: ⌨️ code: 🧵 1/n

Amir Zamir

73,502 views • 1 year ago

$NBIS main thing is, if we have the demand, all the rest are not problems, they're tasks." - Arkady Financing, building, sourcing chips: all tasks. And with one of the best management teams in the industry, those tasks get done smoothly, always with an eye on the next step. The demand is there, and boy oh boy, it's far outstripping supply. Bullish!!

$NBIS main thing is, if we have the demand, all the rest are not problems, they're tasks." - Arkady Financing, building, sourcing chips: all tasks. And with one of the best management teams in the industry, those tasks get done smoothly, always with an eye on the next step. The demand is there, and boy oh boy, it's far outstripping supply. Bullish!!

Ren

23,163 views • 6 days ago

Emad Mostaque says by next year, AI reasoning models like OpenAI's o1 and DeepSeek's R1 will run on smartphones and perform PhD-level tasks with 20 watts of electricity - equivalent to the human brain

Emad Mostaque says by next year, AI reasoning models like OpenAI's o1 and DeepSeek's R1 will run on smartphones and perform PhD-level tasks with 20 watts of electricity - equivalent to the human brain

Tsarathustra

157,256 views • 1 year ago