Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🎯 One CLI to evaluate every VLA benchmark Evaluating a robot policy used to mean wrangling six different repos, six different setups. Not anymore. LeRobot 0.6.0 ships 6 new simulation benchmarks - all runnable through the same lerobot-eval CLI, each with its own docs page, Docker image, and SmolVLA...

LeRobotHF's profile picture

LeRobot

18,820 subscribers

11,307 Aufrufe • vor 1 Monat •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

HERMES AGENT SUPPORTS 7 TYPES OF AI AGENTS. EACH ONE TAKES LESS THAN 90 SECONDS TO SET UP. MOST PEOPLE ONLY BUILD THE FIRST ONE. HERE ARE ALL SEVEN AND WHEN TO USE EACH. 1. BASIC AGENT WITH TOOLS your agent with access to terminal, browser, file system, web search, and calendar. it plans and executes tasks on its own. this is what you get on day one. "find flights to Lisbon under $400" "check my calendar and flag conflicts" "search the web for competitor pricing" set in Desktop app / Dashboard: Tools → enable what you need. when to use: single tasks that need tool access. 2. AGENT WITH MCP SERVERS connect your agent to external services. Notion, Google Drive, GitHub, Slack, databases, APIs, any MCP-compatible service. the agent doesn't scrape these services. it interacts through structured APIs. reads your Notion pages. creates GitHub issues. queries your database. sends Slack messages. set in Desktop app / Dashboard: MCP → Add Server. when to use: your workflow lives across multiple platforms. 3. SEQUENTIAL AGENTS (pipeline) one agent finishes. passes output to the next. assembly line for AI. agent 1: scans inbox for leads. agent 2: qualifies leads against criteria. agent 3: drafts outreach emails. in Hermes: cron jobs with wakeAgent gates. agent 1 writes output to a file. agent 2 wakes only when that file has new data. agent 3 wakes when agent 2 is done. each agent = a separate profile with its own model. when to use: multi-step workflows where each step depends on the previous one finishing. 4. PARALLEL EXECUTION AGENTS multiple agents working at the same time. results merge when all finish. "research these 5 competitors in parallel" in Hermes: delegate_task with batch mode. up to 3 sub-agents running in parallel by default. each gets its own clean context. only summaries return to the parent. delegation: model: "deepseek/deepseek-v4" children run cheap. parent synthesizes. when to use: independent tasks that don't depend on each other. research, data gathering, analysis. 5. AGENTS WITH ROUTERS conditions that send tasks down different paths based on the input. "if sales email → SDR profile. if support ticket → support profile. if calendar invite → EA profile." in Hermes: Kanban decompose. the decomposer reads profile descriptions and routes each task to the best-fit agent. or: Chief of Staff profile that triages and assigns to other profiles. when to use: incoming work that needs different specialists based on type. 6. HUMAN IN THE LOOP the agent does the work. asks for your approval before executing. "I drafted this email. approve before I send?" "this command will delete 3 files. proceed?" in Hermes: approvals.mode: manual (default). every dangerous action needs your confirmation. 60-second timeout. fails closed. or smart mode: LLM assesses risk. safe actions auto-approved. dangerous ones ask you. uncertain ones escalate. when to use: tasks where a mistake has real consequences. emails, deployments, financial transactions, public posts. 7. DYNAMIC SUB-AGENT SPAWNING your main agent realizes it needs help and spawns specialized sub-agents on the fly. "build this feature" → parent delegates: → sub-agent 1: research the API docs → sub-agent 2: write the code → sub-agent 3: write the tests in Hermes: delegate_task with role: orchestrator. raise max_spawn_depth for nested delegation. delegation: max_spawn_depth: 2 orchestrator_enabled: true depth 2 with concurrency 3 = up to 9 parallel workers. each level multiplies the spend. raise depth only when you need multi-level trees. when to use: complex tasks where the agent discovers what help it needs during execution. THE PROGRESSION: start with 1 (tools) and 6 (approvals). add 2 (MCP) when you need external services. add 4 (parallel) when tasks take too long one at a time. add 3 (sequential) when you build multi-step pipelines. add 5 (routing) when you run multiple profiles. add 7 (dynamic) when single-agent reasoning falls short. seven types. each under 90 seconds to configure. the value compounds as you stack them. comment AGENTS and I'll send you 3 ready-to-build agent setups that combine these types into real workflows.
1:15

HERMES AGENT SUPPORTS 7 TYPES OF AI AGENTS. EACH ONE TAKES LESS THAN 90 SECONDS TO SET UP. MOST PEOPLE ONLY BUILD THE FIRST ONE. HERE ARE ALL SEVEN AND WHEN TO USE EACH. 1. BASIC AGENT WITH TOOLS your agent with access to terminal, browser, file system, web search, and calendar. it plans and executes tasks on its own. this is what you get on day one. "find flights to Lisbon under $400" "check my calendar and flag conflicts" "search the web for competitor pricing" set in Desktop app / Dashboard: Tools → enable what you need. when to use: single tasks that need tool access. 2. AGENT WITH MCP SERVERS connect your agent to external services. Notion, Google Drive, GitHub, Slack, databases, APIs, any MCP-compatible service. the agent doesn't scrape these services. it interacts through structured APIs. reads your Notion pages. creates GitHub issues. queries your database. sends Slack messages. set in Desktop app / Dashboard: MCP → Add Server. when to use: your workflow lives across multiple platforms. 3. SEQUENTIAL AGENTS (pipeline) one agent finishes. passes output to the next. assembly line for AI. agent 1: scans inbox for leads. agent 2: qualifies leads against criteria. agent 3: drafts outreach emails. in Hermes: cron jobs with wakeAgent gates. agent 1 writes output to a file. agent 2 wakes only when that file has new data. agent 3 wakes when agent 2 is done. each agent = a separate profile with its own model. when to use: multi-step workflows where each step depends on the previous one finishing. 4. PARALLEL EXECUTION AGENTS multiple agents working at the same time. results merge when all finish. "research these 5 competitors in parallel" in Hermes: delegate_task with batch mode. up to 3 sub-agents running in parallel by default. each gets its own clean context. only summaries return to the parent. delegation: model: "deepseek/deepseek-v4" children run cheap. parent synthesizes. when to use: independent tasks that don't depend on each other. research, data gathering, analysis. 5. AGENTS WITH ROUTERS conditions that send tasks down different paths based on the input. "if sales email → SDR profile. if support ticket → support profile. if calendar invite → EA profile." in Hermes: Kanban decompose. the decomposer reads profile descriptions and routes each task to the best-fit agent. or: Chief of Staff profile that triages and assigns to other profiles. when to use: incoming work that needs different specialists based on type. 6. HUMAN IN THE LOOP the agent does the work. asks for your approval before executing. "I drafted this email. approve before I send?" "this command will delete 3 files. proceed?" in Hermes: approvals.mode: manual (default). every dangerous action needs your confirmation. 60-second timeout. fails closed. or smart mode: LLM assesses risk. safe actions auto-approved. dangerous ones ask you. uncertain ones escalate. when to use: tasks where a mistake has real consequences. emails, deployments, financial transactions, public posts. 7. DYNAMIC SUB-AGENT SPAWNING your main agent realizes it needs help and spawns specialized sub-agents on the fly. "build this feature" → parent delegates: → sub-agent 1: research the API docs → sub-agent 2: write the code → sub-agent 3: write the tests in Hermes: delegate_task with role: orchestrator. raise max_spawn_depth for nested delegation. delegation: max_spawn_depth: 2 orchestrator_enabled: true depth 2 with concurrency 3 = up to 9 parallel workers. each level multiplies the spend. raise depth only when you need multi-level trees. when to use: complex tasks where the agent discovers what help it needs during execution. THE PROGRESSION: start with 1 (tools) and 6 (approvals). add 2 (MCP) when you need external services. add 4 (parallel) when tasks take too long one at a time. add 3 (sequential) when you build multi-step pipelines. add 5 (routing) when you run multiple profiles. add 7 (dynamic) when single-agent reasoning falls short. seven types. each under 90 seconds to configure. the value compounds as you stack them. comment AGENTS and I'll send you 3 ready-to-build agent setups that combine these types into real workflows.

YanXbt

17,312 Aufrufe • vor 1 Monat

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10–15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paper’s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and π0.5 across the tested tasks. 🧵 1.
1:17

Most video-action robot models are a content-creation video generator with an action module attached. LingBot-VA 2.0 from Robbyant, a video-action foundation model, throws that starting point out and trains the whole stack natively for control. And it runs closed-loop at a peak 225 Hz. It's so important because A robot cannot move responsively when its controller pauses to imagine the next few frames. LingBot-VA 2.0 predicts during execution, then corrects using each real observation. And it carries only about 13B video parameters while activating roughly 1.9B per token. Bigger robot models usually mean slower reactions, creating a direct conflict between intelligence and control. LingBot-VA 2.0 is trained from scratch for robot control rather than adapted from a video generator built for content creation. Robbyant, an embodied AI company under Ant Group, built it to learn how scenes change under actions, predict what should happen next, and turn those predictions into real-time robot movements. Most video-action systems inherit a tokenizer and video backbone trained mainly to reproduce visual appearance. LingBot-VA 2.0 rebuilds both parts around physical control. Its semantic visual-action tokenizer maps observations toward features from a frozen vision foundation model and learns compact latent actions from frame-to-frame changes using self-supervised inverse and forward dynamics. Unlabeled web video can therefore carry action-relevant training signals without robot action labels. The policy is causal from the start, so every prediction can use only past observations. Its sparse Mixture-of-Experts video backbone has about 13B total parameters, while about 1.9B are active per token, keeping the compute lower during each step. A high-level vision-language planner breaks long tasks into smaller instructions, while the low-level video-action policy handles continuous movement. Foresight Reasoning predicts future visual states while the robot is already acting, then replaces imagined states with every new real observation. Combined with few-step distillation and systems acceleration, the paper reports a peak asynchronous execution frequency of 225 Hz. The model adapts from 10–15 demonstrations, transfers across robot embodiments, and handles some new tasks zero-shot. In the paper’s own evaluations, it reaches 93.6 average on RoboTwin 2.0 and reports stronger real-world results than LingBot-VA and π0.5 across the tested tasks. 🧵 1.

Rohan Paul

11,253 Aufrufe • vor 1 Monat

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. "Trained model" is a repo of sensorimotor skills instead of floating weights. “Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches. Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;) Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! Deep dive in thread:
0:40

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. "Trained model" is a repo of sensorimotor skills instead of floating weights. “Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches. Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;) Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! Deep dive in thread:

Jim Fan

209,534 Aufrufe • vor 1 Monat

CHINA JUST SOLVED THE PROBLEM THAT'S BEEN BREAKING ROBOT AI FOR A DECADE. and the fix wasn't a smarter model. for years, every robot AI failure got the same diagnosis. the model isn't smart enough. so everyone scaled intelligence. bigger models. more parameters. better reasoning. AGIBOT asked a different question: what if the reasoning was never the problem? there's a gap that runs through every traditional robot AI system. reasoning on one side & motor commands on the other. the brain decides but the body executes something different, because thinking and moving were never actually connected. GO-2 fixes this by reasoning INSIDE the action space, not above it. before moving, it runs a complete mental simulation of every step - like a basketball player mentally tracing the arc of a shot before releasing the ball. watch the demo and you'll see exactly what this means. the robot works through a task queue autonomously. classify toiletries. upright the drink bottle. place headphones in the leather box. mid-execution, a new instruction drops: "my phone's missing. help me find it." it doesn't pause. doesn't reset. it processes the new task and keeps moving. that's not a scripted sequence. that's real-time instruction following on top of an active task queue. that one architectural change is where the numbers come from. > #1 on LIBERO across Spatial, Object, Goal, and Long tasks → 98.5% average success > 86.6% zero-shot accuracy in active disturbance environments > 47.4 on VLABench → best-in-class on objects and textures it's never seen before > 82.9% success trained on simulation only, tested on real hardware sim-to-real is the graveyard of robotics research. models trained in simulation collapse the moment they touch the real world. 82.9% means that graveyard just got a lot smaller. it holds because of how GO-2 trains. deliberately fed imperfect reasoning conditions, then trained to execute robustly anyway. not a researcher assumption. a design decision from a team that ships hardware and knows exactly what breaks. then there's the infrastructure layer. Genie Studio. fleet-wide data collection. cloud training. online post-training in live environments. 10x improvement in training efficiency. task startup reduced to minutes. 2-4x better success rates with 50%+ less data. the model gets smarter every time a robot fails in the field. this isn't a benchmark story. it's a compounding moat. dual CVPR 2026 + ACL 2026 acceptance. computer vision AND natural language processing. top conferences. simultaneously. that doesn't happen with incremental research. the US-China robotics race has been framed as a compute race. a model quality race. it was always an execution race. the robot that wins won't be the smartest one in the lab. it'll be the most reliable one on the floor. full breakdown: is execution reliability the real bottleneck, or are we still underestimating how far reasoning needs to go?
2:55

CHINA JUST SOLVED THE PROBLEM THAT'S BEEN BREAKING ROBOT AI FOR A DECADE. and the fix wasn't a smarter model. for years, every robot AI failure got the same diagnosis. the model isn't smart enough. so everyone scaled intelligence. bigger models. more parameters. better reasoning. AGIBOT asked a different question: what if the reasoning was never the problem? there's a gap that runs through every traditional robot AI system. reasoning on one side & motor commands on the other. the brain decides but the body executes something different, because thinking and moving were never actually connected. GO-2 fixes this by reasoning INSIDE the action space, not above it. before moving, it runs a complete mental simulation of every step - like a basketball player mentally tracing the arc of a shot before releasing the ball. watch the demo and you'll see exactly what this means. the robot works through a task queue autonomously. classify toiletries. upright the drink bottle. place headphones in the leather box. mid-execution, a new instruction drops: "my phone's missing. help me find it." it doesn't pause. doesn't reset. it processes the new task and keeps moving. that's not a scripted sequence. that's real-time instruction following on top of an active task queue. that one architectural change is where the numbers come from. > #1 on LIBERO across Spatial, Object, Goal, and Long tasks → 98.5% average success > 86.6% zero-shot accuracy in active disturbance environments > 47.4 on VLABench → best-in-class on objects and textures it's never seen before > 82.9% success trained on simulation only, tested on real hardware sim-to-real is the graveyard of robotics research. models trained in simulation collapse the moment they touch the real world. 82.9% means that graveyard just got a lot smaller. it holds because of how GO-2 trains. deliberately fed imperfect reasoning conditions, then trained to execute robustly anyway. not a researcher assumption. a design decision from a team that ships hardware and knows exactly what breaks. then there's the infrastructure layer. Genie Studio. fleet-wide data collection. cloud training. online post-training in live environments. 10x improvement in training efficiency. task startup reduced to minutes. 2-4x better success rates with 50%+ less data. the model gets smarter every time a robot fails in the field. this isn't a benchmark story. it's a compounding moat. dual CVPR 2026 + ACL 2026 acceptance. computer vision AND natural language processing. top conferences. simultaneously. that doesn't happen with incremental research. the US-China robotics race has been framed as a compute race. a model quality race. it was always an execution race. the robot that wins won't be the smartest one in the lab. it'll be the most reliable one on the floor. full breakdown: is execution reliability the real bottleneck, or are we still underestimating how far reasoning needs to go?

Shruti

18,622 Aufrufe • vor 4 Monaten

Karpathy said something you'll regret ignoring: "You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding. " He said it while drawing the line between vibe coding and agentic engineering. Agents write more of the code now, but none of that takes the responsibility off you. The assumption underneath that is that a careful enough reader catches the problem. But some failures don't show up in anything there is to read. For instance, a common fear with a RAG agent is that it could hallucinate when a question asks something outside its corpus. But such cases are actually well handled by any competent model now. If nothing in the retrieved context looks relevant, there's no material to build an answer on. Instead, the majority of failures originate when the retrieved context has partial coverage. The retrieval pipeline returns context that's topically correct but doesn't cover the full question, and the model completes the remainder from parametric knowledge. There are no token-level labels in the output to tell what was generated using retrieved context and what came from weights. Both are streamed the same way. Detecting this for production-grade apps needs a metric written for it, one that's also aligned with principles of agentic engineering. And the solution is actually implemented in the eval skill that comes with Google’s Agents CLI. I described the concern to Claude Code in plain English. It read the agent's code, came back with a plan I approved. It then reported that no built-in metric isolates the behaviour and wrote a custom rubric called corpus_abstention. It assigned a single categorical verdict per case rather than aggregating everything into one score, since the built-in raters regenerate their rubrics each run and leave no stable number to trend. → GROUNDED_ANSWER → CORRECT_ABSTENTION → UNGROUNDED_ANSWER (answered entirely from outside knowledge) → MIXED_LEAKAGE (grounded, but slips in one unsupported claim) → WRONG_ABSTENTION (refused something the docs actually covered) After this, it automatically generated 33 scenarios partitioned by where the failure could occur, like: - in-corpus - off-domain - out-of-corpus but plausibly answerable - boundary cases where the topic is covered, but a specific detail isn't. The baseline score was 19 of 33. - Off-domain passed 3 of 3, as expected. - But 6 of 15 in-corpus cases retrieved the right document, cited it correctly, answered accurately, and added a claim the source never made. The root cause was one line in the agent's instruction: "If you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations." The eval skill helped flag this, and then Claude removed it and forced retrieval on every question. This took the suite to 30 of 33, and ungrounded answers went from 6 to 0. The full recording of my run is below, and I worked with the Google Cloud team on this. Agents CLI GitHub repo → (don't forget to star 🌟) I wrote up the full build covering all six steps from install to enterprise registration. It includes the eval scorecard, the instruction loophole the eval caught before deployment, and what the deployment process actually looks like end-to-end. Read it below.
0:29

Karpathy said something you'll regret ignoring: "You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding. " He said it while drawing the line between vibe coding and agentic engineering. Agents write more of the code now, but none of that takes the responsibility off you. The assumption underneath that is that a careful enough reader catches the problem. But some failures don't show up in anything there is to read. For instance, a common fear with a RAG agent is that it could hallucinate when a question asks something outside its corpus. But such cases are actually well handled by any competent model now. If nothing in the retrieved context looks relevant, there's no material to build an answer on. Instead, the majority of failures originate when the retrieved context has partial coverage. The retrieval pipeline returns context that's topically correct but doesn't cover the full question, and the model completes the remainder from parametric knowledge. There are no token-level labels in the output to tell what was generated using retrieved context and what came from weights. Both are streamed the same way. Detecting this for production-grade apps needs a metric written for it, one that's also aligned with principles of agentic engineering. And the solution is actually implemented in the eval skill that comes with Google’s Agents CLI. I described the concern to Claude Code in plain English. It read the agent's code, came back with a plan I approved. It then reported that no built-in metric isolates the behaviour and wrote a custom rubric called corpus_abstention. It assigned a single categorical verdict per case rather than aggregating everything into one score, since the built-in raters regenerate their rubrics each run and leave no stable number to trend. → GROUNDED_ANSWER → CORRECT_ABSTENTION → UNGROUNDED_ANSWER (answered entirely from outside knowledge) → MIXED_LEAKAGE (grounded, but slips in one unsupported claim) → WRONG_ABSTENTION (refused something the docs actually covered) After this, it automatically generated 33 scenarios partitioned by where the failure could occur, like: - in-corpus - off-domain - out-of-corpus but plausibly answerable - boundary cases where the topic is covered, but a specific detail isn't. The baseline score was 19 of 33. - Off-domain passed 3 of 3, as expected. - But 6 of 15 in-corpus cases retrieved the right document, cited it correctly, answered accurately, and added a claim the source never made. The root cause was one line in the agent's instruction: "If you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations." The eval skill helped flag this, and then Claude removed it and forced retrieval on every question. This took the suite to 30 of 33, and ungrounded answers went from 6 to 0. The full recording of my run is below, and I worked with the Google Cloud team on this. Agents CLI GitHub repo → (don't forget to star 🌟) I wrote up the full build covering all six steps from install to enterprise registration. It includes the eval scorecard, the instruction loophole the eval caught before deployment, and what the deployment process actually looks like end-to-end. Read it below.

Akshay 🚀

82,546 Aufrufe • vor 1 Monat

This Chinese developer runs 9 agents on Claude Code under a GPT-5.5 orchestrator and they close 500 client tasks a month without a single assistant. His client work is closed without him, on a single laptop and only three subscriptions. The entire system lives on one MacBook Pro M4 with 128 GB of memory and subscriptions to Claude Code and GPT-5.5 cost him approximately $300 a month. There is no CRM, no team, no office only a terminal window with 9 parallel streams. The orchestrator works with a simple system prompt: «You are the orchestrator of a client inbox. Classify every incoming email into 4 categories: code, content, analysis, communication. Delegate to the corresponding worker agent. When the result is ready, check it for completeness, send it to the client on my behalf, and mark the task as closed. Do not ask clarifying questions.» And the orchestrator checks the inbox every 30 seconds, classifies fresh emails, and distributes them to 9 worker agents on Claude Code, each of whom is responsible for their own class of tasks. Here is an example of how one of them closes a request to refactor a client's auth module: Task: refactor user-auth module Broke the monolith into 3 files by responsibilities Added unit tests, coverage increased to 87% Renamed 4 functions to camelCase according to the style guide PR is ready for review, link below» And so about 50 cycles a day. By noon 25 tasks are closed, by dinner 50, and by the end of the month 500. On average, it takes about 7 minutes from the appearance of an email in the inbox to sending the result to the client. This is more than what a live team of 6 developers, copywriters and analysts working 8 hours a day closes. This is no longer an agency. This is a workstation where an orchestrator replaces a manager, and 9 worker agents replace the staff. The pipeline goes from inbox to closing 500 times a month without human participation at any step.
1:32

This Chinese developer runs 9 agents on Claude Code under a GPT-5.5 orchestrator and they close 500 client tasks a month without a single assistant. His client work is closed without him, on a single laptop and only three subscriptions. The entire system lives on one MacBook Pro M4 with 128 GB of memory and subscriptions to Claude Code and GPT-5.5 cost him approximately $300 a month. There is no CRM, no team, no office only a terminal window with 9 parallel streams. The orchestrator works with a simple system prompt: «You are the orchestrator of a client inbox. Classify every incoming email into 4 categories: code, content, analysis, communication. Delegate to the corresponding worker agent. When the result is ready, check it for completeness, send it to the client on my behalf, and mark the task as closed. Do not ask clarifying questions.» And the orchestrator checks the inbox every 30 seconds, classifies fresh emails, and distributes them to 9 worker agents on Claude Code, each of whom is responsible for their own class of tasks. Here is an example of how one of them closes a request to refactor a client's auth module: Task: refactor user-auth module Broke the monolith into 3 files by responsibilities Added unit tests, coverage increased to 87% Renamed 4 functions to camelCase according to the style guide PR is ready for review, link below» And so about 50 cycles a day. By noon 25 tasks are closed, by dinner 50, and by the end of the month 500. On average, it takes about 7 minutes from the appearance of an email in the inbox to sending the result to the client. This is more than what a live team of 6 developers, copywriters and analysts working 8 hours a day closes. This is no longer an agency. This is a workstation where an orchestrator replaces a manager, and 9 worker agents replace the staff. The pipeline goes from inbox to closing 500 times a month without human participation at any step.

Blaze

29,917 Aufrufe • vor 3 Monaten

NEW ROBOT BENCHMARK: If your robot can do Origami, it can do anything! Called The Robotic Origami Challenge, it is a dexterous-manipulation competition and benchmark held at IROS 2026, organized by 13 co-organizers with the Nippon Origami Association as judge and task curator -> pretty cool to have them on board imho. The evaluation consists of single task: a traditional Japanese paper airplane, in exactly six folds, from a 15×15 cm sheet of ≥60 gsm paper, on a competition-supplied standardized rig (bimanual arms + Sharpa Hands), both remotely and on-site. Teams bring policies, not hardware. An "Origami Grand Master" declares pass/fail on crease accuracy, structural fidelity, symmetry and paper integrity. Among passes, faster folds rank higher, with a 10-minute-per-attempt ceiling and flight explicitly irrelevant to the score. -> I find it interesting how they chose to evaluate the task. Quality is a binary pass or fail, therefore speed becomes the only thing graded here. Speed is currently the bottleneck in dexterous manipulation though, so this choice makes sense. I wonder whether there could be finer ways to grade the qaulity of such a creative task though. When it comes to data, registered teams get 500+ teleoperation episodes (six camera streams, 65-D joint state/action, 10-fingertip 6-axis tactile), an NVIDIA Isaac Sim environment with thin-shell paper physics (plastic creasing + fold memory), digital twins of every partner hand, and a remote eval lab (upload a policy, queue an eval, get scored). Still, I think it is a great dexterity benchmark the field badly needs, it supplies the hardware, an outside human judges, and the pass criterion is externally defined -> all three degrees of freedom are checked! Neutral measurement layer, here we go! The task is engineered to be un-gameable and to isolate pure dexterity. A known figure, exactly six folds, judged on creases, with flight explicitly declared irrelevant (the latter makes sense to me). Therefore, this underlines the goal to focus on dexterity, not task-selection or other strategies. I really like origami as an ideal controlled dexterity task: deterministic goal, deformable medium, sequential, bimanual, precision-bound. I am just not quite satisfied again by the binary pass or fail, I think quality of execution could be finely graded! But again, I understand this is not the goal yet. Also interesting to see the Sharpa Hands as de facto standard for everyone. Total land-grab that anoints Sharpa as the reference dexterous hand, also featured in Google's Gemini Robotics 2. By providing the hardware, the benchmark measures software while quietly making "good on Sharpa" the definition of good, and Sharpa gets real world data and feedback as a bonus. That's smart, the data flywheel starts spinning. The provided dataset is the richest tactile-manipulation corpus I have seen yet: 10-fingertip 6-axis tactile, plus plastic creasing and fold memory. 500+ teleop episodes with six camera streams, 65-D joint state/action, and ten fingertip 6-axis tactile sensors. The force/tactile channel are parts of the the benchmark's data, this is the first time I see this. Credits where it's due: organizers include Yang Gao, Noriaki Hirose, Steve Xie, Chris Paxton, Jiafei Duan, Michael Cho - Rbt/Acc, Michael Yuan, Haoquan Fang, and others.
0:36

NEW ROBOT BENCHMARK: If your robot can do Origami, it can do anything! Called The Robotic Origami Challenge, it is a dexterous-manipulation competition and benchmark held at IROS 2026, organized by 13 co-organizers with the Nippon Origami Association as judge and task curator -> pretty cool to have them on board imho. The evaluation consists of single task: a traditional Japanese paper airplane, in exactly six folds, from a 15×15 cm sheet of ≥60 gsm paper, on a competition-supplied standardized rig (bimanual arms + Sharpa Hands), both remotely and on-site. Teams bring policies, not hardware. An "Origami Grand Master" declares pass/fail on crease accuracy, structural fidelity, symmetry and paper integrity. Among passes, faster folds rank higher, with a 10-minute-per-attempt ceiling and flight explicitly irrelevant to the score. -> I find it interesting how they chose to evaluate the task. Quality is a binary pass or fail, therefore speed becomes the only thing graded here. Speed is currently the bottleneck in dexterous manipulation though, so this choice makes sense. I wonder whether there could be finer ways to grade the qaulity of such a creative task though. When it comes to data, registered teams get 500+ teleoperation episodes (six camera streams, 65-D joint state/action, 10-fingertip 6-axis tactile), an NVIDIA Isaac Sim environment with thin-shell paper physics (plastic creasing + fold memory), digital twins of every partner hand, and a remote eval lab (upload a policy, queue an eval, get scored). Still, I think it is a great dexterity benchmark the field badly needs, it supplies the hardware, an outside human judges, and the pass criterion is externally defined -> all three degrees of freedom are checked! Neutral measurement layer, here we go! The task is engineered to be un-gameable and to isolate pure dexterity. A known figure, exactly six folds, judged on creases, with flight explicitly declared irrelevant (the latter makes sense to me). Therefore, this underlines the goal to focus on dexterity, not task-selection or other strategies. I really like origami as an ideal controlled dexterity task: deterministic goal, deformable medium, sequential, bimanual, precision-bound. I am just not quite satisfied again by the binary pass or fail, I think quality of execution could be finely graded! But again, I understand this is not the goal yet. Also interesting to see the Sharpa Hands as de facto standard for everyone. Total land-grab that anoints Sharpa as the reference dexterous hand, also featured in Google's Gemini Robotics 2. By providing the hardware, the benchmark measures software while quietly making "good on Sharpa" the definition of good, and Sharpa gets real world data and feedback as a bonus. That's smart, the data flywheel starts spinning. The provided dataset is the richest tactile-manipulation corpus I have seen yet: 10-fingertip 6-axis tactile, plus plastic creasing and fold memory. 500+ teleop episodes with six camera streams, 65-D joint state/action, and ten fingertip 6-axis tactile sensors. The force/tactile channel are parts of the the benchmark's data, this is the first time I see this. Credits where it's due: organizers include Yang Gao, Noriaki Hirose, Steve Xie, Chris Paxton, Jiafei Duan, Michael Cho - Rbt/Acc, Michael Yuan, Haoquan Fang, and others.

Léo

28,568 Aufrufe • vor 20 Tagen

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:
1:17

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill. Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible. Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream. We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached. As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first. Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset. A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities: - Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090. - Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor. - Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! Links in thread:

Jim Fan

227,634 Aufrufe • vor 6 Monaten

This Chinese developer launched 6 agents under 1 orchestrator, and they run his UI design agency at $32,000 a month on their own. He built a system of 6 agents on Claude Sonnet 4.6 that single-handedly runs his agency for UI auditing and redesign for SaaS startups and e-commerce. No contractors, no project manager, and no team. Just him, a MacBook, and 1 API key. Traditional design agencies out of Shenzhen keep teams of 8 people on salaries for the same volume, while he keeps only API tokens. 6 agents work through a single orchestrator on Claude Code Router. Usage is about 4 million tokens a day, the average API bill is just $480 a month. All 6 go through MCP servers and write shared state to the file system, without shared state in memory and without race conditions. And here is the system prompt he gave the orchestrator before launch: "you are the orchestrator of a one-man UI agency. you delegate read-only research tasks to 5 sub-agents and own all writes. sub-agents: // Hunter (finds SaaS and e-commerce sites with outdated UI) // Auditor (runs each site through Lighthouse, accessibility, and design system checks) // Pitcher (writes cold outreach and redesign proposals with before/after screenshots) // Splitter (breaks accepted projects into typed milestones) // Designer (generates Figma mockups and Tailwind components) // Checker (runs evals on every artifact before it leaves the harness). you never let 2 sub-agents touch 1 file. you stop and request human approval only when an invoice exceeds $5,000 or when the design system eval score drops below 0.88." Meaning the system knows exactly what it is and within what boundaries it operates. It knows it is supposed to find clients on its own. It knows it is supposed to write proposals with screenshots and mockups without intervention. It knows the human only plugs in when the amounts go above $5,000 or when the design system eval does not converge. → The system runs 24 hours a day → Hunter finds about 200 sites with outdated UI a day → Auditor runs each one through Lighthouse and WCAG → Pitcher prepares about 28 personalized proposals with before/after screenshots → Splitter breaks 3 accepted projects per week into milestones → Designer generates mockups and components, Checker runs evals on every artifact And only when the invoice breaks $5,000 or the eval drops below 0.88 does the orchestrator wake the human. Here is what the system outputs in his log during 1 of the sessions: "hunter report, tuesday: 213 sites found, 31 with last redesign before 2020, 14 with Lighthouse score below 65, 6 with active redesign RFP. passing top 6 to auditor." "pitcher: 27 cold outreach sent with before/after screenshots, 5 replies, 3 discovery calls scheduled. passing to splitter." "designer: milestone 2 of Lotus Tea Co redesign complete. Figma frames exported to /Users/dev/agency/clients/lotus/v2. checker running design system evals." "eval flag: proposal for $6,800 exceeds the approved limit of $5,000. sending for manual review." He has no remote server. No separate backend. Just a local file sandbox in /Users/dev/agency, an MCP router, and an API key to Claude. Out of everything I have seen this year, this is the cleanest one-person UI design agency: $480 in, about $32,000 out, and between them 6 prompts and 1 file system.
1:07

This Chinese developer launched 6 agents under 1 orchestrator, and they run his UI design agency at $32,000 a month on their own. He built a system of 6 agents on Claude Sonnet 4.6 that single-handedly runs his agency for UI auditing and redesign for SaaS startups and e-commerce. No contractors, no project manager, and no team. Just him, a MacBook, and 1 API key. Traditional design agencies out of Shenzhen keep teams of 8 people on salaries for the same volume, while he keeps only API tokens. 6 agents work through a single orchestrator on Claude Code Router. Usage is about 4 million tokens a day, the average API bill is just $480 a month. All 6 go through MCP servers and write shared state to the file system, without shared state in memory and without race conditions. And here is the system prompt he gave the orchestrator before launch: "you are the orchestrator of a one-man UI agency. you delegate read-only research tasks to 5 sub-agents and own all writes. sub-agents: // Hunter (finds SaaS and e-commerce sites with outdated UI) // Auditor (runs each site through Lighthouse, accessibility, and design system checks) // Pitcher (writes cold outreach and redesign proposals with before/after screenshots) // Splitter (breaks accepted projects into typed milestones) // Designer (generates Figma mockups and Tailwind components) // Checker (runs evals on every artifact before it leaves the harness). you never let 2 sub-agents touch 1 file. you stop and request human approval only when an invoice exceeds $5,000 or when the design system eval score drops below 0.88." Meaning the system knows exactly what it is and within what boundaries it operates. It knows it is supposed to find clients on its own. It knows it is supposed to write proposals with screenshots and mockups without intervention. It knows the human only plugs in when the amounts go above $5,000 or when the design system eval does not converge. → The system runs 24 hours a day → Hunter finds about 200 sites with outdated UI a day → Auditor runs each one through Lighthouse and WCAG → Pitcher prepares about 28 personalized proposals with before/after screenshots → Splitter breaks 3 accepted projects per week into milestones → Designer generates mockups and components, Checker runs evals on every artifact And only when the invoice breaks $5,000 or the eval drops below 0.88 does the orchestrator wake the human. Here is what the system outputs in his log during 1 of the sessions: "hunter report, tuesday: 213 sites found, 31 with last redesign before 2020, 14 with Lighthouse score below 65, 6 with active redesign RFP. passing top 6 to auditor." "pitcher: 27 cold outreach sent with before/after screenshots, 5 replies, 3 discovery calls scheduled. passing to splitter." "designer: milestone 2 of Lotus Tea Co redesign complete. Figma frames exported to /Users/dev/agency/clients/lotus/v2. checker running design system evals." "eval flag: proposal for $6,800 exceeds the approved limit of $5,000. sending for manual review." He has no remote server. No separate backend. Just a local file sandbox in /Users/dev/agency, an MCP router, and an API key to Claude. Out of everything I have seen this year, this is the cleanest one-person UI design agency: $480 in, about $32,000 out, and between them 6 prompts and 1 file system.

Blaze

56,062 Aufrufe • vor 3 Monaten

JUST IN: Perplexity launched "Perplexity Computer" — and it might be the most complete AI agent system available right now. Not a chatbot upgrade. Not a research tool with a new name. A system that plans entire projects, delegates to specialist AI models, and runs autonomously for hours, days, or months (their words). Here's what makes the architecture genuinely different: → Opus 4.6 handles core reasoning and orchestration → Gemini handles deep research (spawning its own sub-agents) → Grok handles lightweight speed tasks → Veo 3.1 handles video generation → Nano Banana handles image creation → ChatGPT 5.2 handles long-context recall and wide search → You can override model choices per subtask 19 models total. Each task runs in an isolated environment with a real filesystem, real browser, and real tool integrations. You describe an outcome. It breaks it into tasks and subtasks, creates sub-agents for each, and coordinates them automatically. When a sub-agent hits a problem, it spawns more sub-agents to solve it. And it connects to your existing stack — GitHub, Google Drive, Gmail, Slack, Jira, Linear, Notion, Confluence, Ahrefs, Airtable, and more. Critically, it doesn't just run once. It can run on a schedule. Reading your docs, checking your project boards, pulling from your CRM, and acting on what it finds. Market monitoring. Competitor tracking. Weekly reports with charts. Content pipelines. CRON jobs that actually execute. Not "AI that helps you once." AI that runs in the background for days or months. Think of it as managed OpenClaw — similar autonomous capability (scheduled tasks, multi-step workflows, tool integrations) but fully managed. No Mac Mini. No security config. No infrastructure to maintain. I tested it with a complex prompt — a full stock trading simulator with what-if scenarios, correlation heatmaps, sentiment analysis, and a Bloomberg Terminal aesthetic. Two prompts later: deployed to Netlify via GitHub, with working CRON jobs updating live data. I've started using it to analyze my portfolio. But coding is just one lane. This thing researches, writes reports, generates datasets, creates videos, processes documents, and connects to your existing tools — all in one coordinated workflow. The real shift: you don't choose a model anymore. You describe what you need. The system routes each piece of work to whichever model does it best — and spawns new agents when it hits a wall. 19 models, dynamic sub-agents, scheduled tasks, and your entire tool stack connected. Thoughts?
0:37

JUST IN: Perplexity launched "Perplexity Computer" — and it might be the most complete AI agent system available right now. Not a chatbot upgrade. Not a research tool with a new name. A system that plans entire projects, delegates to specialist AI models, and runs autonomously for hours, days, or months (their words). Here's what makes the architecture genuinely different: → Opus 4.6 handles core reasoning and orchestration → Gemini handles deep research (spawning its own sub-agents) → Grok handles lightweight speed tasks → Veo 3.1 handles video generation → Nano Banana handles image creation → ChatGPT 5.2 handles long-context recall and wide search → You can override model choices per subtask 19 models total. Each task runs in an isolated environment with a real filesystem, real browser, and real tool integrations. You describe an outcome. It breaks it into tasks and subtasks, creates sub-agents for each, and coordinates them automatically. When a sub-agent hits a problem, it spawns more sub-agents to solve it. And it connects to your existing stack — GitHub, Google Drive, Gmail, Slack, Jira, Linear, Notion, Confluence, Ahrefs, Airtable, and more. Critically, it doesn't just run once. It can run on a schedule. Reading your docs, checking your project boards, pulling from your CRM, and acting on what it finds. Market monitoring. Competitor tracking. Weekly reports with charts. Content pipelines. CRON jobs that actually execute. Not "AI that helps you once." AI that runs in the background for days or months. Think of it as managed OpenClaw — similar autonomous capability (scheduled tasks, multi-step workflows, tool integrations) but fully managed. No Mac Mini. No security config. No infrastructure to maintain. I tested it with a complex prompt — a full stock trading simulator with what-if scenarios, correlation heatmaps, sentiment analysis, and a Bloomberg Terminal aesthetic. Two prompts later: deployed to Netlify via GitHub, with working CRON jobs updating live data. I've started using it to analyze my portfolio. But coding is just one lane. This thing researches, writes reports, generates datasets, creates videos, processes documents, and connects to your existing tools — all in one coordinated workflow. The real shift: you don't choose a model anymore. You describe what you need. The system routes each piece of work to whichever model does it best — and spawns new agents when it hits a wall. 19 models, dynamic sub-agents, scheduled tasks, and your entire tool stack connected. Thoughts?

Paweł Huryn

219,681 Aufrufe • vor 6 Monaten

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:
0:58

New model: your robot can now pack your suitcase 🧳 Xiaomi has released a new robot foundation model. Called Xiaomi-Robotics-1, it is designed to have a robot pick things up and move them around. But first, DEFINITIONS: - Mixture-of-Transformers (MoT): An architecture where separate transformer "experts" (e.g., one for vision-language, one for actions) share a single attention stream, so each modality gets specialized parameters without losing joint reasoning. - Vision-language model (VLM): A model that jointly understands images and text. - Diffusion transformer: A transformer trained to turn noise into structured outputs by iterative denoising, here generating robot actions rather than images. - Action chunks: Short sequences of future actions (e.g., the next ~50 motor commands) predicted in one shot instead of one step at a time. - Flow matching: A faster version of diffusion. The model learns a straight-line velocity field from noise to the target action, so it needs only a few integration steps instead of many denoising ones. Its peculiarity comes from its two stage training: 1. 100,000 hours of video shot through a UMI rig: a handheld 3D-printed gripper with a camera, worn by humans doing ordinary tasks in homes, shops, factories and offices. 2. Adapt to actual robot bodies with ~10,000 hours of real-robot data. It replaces the standard approach of teleoperating a real robot for every hour of training data. Its architecture is a Mixture-of-Transformers pairing a pre-trained Qwen3-VL vision-language model with a diffusion transformer that emits action chunks via flow matching, released in 2.6B, 5.1B and 10.5B parameter variants. However, if you read the entire paper ("Scaling VLA Models with over 100K Hours"), you realize that all of the scaling experiments on 20k hours. Therefore the headline "out-of-the-box success climbing 26% → 75% as pre-training data grows" tops out at 100% of 20k hours! What the full corpus does to that curve is never shown -> and this where things would become interesting! Xiaomi's own conclusion is that model size has stopped mattering and data is the binding constraint. The performance gap among different model sizes are less pronounced than those observed across different data scales. This result suggests that model capacity at the billions-parameter scale may already be sufficient to capture the current dataset's distribution. Which further asks the same question: why not use the 100k video hours? Anyway, I would definitely love to have a couple robots at home that can cooperate to pack my suitcase with items relevant to my next destination:

Léo

15,662 Aufrufe • vor 23 Tagen

This Chinese guy created agents in Claude Code for landing pages and single-handedly serves 47 small businesses a month, taking $400 from each. He built a system of 7 agents on Claude Sonnet 4.6 that analyzes Google Maps in small towns, finds small businesses without websites there, and over 1 weekend takes each one to a finished mockup with video and cold message. No assistant, no sales team, no SDR. Just him, a MacBook, an iPhone, and 1 API key. And traditional web design agencies keep teams of 8 people on salary for the same order flow, while his expenses are only tokens and subscriptions to Lovable, Higgsfield, and Calendly. 7 agents work through 1 orchestrator on Claude Code Router. Usage is about 3 million tokens a day, the average API bill is about $480 a month. All 7 go through MCP servers and write shared state to the file system, without shared state in memory and without race conditions, and 1 of them lives right in the iPhone and picks up positive replies from the subway, a taxi, or on walks. And here is the system prompt he put into the orchestrator before launch: "You are the orchestrator of a solo agency that sells ready-made websites to local businesses. You delegate read-only tasks to 6 sub-agents and own all writes. sub-agents: // Scout (walks through Google Maps in selected cities, looks for narrow niches: 5+ years on the map, fewer than 50 reviews, no website or a website from 2014, but high ratings) // Diagnoser (for each lead writes a 50-word diagnosis, hero angle, tone matched to the industry, and a cold message under 70 words) // Builder (generates a landing page mockup in Lovable through MCP only for the top 5 leads per day, with the sharpest diagnoses and the biggest gap) // Filmer (pulls 5 screenshots of the mockup and through Higgsfield renders a 10-second vertical video 1080x1920 with a soft zoom) // Pitcher (sends a personalized cold message through the right channel for the niche: email to roofers, SMS to tradesmen, IG DM to salons, LinkedIn to realtors) // Checker (runs every message through evals for personalization, absence of AI markers and buzzwords before sending) // Mobile (lives in the iPhone, handles positive replies in real time, books Zoom calls in Calendly through MCP while the owner is on the go). You never let 2 sub-agents touch 1 lead. You stop and request approval from the human only when a deal exceeds $3,000 or the reply rate in a niche for the day drops below 12%." Meaning the system knows what it is and within what boundaries it is allowed to act. It knows it is supposed to find leads on its own. It knows it is supposed to take each one to a mockup, video, and cold message without intervention. It knows the human only steps in when a deal goes above $3,000 or the reply rate stops converging. → The system runs 24 hours a day → Scout goes through about 220 local businesses on Google Maps per day and leaves 30 new leads in the queue → Diagnoser outputs 30 structured diagnoses + briefs + cold messages per day → Builder assembles 3 to 5 finished landing pages in Lovable for the sharpest leads → Filmer renders a 10-second vertical video in Higgsfield for each one → Pitcher sends 30 personalized messages per day across 4 channels with a reply rate of about 14% → Checker runs every message through evals before sending And only when a deal breaks $3,000 or the reply rate for the day drops below 12% does the orchestrator wake the owner. And when the owner at that moment is sitting in the subway or a taxi, the Mobile agent in his iPhone picks up 1 move on its own: replies to a fresh positive reply from a dentist, books a Zoom through Calendly synced to the local time of the client, and puts the lead back in the queue. The owner only has to tap "approve" and in just 10 minutes join the call. Here is what the system writes in his log during 1 of the Saturdays: "scout report: 218 businesses checked in Austin, Denver, and Miami, 34 without a website, 19 with a website from 2014, 6 with an active redesign request in reviews. passing top 30 to diagnoser." "pitcher: 30 cold messages sent across 4 channels, 14 replies, 5 positive, 3 Zoom calls booked for Sunday. passing to closer." "builder: landing page for Westside Cosmetic Dentistry built in Lovable, 5 sections, mobile, soft beige. URL placed at /Users/dev/maps-agency/clients/westside/v1. filmer launching Higgsfield." "eval flag: deal with The Lotus Salon at $3,400 exceeds the approved limit of $3,000. sending for manual review." He has no server of his own and no separate backend. Just a local file sandbox at /Users/dev/maps-agency, an MCP router, 1 API key to Claude, and the same key forwarded to Claude Code on his iPhone. Out of everything I have seen this year, this is the cleanest one-person agency for selling websites to small businesses: $480 a month on the API, about $18,800 into the account, and between them 7 prompts, 1 file system, and 1 phone in the pocket.
2:02

This Chinese guy created agents in Claude Code for landing pages and single-handedly serves 47 small businesses a month, taking $400 from each. He built a system of 7 agents on Claude Sonnet 4.6 that analyzes Google Maps in small towns, finds small businesses without websites there, and over 1 weekend takes each one to a finished mockup with video and cold message. No assistant, no sales team, no SDR. Just him, a MacBook, an iPhone, and 1 API key. And traditional web design agencies keep teams of 8 people on salary for the same order flow, while his expenses are only tokens and subscriptions to Lovable, Higgsfield, and Calendly. 7 agents work through 1 orchestrator on Claude Code Router. Usage is about 3 million tokens a day, the average API bill is about $480 a month. All 7 go through MCP servers and write shared state to the file system, without shared state in memory and without race conditions, and 1 of them lives right in the iPhone and picks up positive replies from the subway, a taxi, or on walks. And here is the system prompt he put into the orchestrator before launch: "You are the orchestrator of a solo agency that sells ready-made websites to local businesses. You delegate read-only tasks to 6 sub-agents and own all writes. sub-agents: // Scout (walks through Google Maps in selected cities, looks for narrow niches: 5+ years on the map, fewer than 50 reviews, no website or a website from 2014, but high ratings) // Diagnoser (for each lead writes a 50-word diagnosis, hero angle, tone matched to the industry, and a cold message under 70 words) // Builder (generates a landing page mockup in Lovable through MCP only for the top 5 leads per day, with the sharpest diagnoses and the biggest gap) // Filmer (pulls 5 screenshots of the mockup and through Higgsfield renders a 10-second vertical video 1080x1920 with a soft zoom) // Pitcher (sends a personalized cold message through the right channel for the niche: email to roofers, SMS to tradesmen, IG DM to salons, LinkedIn to realtors) // Checker (runs every message through evals for personalization, absence of AI markers and buzzwords before sending) // Mobile (lives in the iPhone, handles positive replies in real time, books Zoom calls in Calendly through MCP while the owner is on the go). You never let 2 sub-agents touch 1 lead. You stop and request approval from the human only when a deal exceeds $3,000 or the reply rate in a niche for the day drops below 12%." Meaning the system knows what it is and within what boundaries it is allowed to act. It knows it is supposed to find leads on its own. It knows it is supposed to take each one to a mockup, video, and cold message without intervention. It knows the human only steps in when a deal goes above $3,000 or the reply rate stops converging. → The system runs 24 hours a day → Scout goes through about 220 local businesses on Google Maps per day and leaves 30 new leads in the queue → Diagnoser outputs 30 structured diagnoses + briefs + cold messages per day → Builder assembles 3 to 5 finished landing pages in Lovable for the sharpest leads → Filmer renders a 10-second vertical video in Higgsfield for each one → Pitcher sends 30 personalized messages per day across 4 channels with a reply rate of about 14% → Checker runs every message through evals before sending And only when a deal breaks $3,000 or the reply rate for the day drops below 12% does the orchestrator wake the owner. And when the owner at that moment is sitting in the subway or a taxi, the Mobile agent in his iPhone picks up 1 move on its own: replies to a fresh positive reply from a dentist, books a Zoom through Calendly synced to the local time of the client, and puts the lead back in the queue. The owner only has to tap "approve" and in just 10 minutes join the call. Here is what the system writes in his log during 1 of the Saturdays: "scout report: 218 businesses checked in Austin, Denver, and Miami, 34 without a website, 19 with a website from 2014, 6 with an active redesign request in reviews. passing top 30 to diagnoser." "pitcher: 30 cold messages sent across 4 channels, 14 replies, 5 positive, 3 Zoom calls booked for Sunday. passing to closer." "builder: landing page for Westside Cosmetic Dentistry built in Lovable, 5 sections, mobile, soft beige. URL placed at /Users/dev/maps-agency/clients/westside/v1. filmer launching Higgsfield." "eval flag: deal with The Lotus Salon at $3,400 exceeds the approved limit of $3,000. sending for manual review." He has no server of his own and no separate backend. Just a local file sandbox at /Users/dev/maps-agency, an MCP router, 1 API key to Claude, and the same key forwarded to Claude Code on his iPhone. Out of everything I have seen this year, this is the cleanest one-person agency for selling websites to small businesses: $480 a month on the API, about $18,800 into the account, and between them 7 prompts, 1 file system, and 1 phone in the pocket.

Blaze

2,716,136 Aufrufe • vor 3 Monaten

The power of the Claw, in the palm of a robot hand. Agentic robotics is here! Today, we open-source CaP-X: vibe agents, alive in the physical world. They incarnate as robot arms and humanoids with a rich set of perception APIs, actuation APIs, and auto synthesize skill libraries as they go. CaP-X is a strict superset of our old stack, because policies like VLAs are “just” API calls as well. It solves many tasks zero-shot that a learned policy would struggle with. And we are doing much more than vibing. CaP-X is our most systematic, scientific study on agentic robotics so far: - We build a comprehensive agentic toolkit: perception (SAM3 segmentation, Molmo pointing, depth, point cloud), control (IK solvers, grasp planner, navigation), and visualization (EEF, mask overlays) that work across different robots. - CaP-Gym: LLM’s first Physical Exam! 187 manipulation tasks across RoboSuite, LIBERO-PRO, and BEHAVIOR. Tabletop, bimanual, mobile manipulation. Sim and real. Can’t wait to see the gradients flow from CaP-Gym to the next wave of frontier LLM releases. - CaP-Bench: we benchmark 12 frontier LLMs/VLMs (Gemini, GPT, Opus, Qwen, DeepSeek, Kimi, and more) across 8 evaluation tiers. We systematically vary API abstraction level, agentic harness, and visual grounding methods. Lots of insights in our paper. - CaP-Agent0: a training-free agentic harness that matches or exceeds human expert code on 4 out of 7 tasks without task-specific tuning. - CaP-RL: if you get a gym, you get RL ;). A 7B OSS model jumps from 20% to 72% success after only 50 training iterations. The synthesized programs transfer to real robots with minimal sim-to-real gap. 3 years ago, our team created Voyager, one of the earliest agentic AI that plays and learns in Minecraft continuously. Its key ideas — skill libraries, self-reflection loops, and in-context planning — have since influenced many modern agentic designs. Today, the agent graduates from Minecraft and gets a real job. It’s April Fool’s, but this Claw is getting its hands dirty for real! Link in thread:
1:31

The power of the Claw, in the palm of a robot hand. Agentic robotics is here! Today, we open-source CaP-X: vibe agents, alive in the physical world. They incarnate as robot arms and humanoids with a rich set of perception APIs, actuation APIs, and auto synthesize skill libraries as they go. CaP-X is a strict superset of our old stack, because policies like VLAs are “just” API calls as well. It solves many tasks zero-shot that a learned policy would struggle with. And we are doing much more than vibing. CaP-X is our most systematic, scientific study on agentic robotics so far: - We build a comprehensive agentic toolkit: perception (SAM3 segmentation, Molmo pointing, depth, point cloud), control (IK solvers, grasp planner, navigation), and visualization (EEF, mask overlays) that work across different robots. - CaP-Gym: LLM’s first Physical Exam! 187 manipulation tasks across RoboSuite, LIBERO-PRO, and BEHAVIOR. Tabletop, bimanual, mobile manipulation. Sim and real. Can’t wait to see the gradients flow from CaP-Gym to the next wave of frontier LLM releases. - CaP-Bench: we benchmark 12 frontier LLMs/VLMs (Gemini, GPT, Opus, Qwen, DeepSeek, Kimi, and more) across 8 evaluation tiers. We systematically vary API abstraction level, agentic harness, and visual grounding methods. Lots of insights in our paper. - CaP-Agent0: a training-free agentic harness that matches or exceeds human expert code on 4 out of 7 tasks without task-specific tuning. - CaP-RL: if you get a gym, you get RL ;). A 7B OSS model jumps from 20% to 72% success after only 50 training iterations. The synthesized programs transfer to real robots with minimal sim-to-real gap. 3 years ago, our team created Voyager, one of the earliest agentic AI that plays and learns in Minecraft continuously. Its key ideas — skill libraries, self-reflection loops, and in-context planning — have since influenced many modern agentic designs. Today, the agent graduates from Minecraft and gets a real job. It’s April Fool’s, but this Claw is getting its hands dirty for real! Link in thread:

Jim Fan

81,638 Aufrufe • vor 4 Monaten

NEW RESEARCH: You can now create a new robot optimized for any given task! I love this new project by Huy Ha, Shuran Song, and others. Called "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design", it generates a robot's physical design and its controller together from a task spec. DEFINITIONS: - Reward function: A scoring rule that assigns a number to how well a behavior achieves the task. Here, it is the objective the generated design is pushed to maximize (e.g., track the target motion with low error). - Tokenizing: dividing continuous or structured data (a robot's links, joints, motor specs, states, actions) into a discrete vocabulary of symbols a transformer can process, the same step that turned pixels and audio into "language" for these models. - Diffusion transformer (DiT): A transformer trained to turn random noise into structured output through iterative denoising. Here, it generates robot bodies and trajectories instead of images. - MuJoCo: The standard fast physics simulator for robotics research (DeepMind-maintained). The Menagerie is its curated zoo of ready-to-use robot models. - CMA-ES: Covariance Matrix Adaptation Evolution Strategy, the workhorse black-box optimizer: it evolves a population of candidate designs, keeps the best, and needs thousands of simulator rollouts. - Bimanual multi-trajectory optimization: Finding one design/controller that performs well across several target motions for a two-armed robot at once, harder than optimizing for a single arm and a single motion. - BERT/MAE masked-modeling trick: Train one model to fill in whatever parts of the input you hide (words for BERT, image patches for MAE); at inference, choosing what to mask chooses the task, so masking the body makes it a designer and masking the actions makes it a controller. In practice, you give it a target end-effector motion and a reward function, and it outputs a complete embodiment (link, joint, motor, and inertial property), as well as a controller to drive it. It works by tokenizing both the body (links/joints/motors) and the dynamics (states/actions) into a compact scheme called RoboTokens, training a diffusion transformer (DiT) over them. The same model predicts dynamics using those predictions ("Dynamics Self-Guidance") to push generated designs toward higher reward at inference time. Masking different token types (using the BERT/MAE masked-modeling trick) lets the one model do three jobs: generate an embodiment, control an arbitrary embodiment, or design one conditioned on a motion. It is trained on 11 robots from the MuJoCo Menagerie (0.65 kg hand to 67.5 kg quadruped, 6–35 joints), and validated in sim and on a physical ALOHA doing cloth flinging. I like the fact that this approach inverts the entire recent robotics ideas: designing a policy for a fixed robot -> designing the robot for a fixed task. Every other approach assumes the body is given and learns a controller. Transformer Transformer takes the task (target motion + reward), then generates the body and controller jointly. In practice, it is a ~180× speedup over the standard optimizer at equal-or-better quality. It reaches "CMA-ES-level quality in seconds" and finishes bimanual multi-trajectory optimization in that is worth underlining nowadays! Also worth mentioning: this is the lab behind UMI and Handroid, that I mentioned here previously! The team seems extremely creative, i love these out-of-the-box approaches. Enjoy watching the demo of robot optimization in 3D, data acquisition, then real-life testing:
0:28

NEW RESEARCH: You can now create a new robot optimized for any given task! I love this new project by Huy Ha, Shuran Song, and others. Called "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design", it generates a robot's physical design and its controller together from a task spec. DEFINITIONS: - Reward function: A scoring rule that assigns a number to how well a behavior achieves the task. Here, it is the objective the generated design is pushed to maximize (e.g., track the target motion with low error). - Tokenizing: dividing continuous or structured data (a robot's links, joints, motor specs, states, actions) into a discrete vocabulary of symbols a transformer can process, the same step that turned pixels and audio into "language" for these models. - Diffusion transformer (DiT): A transformer trained to turn random noise into structured output through iterative denoising. Here, it generates robot bodies and trajectories instead of images. - MuJoCo: The standard fast physics simulator for robotics research (DeepMind-maintained). The Menagerie is its curated zoo of ready-to-use robot models. - CMA-ES: Covariance Matrix Adaptation Evolution Strategy, the workhorse black-box optimizer: it evolves a population of candidate designs, keeps the best, and needs thousands of simulator rollouts. - Bimanual multi-trajectory optimization: Finding one design/controller that performs well across several target motions for a two-armed robot at once, harder than optimizing for a single arm and a single motion. - BERT/MAE masked-modeling trick: Train one model to fill in whatever parts of the input you hide (words for BERT, image patches for MAE); at inference, choosing what to mask chooses the task, so masking the body makes it a designer and masking the actions makes it a controller. In practice, you give it a target end-effector motion and a reward function, and it outputs a complete embodiment (link, joint, motor, and inertial property), as well as a controller to drive it. It works by tokenizing both the body (links/joints/motors) and the dynamics (states/actions) into a compact scheme called RoboTokens, training a diffusion transformer (DiT) over them. The same model predicts dynamics using those predictions ("Dynamics Self-Guidance") to push generated designs toward higher reward at inference time. Masking different token types (using the BERT/MAE masked-modeling trick) lets the one model do three jobs: generate an embodiment, control an arbitrary embodiment, or design one conditioned on a motion. It is trained on 11 robots from the MuJoCo Menagerie (0.65 kg hand to 67.5 kg quadruped, 6–35 joints), and validated in sim and on a physical ALOHA doing cloth flinging. I like the fact that this approach inverts the entire recent robotics ideas: designing a policy for a fixed robot -> designing the robot for a fixed task. Every other approach assumes the body is given and learns a controller. Transformer Transformer takes the task (target motion + reward), then generates the body and controller jointly. In practice, it is a ~180× speedup over the standard optimizer at equal-or-better quality. It reaches "CMA-ES-level quality in seconds" and finishes bimanual multi-trajectory optimization in that is worth underlining nowadays! Also worth mentioning: this is the lab behind UMI and Handroid, that I mentioned here previously! The team seems extremely creative, i love these out-of-the-box approaches. Enjoy watching the demo of robot optimization in 3D, data acquisition, then real-life testing:

Léo

26,186 Aufrufe • vor 22 Tagen

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!
0:27

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Akshay 🚀

46,435 Aufrufe • vor 9 Monaten

This Chinese guy created agents in Claude Code for MCP servers and single-handedly serves 6 marketing agencies a month from one iPhone, earning $5,000 from each. Inside he runs a pipeline of 7 agents on Claude Sonnet 4.6 that every Monday pulls a scan of the tech stack from a selected agency, develops an MCP server for its ad accounts, and over the course of a week brings it to production code ready to connect to Claude Desktop. No DevOps, no senior developer, no project manager. Just a Mac Mini in a work corner, an iPhone in the pocket, and a single API key. And traditional dev shops keep 5 people on project rates for the same contract, while his entire P&L is tokens, dirt-cheap hosting on Cloudflare, and Calendly. 7 agents run under a shared orchestrator-router and burn about 5 million tokens a day, which in the API bill comes out to $540 a month. The Mac Mini itself sits at home and keeps the entire orchestrator running 24/7, and from the iPhone the owner connects to it through a secure remote terminal and sees the output of any session right on the smartphone screen, wherever he happens to be. His starting system prompt looks like this: "you run a solo shop for custom MCP servers for marketing agencies. you hand out read-only tasks to 6 sub-agents and own all commits and shipping yourself. sub-agents: // Hunter (finds marketing agencies of 15 to 60 people that have no MCP access to Google Ads, Meta Ads, TikTok Ads, and HubSpot) // Mapper (pulls their tech stack, identifies 3 to 5 integration pains, and simultaneously writes the technical spec for the server: which tools, resources, and prompts to export through MCP, which auth flow and rate limit) // Coder (generates an MCP server in Python through the MCP SDK, deploys 8 to 15 tools for ad accounts and CRM) // Validator (connects the server to Claude Desktop, runs real client API keys in a sandbox, and checks for compliance with the MCP spec) // Shipper (writes a README, integration guide, deployment manual, packages the server, and hosts it on Cloudflare Workers or pushes to the GitHub of the client) // Mobile (always online on the iPhone, books demo calls in Calendly, picks up hot fixes, and confirms contracts through a secure remote terminal to the Mac Mini). only 1 owner agent works on 1 contract, no overlaps. you pull the owner out of observation mode only when a deal goes above $7,500 or the test coverage of the server drops below 85%." This prompt gives the system an understanding of its role and the limits of intervention from the very first line. It knows it is supposed to find agencies on its own. It knows it is supposed to bring every MCP server to production on its own. It knows it connects the live owner only on large deals or when the tests do not converge. → The pipeline runs without breaks, day or night → Hunter goes through about 130 marketing agencies on LinkedIn and Clutch per day → Mapper rolls out 4 audit reports with the tech stack and a final spec for each → Coder writes 1 to 2 MCP servers per week in Python with 8 to 15 tools → Validator validates every server through Claude Desktop with real client API keys → Shipper rolls out the full documentation package and pushes the finished product to Cloudflare Workers or the GitHub of the client And only when a contract breaks $7,500 or test coverage drops below 85% does the orchestrator pull the owner from whatever he is doing. And when the owner at that moment is behind the wheel or at a meeting in a coworking space, the Mobile agent in his iPhone picks up 1 contract in progress: confirms a meeting with the agency CMO in Calendly, opens a live demo of the MCP server through a secure terminal to the Mac Mini, and writes the test result to the shared state. The owner just swipes "approve" and in 15 minutes joins the Zoom demo. The fresh system log from last Wednesday looks like this: "hunter report: 132 agencies checked on LinkedIn and Clutch, 19 without MCP integrations, 8 with active requests for AI tooling in job posts, 4 with an open Q4 budget. passing to mapper." "coder: MCP server for Northwave Performance Marketing built in Python, 11 tools for Google Ads, Meta Ads, and GA4, 320 lines of code. exported to /Users/dev/mcp-shop/clients/northwave/server.py. validator connecting to Claude Desktop." "validator: 11 tools passed validation through Claude Desktop, test coverage 92%, average latency 380 ms. passing to shipper." "eval flag: contract with Pacific Reach Agency at $8,200 exceeds the approved limit of $7,500. sending for manual review." In his work setup there is no cloud server, no external team, and not even a separate office. At home sits a Mac Mini with a sandbox at /Users/dev/mcp-shop, on top runs an MCP router with a single API key to Claude, and the same key is forwarded to a secure terminal on the iPhone. Out of everything I have seen this year, this is the cleanest solo shop for custom MCP servers for marketing agencies: $540 a month on the API, about $30,000 into the account, and between them 7 system prompts, 1 Mac Mini in a work corner, and 1 iPhone that never leaves the pocket.
0:33

This Chinese guy created agents in Claude Code for MCP servers and single-handedly serves 6 marketing agencies a month from one iPhone, earning $5,000 from each. Inside he runs a pipeline of 7 agents on Claude Sonnet 4.6 that every Monday pulls a scan of the tech stack from a selected agency, develops an MCP server for its ad accounts, and over the course of a week brings it to production code ready to connect to Claude Desktop. No DevOps, no senior developer, no project manager. Just a Mac Mini in a work corner, an iPhone in the pocket, and a single API key. And traditional dev shops keep 5 people on project rates for the same contract, while his entire P&L is tokens, dirt-cheap hosting on Cloudflare, and Calendly. 7 agents run under a shared orchestrator-router and burn about 5 million tokens a day, which in the API bill comes out to $540 a month. The Mac Mini itself sits at home and keeps the entire orchestrator running 24/7, and from the iPhone the owner connects to it through a secure remote terminal and sees the output of any session right on the smartphone screen, wherever he happens to be. His starting system prompt looks like this: "you run a solo shop for custom MCP servers for marketing agencies. you hand out read-only tasks to 6 sub-agents and own all commits and shipping yourself. sub-agents: // Hunter (finds marketing agencies of 15 to 60 people that have no MCP access to Google Ads, Meta Ads, TikTok Ads, and HubSpot) // Mapper (pulls their tech stack, identifies 3 to 5 integration pains, and simultaneously writes the technical spec for the server: which tools, resources, and prompts to export through MCP, which auth flow and rate limit) // Coder (generates an MCP server in Python through the MCP SDK, deploys 8 to 15 tools for ad accounts and CRM) // Validator (connects the server to Claude Desktop, runs real client API keys in a sandbox, and checks for compliance with the MCP spec) // Shipper (writes a README, integration guide, deployment manual, packages the server, and hosts it on Cloudflare Workers or pushes to the GitHub of the client) // Mobile (always online on the iPhone, books demo calls in Calendly, picks up hot fixes, and confirms contracts through a secure remote terminal to the Mac Mini). only 1 owner agent works on 1 contract, no overlaps. you pull the owner out of observation mode only when a deal goes above $7,500 or the test coverage of the server drops below 85%." This prompt gives the system an understanding of its role and the limits of intervention from the very first line. It knows it is supposed to find agencies on its own. It knows it is supposed to bring every MCP server to production on its own. It knows it connects the live owner only on large deals or when the tests do not converge. → The pipeline runs without breaks, day or night → Hunter goes through about 130 marketing agencies on LinkedIn and Clutch per day → Mapper rolls out 4 audit reports with the tech stack and a final spec for each → Coder writes 1 to 2 MCP servers per week in Python with 8 to 15 tools → Validator validates every server through Claude Desktop with real client API keys → Shipper rolls out the full documentation package and pushes the finished product to Cloudflare Workers or the GitHub of the client And only when a contract breaks $7,500 or test coverage drops below 85% does the orchestrator pull the owner from whatever he is doing. And when the owner at that moment is behind the wheel or at a meeting in a coworking space, the Mobile agent in his iPhone picks up 1 contract in progress: confirms a meeting with the agency CMO in Calendly, opens a live demo of the MCP server through a secure terminal to the Mac Mini, and writes the test result to the shared state. The owner just swipes "approve" and in 15 minutes joins the Zoom demo. The fresh system log from last Wednesday looks like this: "hunter report: 132 agencies checked on LinkedIn and Clutch, 19 without MCP integrations, 8 with active requests for AI tooling in job posts, 4 with an open Q4 budget. passing to mapper." "coder: MCP server for Northwave Performance Marketing built in Python, 11 tools for Google Ads, Meta Ads, and GA4, 320 lines of code. exported to /Users/dev/mcp-shop/clients/northwave/server.py. validator connecting to Claude Desktop." "validator: 11 tools passed validation through Claude Desktop, test coverage 92%, average latency 380 ms. passing to shipper." "eval flag: contract with Pacific Reach Agency at $8,200 exceeds the approved limit of $7,500. sending for manual review." In his work setup there is no cloud server, no external team, and not even a separate office. At home sits a Mac Mini with a sandbox at /Users/dev/mcp-shop, on top runs an MCP router with a single API key to Claude, and the same key is forwarded to a secure terminal on the iPhone. Out of everything I have seen this year, this is the cleanest solo shop for custom MCP servers for marketing agencies: $540 a month on the API, about $30,000 into the account, and between them 7 system prompts, 1 Mac Mini in a work corner, and 1 iPhone that never leaves the pocket.

Blaze

55,926 Aufrufe • vor 3 Monaten

Video Walkthrough of My Daily Process: How I merge my Finviz screener, TradingView watchlists, and a 'Compression' screener to generate stalk & focused ideas. Here’s a quick walkthrough of how I generate my stalk/focused ideas—also shared exclusively with my X subscribers through a daily pre-market tweet condensed into a 5-minute reference. A breakdown of the process; 1. Tradingview as my based charting and watchlist management platform. It is tile next to my finviz web browser. 2. I have 13 preset screeners across both platforms , 9 in finviz (post-market to watchlist), 4 in tradingview (watchlist compression, pre-market gapper of stock & etf, watchlist RVOL sorted). Details of each screener are shared in Chapter 3 of You can also get direct Shared Screen access from 3. I copy each screened result from Finviz and paste it into its corresponding TradingView watchlist (e.g., “Hottest Stock” results go into the “Hottest Stock” watchlist). Erik Carell has built a Finviz API workaround that lets you import an entire screen directly into a TradingView watchlist. 4. Screened results aren’t usually actionable on their own, so I add an extra layer— “compression” screener within TradingView—and run it through each dedicated watchlist. This is what I refer to as a “screen within a screen.” My watchlists are color-coded to show which screener each stock came from—and to highlight when a name appears across multiple screeners (e.g.,🔴= Hottest Stock). 5. I review each name that passes the “compression” screener, evaluating them one by one on the chart to determine whether they qualify for my stalk/focused idea watchlist. The criteria I use are outlined in my “15 Hard Rules” in Chapter 6. 6. The same process is then applied at the ETF level, since TradingView separates its Stock and ETF screeners into two different sections. 7. On top of that, I manually review over 160 ETFs to track day-by-day price action/RS across industry groups (not shown in the video). The full workflow—including post-market study—takes at least 2 hours per session. The process flows as follows: Screening → Watchlist Management → Focus List Rebuild & Preparation → Qualitative Market Reading for Situational Awareness → Portfolio Stop Management (when needed). No single screener will ever capture every opportunity. To stay ahead of the market, you need unwavering dedication, discipline, and consistency. Eventually, the market rewards that effort with the strong, or trending moves. But first, you need a strategy and process that fits your lifestyle and is sustainable over the long term. I hope you all find this helpful as we navigate this challenging yet financially rewarding journey.
10:19

Video Walkthrough of My Daily Process: How I merge my Finviz screener, TradingView watchlists, and a 'Compression' screener to generate stalk & focused ideas. Here’s a quick walkthrough of how I generate my stalk/focused ideas—also shared exclusively with my X subscribers through a daily pre-market tweet condensed into a 5-minute reference. A breakdown of the process; 1. Tradingview as my based charting and watchlist management platform. It is tile next to my finviz web browser. 2. I have 13 preset screeners across both platforms , 9 in finviz (post-market to watchlist), 4 in tradingview (watchlist compression, pre-market gapper of stock & etf, watchlist RVOL sorted). Details of each screener are shared in Chapter 3 of You can also get direct Shared Screen access from 3. I copy each screened result from Finviz and paste it into its corresponding TradingView watchlist (e.g., “Hottest Stock” results go into the “Hottest Stock” watchlist). Erik Carell has built a Finviz API workaround that lets you import an entire screen directly into a TradingView watchlist. 4. Screened results aren’t usually actionable on their own, so I add an extra layer— “compression” screener within TradingView—and run it through each dedicated watchlist. This is what I refer to as a “screen within a screen.” My watchlists are color-coded to show which screener each stock came from—and to highlight when a name appears across multiple screeners (e.g.,🔴= Hottest Stock). 5. I review each name that passes the “compression” screener, evaluating them one by one on the chart to determine whether they qualify for my stalk/focused idea watchlist. The criteria I use are outlined in my “15 Hard Rules” in Chapter 6. 6. The same process is then applied at the ETF level, since TradingView separates its Stock and ETF screeners into two different sections. 7. On top of that, I manually review over 160 ETFs to track day-by-day price action/RS across industry groups (not shown in the video). The full workflow—including post-market study—takes at least 2 hours per session. The process flows as follows: Screening → Watchlist Management → Focus List Rebuild & Preparation → Qualitative Market Reading for Situational Awareness → Portfolio Stop Management (when needed). No single screener will ever capture every opportunity. To stay ahead of the market, you need unwavering dedication, discipline, and consistency. Eventually, the market rewards that effort with the strong, or trending moves. But first, you need a strategy and process that fits your lifestyle and is sustainable over the long term. I hope you all find this helpful as we navigate this challenging yet financially rewarding journey.

Jeff Sun, CFTe

381,589 Aufrufe • vor 8 Monaten

The selloff in Micron is one of the best buying opportunities you'll see this year (Save this). Sanjay Mehrotra just explained exactly why the old mental model for Micron, cyclical, commodity, mean reverting no longer applies. Every AI system, regardless of what device it runs on, requires more memory at higher performance to unlock its full potential. From data centers to smartphones to autonomous vehicles, memory is no longer a supporting actor but rather the critical bottleneck determining how fast AI can move. What makes this cycle structurally different starts with what happened in 2023. Certain customers drove industry pricing to one third of 2022 levels, forcing Micron into severe losses while still requiring $10 billion in investment just to stay competitive. Most companies in that situation cut spending and survive but Micron invested through the pain with the vision that the other side would be worth it. Those 2023 investments are now producing 84.9% gross margins, $41.46 billion in quarterly revenue, and Q4 guidance of $50 billion up from $11.3 billion in the same quarter just one year ago. That is what it looks like when a company bets on itself at exactly the right moment. Even Micron's own largest customers, Nvidia, Google, Amazon could not forecast the scale of AI memory demand that materialized. When the biggest technology companies in the world cannot project their own memory requirements, you are watching a structural transformation that nobody had models to predict, still in its early innings. Supply cannot respond quickly enough to close that gap. Mehrotra confirmed on air that tightness extends beyond 2027, new domestic fabs take years to bring online, and new HBM capacity which requires advanced 3D stacking that compounds in complexity at every generation won't meaningfully arrive until late 2028. There is no fast fix to a shortage of the most valuable memory on earth. The strategic customer agreements are the most underappreciated part of the entire story. Multi-year contracts with volume commitments and price floors now cover roughly 20% of DRAM volume and 30% of NAND volume, locking in a $100 billion contractual revenue base. The old Micron was at the mercy of customers who could crater prices overnight while the new Micron has contractual floors that make the 2023 scenario structurally impossible to repeat. Long Micron and make sure to follow me Melvin for more deep dives into AI and memory.
6:21

The selloff in Micron is one of the best buying opportunities you'll see this year (Save this). Sanjay Mehrotra just explained exactly why the old mental model for Micron, cyclical, commodity, mean reverting no longer applies. Every AI system, regardless of what device it runs on, requires more memory at higher performance to unlock its full potential. From data centers to smartphones to autonomous vehicles, memory is no longer a supporting actor but rather the critical bottleneck determining how fast AI can move. What makes this cycle structurally different starts with what happened in 2023. Certain customers drove industry pricing to one third of 2022 levels, forcing Micron into severe losses while still requiring $10 billion in investment just to stay competitive. Most companies in that situation cut spending and survive but Micron invested through the pain with the vision that the other side would be worth it. Those 2023 investments are now producing 84.9% gross margins, $41.46 billion in quarterly revenue, and Q4 guidance of $50 billion up from $11.3 billion in the same quarter just one year ago. That is what it looks like when a company bets on itself at exactly the right moment. Even Micron's own largest customers, Nvidia, Google, Amazon could not forecast the scale of AI memory demand that materialized. When the biggest technology companies in the world cannot project their own memory requirements, you are watching a structural transformation that nobody had models to predict, still in its early innings. Supply cannot respond quickly enough to close that gap. Mehrotra confirmed on air that tightness extends beyond 2027, new domestic fabs take years to bring online, and new HBM capacity which requires advanced 3D stacking that compounds in complexity at every generation won't meaningfully arrive until late 2028. There is no fast fix to a shortage of the most valuable memory on earth. The strategic customer agreements are the most underappreciated part of the entire story. Multi-year contracts with volume commitments and price floors now cover roughly 20% of DRAM volume and 30% of NAND volume, locking in a $100 billion contractual revenue base. The old Micron was at the mercy of customers who could crater prices overnight while the new Micron has contractual floors that make the 2023 scenario structurally impossible to repeat. Long Micron and make sure to follow me Melvin for more deep dives into AI and memory.

Melvin

131,391 Aufrufe • vor 1 Monat

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

271,839 Aufrufe • vor 2 Monaten