Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Async, automatic, non-blocking context compaction for long-running agents. Last week I gave a talk called Space Machine Sandboxes at the Daytona AI builders meetup about patterns for long-running agents. I work a lot on voice AI agents, which are fundamentally multi-turn, long-context loops. I also build lots of other... show more

kwindla

12,364 subscribers

25,877 Aufrufe • vor 9 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

The best way to learn AI is to build with agents. To help with that, we've launched hands-on labs and a new series on Agentic Engineering. First topic: Agent Skills. Next in the pipeline: planning, context engineering, multi-agent systems, long-running agents,.. Go build!

The best way to learn AI is to build with agents. To help with that, we've launched hands-on labs and a new series on Agentic Engineering. First topic: Agent Skills. Next in the pipeline: planning, context engineering, multi-agent systems, long-running agents,.. Go build!

elvis

31,802 Aufrufe • vor 5 Monaten

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

kwindla

27,086 Aufrufe • vor 8 Monaten

Google keeps sharing a ton of free content! Here is one of their most complete examples: a long-running AI agent that pauses, resumes, and never loses context. It simulates the onboarding of a new employee. There are the 3 architectural patterns you'll learn from this:

Google keeps sharing a ton of free content! Here is one of their most complete examples: a long-running AI agent that pauses, resumes, and never loses context. It simulates the onboarding of a new employee. There are the 3 architectural patterns you'll learn from this:

Santiago

20,040 Aufrufe • vor 4 Monaten

I GAVE TEAMILY AI A WEBSITE TO BUILD. finally not another one person + a bunch of agents tool. Teamily AI is a Humans + Agents platform where multiple people and agents can work in the same group with shared context. → one handled the frontend → one built the backend on top of it → one reviewed the code I built an AI developer portfolio for startup founders, then changed the direction midway. the best part? the team adapted without starting from scratch. then my teammate gave feedback and the agents updated the live site in Website Builder. this feels way closer to having an actual AI team than just using a chatbot. Teamily AI is a human-agent collaborative workspace, great for small businesses and teams. built with Website Builder Teamily AI

I GAVE TEAMILY AI A WEBSITE TO BUILD. finally not another one person + a bunch of agents tool. Teamily AI is a Humans + Agents platform where multiple people and agents can work in the same group with shared context. → one handled the frontend → one built the backend on top of it → one reviewed the code I built an AI developer portfolio for startup founders, then changed the direction midway. the best part? the team adapted without starting from scratch. then my teammate gave feedback and the agents updated the live site in Website Builder. this feels way closer to having an actual AI team than just using a chatbot. Teamily AI is a human-agent collaborative workspace, great for small businesses and teams. built with Website Builder Teamily AI

Aman

11,666 Aufrufe • vor 17 Tagen

Andrej Karpathy just broke the entire premise of modern AI: "The new computer isn't a chip - It's a language model running agent loops." Agents talking to agents -loops spawning sub-loops The stack being built RIGHT NOW: agent loop + memory + sub-agents + harness In a 20-minute talk, Karpathy explain the full reasoning stack

Andrej Karpathy just broke the entire premise of modern AI: "The new computer isn't a chip - It's a language model running agent loops." Agents talking to agents -loops spawning sub-loops The stack being built RIGHT NOW: agent loop + memory + sub-agents + harness In a 20-minute talk, Karpathy explain the full reasoning stack

Ricker

32,015 Aufrufe • vor 3 Monaten

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent with the guide below.

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent with the guide below.

Roan

253,033 Aufrufe • vor 2 Monaten

Introducing Electric Agents! Agents are not compute. Agents are data. Multi-agent is a sync problem. Electric Agents is the first agent platform built on sync. Use it to build scalable, collaborative, long-lived multi-agent systems: 🧵👇

Introducing Electric Agents! Agents are not compute. Agents are data. Multi-agent is a sync problem. Electric Agents is the first agent platform built on sync. Use it to build scalable, collaborative, long-lived multi-agent systems: 🧵👇

Electric

82,158 Aufrufe • vor 5 Monaten

Google Engineers just dropped a 1-hour course on agentic engineering from scratch: 0% - 00:00 – How to build your first AI agent 25% - 08:24 – Build agent memory (short, persistent, long) 50% - 28:34 – Agentic loops, long-running AI agents 100% - 1:00:22 – Multi-agentic systems This 1-hour guide will replace 10 paid agentic courses on the internet. Watch it today, then read how to run multiple agents in the article below.

Google Engineers just dropped a 1-hour course on agentic engineering from scratch: 0% - 00:00 – How to build your first AI agent 25% - 08:24 – Build agent memory (short, persistent, long) 50% - 28:34 – Agentic loops, long-running AI agents 100% - 1:00:22 – Multi-agentic systems This 1-hour guide will replace 10 paid agentic courses on the internet. Watch it today, then read how to run multiple agents in the article below.

Ricker

11,708 Aufrufe • vor 1 Monat

$Microsoft just dropped a 17-page paper - "Less Context, Better Agents" - proving the thing nobody building agents wants to admit: Your agent isn't failing because it needs more context. It's drowning in the context it already has. They ran GPT-5 four ways on one 50-task benchmark (via MCP): Full history → bloated, pricey, more errors Trim to the last 5 tool calls → 79% done Add light summarization → 91.6%, on a fraction of the tokens Less context. Fewer tokens. Higher completion. Everyone's racing to cram MORE into the window. Microsoft just published the receipt that the opposite wins for long-running agents. Rewired how I build agents this week. Paper ↓$

Microsoft just dropped a 17-page paper - "Less Context, Better Agents" - proving the thing nobody building agents wants to admit: Your agent isn't failing because it needs more context. It's drowning in the context it already has. They ran GPT-5 four ways on one 50-task benchmark (via MCP): Full history → bloated, pricey, more errors Trim to the last 5 tool calls → 79% done Add light summarization → 91.6%, on a fraction of the tokens Less context. Fewer tokens. Higher completion. Everyone's racing to cram MORE into the window. Microsoft just published the receipt that the opposite wins for long-running agents. Rewired how I build agents this week. Paper ↓

Archive

18,242 Aufrufe • vor 4 Monaten

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent.

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent.

Dhairya Karekar

133,800 Aufrufe • vor 2 Monaten

Anthropic engineer: “80% of our engineers are using Loops. Now everyone is exploring how to build right Context for agents.” in 3-6 months, we’ll all become context engineers. Context is the most important part of agents.” in this 1-hour talk, Anthropic engineer shows how to build a self-improving agents from sratch. It’s worth more than a $500 agentic course. Watch this video, then read the article below on how to become a context architect.

Anthropic engineer: “80% of our engineers are using Loops. Now everyone is exploring how to build right Context for agents.” in 3-6 months, we’ll all become context engineers. Context is the most important part of agents.” in this 1-hour talk, Anthropic engineer shows how to build a self-improving agents from sratch. It’s worth more than a $500 agentic course. Watch this video, then read the article below on how to become a context architect.

Codez

36,334 Aufrufe • vor 3 Monaten

Long-running AI agents can run tasks over hours, days or weeks. Here's a few of my thoughts on them from Google Cloud's Agent Factory.

Long-running AI agents can run tasks over hours, days or weeks. Here's a few of my thoughts on them from Google Cloud's Agent Factory.

Addy Osmani

47,338 Aufrufe • vor 4 Monaten

gave the agents a simple tool to find and message other active agents. have this in pi as well. going full gastown here. so now an agent can talk to it's background subagents, and other agents running on my Android phone. useful for context transfer and coordination.

gave the agents a simple tool to find and message other active agents. have this in pi as well. going full gastown here. so now an agent can talk to it's background subagents, and other agents running on my Android phone. useful for context transfer and coordination.

Mario Zechner

13,326 Aufrufe • vor 21 Tagen

We blew past the Turing test for voice agents about a year ago. I don't think we talk about this enough. We have lots of data from real-world voice agent deployments. In contexts like customer support, people often forget they are talking to a voice agent. This is true even when the system tells people they are talking to an AI agent at the beginning of the session. (Which I think is a best practice that almost all AI agents should adhere to.) So ... when will this happen for realtime, conversational AI video? Well, we're getting very close. Tavus just released a new realtime video model that, at its best, passes the Turing test for conversational video. It's pretty amazing. The model is full-duplex (it can talk and listen at the same time), and does vision and audio input and audio+video output in realtime. Both on the nascent conversational video benchmarks, and in my testing, it's a pretty big step forward. Here's an unedited video of one of my conversations with the new model. I think you can see a few interesting things here. When the model responds as quickly as you expect a human to respond, the experience is magical. The responses are conversational and natural; there's no leaning on stock filler phrases to hit the response time goals. The integration between the video and audio feels natural, which has historically been an uncanny valley thing that most video avatar experiences didn't quite get past. The model can do hand gestures! If you're deep into this stuff, you'll find that a little bit mind-blowing. And, in keeping with all the stuff that Tavus has released, the fact that the model can "see" as well as hear and talk is integral to the experience and a big part of what makes the conversation feel useful and interesting. At the very beginning of the conversation, you can see that the model and I talk over each other. Which is something that happens a lot in human-to-human video call conversations. We both recover from this gracefully, which, again, has been something that rarely worked quite perfectly in previous generations of realtime video systems. Even more than the "when it works perfectly" capabilities, it's probably these recovery and robustness moments that get us all the way to experiences that feel completely natural.

kwindla

39,944 Aufrufe • vor 24 Tagen

"We are clearly entering a world where a product-minded engineer is now empowered to produce software without writing a line of code for it." In this conversation, Temporal CEO Samar Abbas joins a16z GPs Sarah Wang and Raghu Raghuram to cover: - Why agents are going from short-lived and interactive to long-running and async - Why the engineer of the future manages 15 parallel AI tasks at once - How OpenAI Codex runs millions of concurrent agent executions on Temporal - How real-time context engineering is exploding on Temporal - Why SaaS isn't dead, and value is migrating to APIs 00:00 Introduction 04:03 Temporal's origin story 11:14 Why agents raise the stakes 16:00 Specialized agents need durable RPC 25:20 Deep research agents 30:58 Execution histories as a superpower 39:04 Minimal viable long-running agent architecture 45:07 Context engineering at scale 52:40 Where value accrues: The "five-layer cake" and breakout AI applications Samar Abbas Sarah Wang Raghu Raghuram

"We are clearly entering a world where a product-minded engineer is now empowered to produce software without writing a line of code for it." In this conversation, Temporal CEO Samar Abbas joins a16z GPs Sarah Wang and Raghu Raghuram to cover: - Why agents are going from short-lived and interactive to long-running and async - Why the engineer of the future manages 15 parallel AI tasks at once - How OpenAI Codex runs millions of concurrent agent executions on Temporal - How real-time context engineering is exploding on Temporal - Why SaaS isn't dead, and value is migrating to APIs 00:00 Introduction 04:03 Temporal's origin story 11:14 Why agents raise the stakes 16:00 Specialized agents need durable RPC 25:20 Deep research agents 30:58 Execution histories as a superpower 39:04 Minimal viable long-running agent architecture 45:07 Context engineering at scale 52:40 Where value accrues: The "five-layer cake" and breakout AI applications Samar Abbas Sarah Wang Raghu Raghuram

a16z

46,313 Aufrufe • vor 8 Monaten

We just solved the long-horizon planning & execution issue with Agents 🤯! Excited to announce that MultiOn AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task 🤩!! Our latest upgrade cracks the goal divergence & looping issues that have long plagued agents like AutoGPT 🔥 Here's a demo of it working as a AI news research agent, continuously searching for the latest AI news and tweeting it out everyday, all autonomously of course 🚀: 🔊

We just solved the long-horizon planning & execution issue with Agents 🤯! Excited to announce that MultiOn AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task 🤩!! Our latest upgrade cracks the goal divergence & looping issues that have long plagued agents like AutoGPT 🔥 Here's a demo of it working as a AI news research agent, continuously searching for the latest AI news and tweeting it out everyday, all autonomously of course 🚀: 🔊

Div Garg

309,990 Aufrufe • vor 2 Jahren

Sub-agents in (latent) space! We’ve been working on a side project. As far as I know, this is the first massively multiplayer, completely LLM-driven game. Come play Gradient Bang with us. See if you can catch me on the leaderboard. This whole thing started because I wanted to explore a bunch of things I’m currently obsessed with, in an application of non-trivial size, that felt both new and old at the same time. So … a retro-style space trading game built entirely around interacting with and managing multiple LLMs. Factorio, but instead of clicking, you cajole your ship AI into tasking other AIs to do things for you. Some of the things we’ve been thinking about as we hack on Gradient Bang: - Sub-agent orchestration - Partial context sharing between multiple LLM inference loops - Managing very long contexts, and episodic memory across user sessions - World events and large volumes of structured data input as part of human/agent conversations - Dynamic user interfaces, driven/created on the fly by LLMs - And, of course, voice as primary input If you’ve been building coding harnesses, or writing Open Claw agents, or doing pretty much anything that pushes the boundaries of AI-native development these days, you’re probably thinking about these things too! This is all built with Pipecat AI, the back end is Supabase, the React front end is deployed to Vercel, and all the code is open source.

Sub-agents in (latent) space! We’ve been working on a side project. As far as I know, this is the first massively multiplayer, completely LLM-driven game. Come play Gradient Bang with us. See if you can catch me on the leaderboard. This whole thing started because I wanted to explore a bunch of things I’m currently obsessed with, in an application of non-trivial size, that felt both new and old at the same time. So … a retro-style space trading game built entirely around interacting with and managing multiple LLMs. Factorio, but instead of clicking, you cajole your ship AI into tasking other AIs to do things for you. Some of the things we’ve been thinking about as we hack on Gradient Bang: - Sub-agent orchestration - Partial context sharing between multiple LLM inference loops - Managing very long contexts, and episodic memory across user sessions - World events and large volumes of structured data input as part of human/agent conversations - Dynamic user interfaces, driven/created on the fly by LLMs - And, of course, voice as primary input If you’ve been building coding harnesses, or writing Open Claw agents, or doing pretty much anything that pushes the boundaries of AI-native development these days, you’re probably thinking about these things too! This is all built with Pipecat AI, the back end is Supabase, the React front end is deployed to Vercel, and all the code is open source.

kwindla

456,975 Aufrufe • vor 6 Monaten

inside Smallest AI office, Sudarshan Kamath on what's the missing piece for voice agents!! why Lightning TTS, ultra-low latency voice models needs HydraDB the knowledge and context layer for AI Agents!!

inside Smallest AI office, Sudarshan Kamath on what's the missing piece for voice agents!! why Lightning TTS, ultra-low latency voice models needs HydraDB the knowledge and context layer for AI Agents!!

Harnoor Singh

32,669 Aufrufe • vor 5 Monaten