Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Async, automatic, non-blocking context compaction for long-running agents. Last week I gave a talk called Space Machine Sandboxes at the Daytona AI builders meetup about patterns for long-running agents. I work a lot on voice AI agents, which are fundamentally multi-turn, long-context loops. I also build lots of other...

kwindla's profile picture

kwindla

12,364 subscribers

25,877 görüntüleme • 4 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

The best way to learn AI is to build with agents. To help with that, we've launched hands-on labs and a new series on Agentic Engineering. First topic: Agent Skills. Next in the pipeline: planning, context engineering, multi-agent systems, long-running agents,.. Go build!
9:09

The best way to learn AI is to build with agents. To help with that, we've launched hands-on labs and a new series on Agentic Engineering. First topic: Agent Skills. Next in the pipeline: planning, context engineering, multi-agent systems, long-running agents,.. Go build!

elvis

29,596 görüntüleme • 15 gün önce

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.
4:39

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) Mark Backman made a Pipecat AI PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

kwindla

26,812 görüntüleme • 3 ay önce

Introducing Electric Agents! Agents are not compute. Agents are data. Multi-agent is a sync problem. Electric Agents is the first agent platform built on sync. Use it to build scalable, collaborative, long-lived multi-agent systems: 🧵👇
0:35

Introducing Electric Agents! Agents are not compute. Agents are data. Multi-agent is a sync problem. Electric Agents is the first agent platform built on sync. Use it to build scalable, collaborative, long-lived multi-agent systems: 🧵👇

Electric

72,866 görüntüleme • 1 ay önce

Long-running AI agents can run tasks over hours, days or weeks. Here's a few of my thoughts on them from Google Cloud's Agent Factory.
3:56

Long-running AI agents can run tasks over hours, days or weeks. Here's a few of my thoughts on them from Google Cloud's Agent Factory.

Addy Osmani

33,359 görüntüleme • 4 gün önce

Can AI agents really manage other agents? Most of us can't even get one AI agent to really work well. But it works in Replit ⠕ v3. For real. And it's so cool. I had it do a 2.75 hour autonomous security audit. One agent managing other AI agents. Here's how it worked:🧵
3:39

Can AI agents really manage other agents? Most of us can't even get one AI agent to really work well. But it works in Replit ⠕ v3. For real. And it's so cool. I had it do a 2.75 hour autonomous security audit. One agent managing other AI agents. Here's how it worked:🧵

Jason ✨👾SaaStr.Ai✨ Lemkin

14,102 görüntüleme • 8 ay önce

We just solved the long-horizon planning & execution issue with Agents 🤯! Excited to announce that MultiOn AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task 🤩!! Our latest upgrade cracks the goal divergence & looping issues that have long plagued agents like AutoGPT 🔥 Here's a demo of it working as a AI news research agent, continuously searching for the latest AI news and tweeting it out everyday, all autonomously of course 🚀: 🔊
9:39

We just solved the long-horizon planning & execution issue with Agents 🤯! Excited to announce that MultiOn AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task 🤩!! Our latest upgrade cracks the goal divergence & looping issues that have long plagued agents like AutoGPT 🔥 Here's a demo of it working as a AI news research agent, continuously searching for the latest AI news and tweeting it out everyday, all autonomously of course 🚀: 🔊

Div Garg

309,949 görüntüleme • 2 yıl önce

"We are clearly entering a world where a product-minded engineer is now empowered to produce software without writing a line of code for it." In this conversation, Temporal CEO Samar Abbas joins a16z GPs Sarah Wang and Raghu Raghuram to cover: - Why agents are going from short-lived and interactive to long-running and async - Why the engineer of the future manages 15 parallel AI tasks at once - How OpenAI Codex runs millions of concurrent agent executions on Temporal - How real-time context engineering is exploding on Temporal - Why SaaS isn't dead, and value is migrating to APIs 00:00 Introduction 04:03 Temporal's origin story 11:14 Why agents raise the stakes 16:00 Specialized agents need durable RPC 25:20 Deep research agents 30:58 Execution histories as a superpower 39:04 Minimal viable long-running agent architecture 45:07 Context engineering at scale 52:40 Where value accrues: The "five-layer cake" and breakout AI applications Samar Abbas Sarah Wang Raghu Raghuram
1:02:30

"We are clearly entering a world where a product-minded engineer is now empowered to produce software without writing a line of code for it." In this conversation, Temporal CEO Samar Abbas joins a16z GPs Sarah Wang and Raghu Raghuram to cover: - Why agents are going from short-lived and interactive to long-running and async - Why the engineer of the future manages 15 parallel AI tasks at once - How OpenAI Codex runs millions of concurrent agent executions on Temporal - How real-time context engineering is exploding on Temporal - Why SaaS isn't dead, and value is migrating to APIs 00:00 Introduction 04:03 Temporal's origin story 11:14 Why agents raise the stakes 16:00 Specialized agents need durable RPC 25:20 Deep research agents 30:58 Execution histories as a superpower 39:04 Minimal viable long-running agent architecture 45:07 Context engineering at scale 52:40 Where value accrues: The "five-layer cake" and breakout AI applications Samar Abbas Sarah Wang Raghu Raghuram

a16z

45,496 görüntüleme • 3 ay önce

inside Smallest AI office, Sudarshan Kamath on what's the missing piece for voice agents!! why Lightning TTS, ultra-low latency voice models needs HydraDB the knowledge and context layer for AI Agents!!
5:43

inside Smallest AI office, Sudarshan Kamath on what's the missing piece for voice agents!! why Lightning TTS, ultra-low latency voice models needs HydraDB the knowledge and context layer for AI Agents!!

Harnoor Singh

32,143 görüntüleme • 1 ay önce

Confused by AI Agents and how to PROFIT BIG? Here is a beginners guide to trading AI Agents! I also show you 15 AI Agent tokens across BASE, SOL, & AVAX that are ripe for buying on this dip. (BONUS: One of these AI Agents wants to KILL threadguy!!!)
40:37

Confused by AI Agents and how to PROFIT BIG? Here is a beginners guide to trading AI Agents! I also show you 15 AI Agent tokens across BASE, SOL, & AVAX that are ripe for buying on this dip. (BONUS: One of these AI Agents wants to KILL threadguy!!!)

'Stache

13,641 görüntüleme • 1 yıl önce

Sub-agents in (latent) space! We’ve been working on a side project. As far as I know, this is the first massively multiplayer, completely LLM-driven game. Come play Gradient Bang with us. See if you can catch me on the leaderboard. This whole thing started because I wanted to explore a bunch of things I’m currently obsessed with, in an application of non-trivial size, that felt both new and old at the same time. So … a retro-style space trading game built entirely around interacting with and managing multiple LLMs. Factorio, but instead of clicking, you cajole your ship AI into tasking other AIs to do things for you. Some of the things we’ve been thinking about as we hack on Gradient Bang: - Sub-agent orchestration - Partial context sharing between multiple LLM inference loops - Managing very long contexts, and episodic memory across user sessions - World events and large volumes of structured data input as part of human/agent conversations - Dynamic user interfaces, driven/created on the fly by LLMs - And, of course, voice as primary input If you’ve been building coding harnesses, or writing Open Claw agents, or doing pretty much anything that pushes the boundaries of AI-native development these days, you’re probably thinking about these things too! This is all built with Pipecat AI, the back end is Supabase, the React front end is deployed to Vercel, and all the code is open source.
1:03

Sub-agents in (latent) space! We’ve been working on a side project. As far as I know, this is the first massively multiplayer, completely LLM-driven game. Come play Gradient Bang with us. See if you can catch me on the leaderboard. This whole thing started because I wanted to explore a bunch of things I’m currently obsessed with, in an application of non-trivial size, that felt both new and old at the same time. So … a retro-style space trading game built entirely around interacting with and managing multiple LLMs. Factorio, but instead of clicking, you cajole your ship AI into tasking other AIs to do things for you. Some of the things we’ve been thinking about as we hack on Gradient Bang: - Sub-agent orchestration - Partial context sharing between multiple LLM inference loops - Managing very long contexts, and episodic memory across user sessions - World events and large volumes of structured data input as part of human/agent conversations - Dynamic user interfaces, driven/created on the fly by LLMs - And, of course, voice as primary input If you’ve been building coding harnesses, or writing Open Claw agents, or doing pretty much anything that pushes the boundaries of AI-native development these days, you’re probably thinking about these things too! This is all built with Pipecat AI, the back end is Supabase, the React front end is deployed to Vercel, and all the code is open source.

kwindla

453,010 görüntüleme • 1 ay önce

I don’t know how long the AI Agents hype will continue. But I do know that AI crypto as a whole is a long-term narrative, and I have kept this thesis publicly for 2 years already. Today I’ll be showing all the different sub-sectors in AI Crypto not only including AI agents, but also AI blockchains, DePin, AI Gaming, Launchpads, and AI Memes. 01:53 Large Caps 02:33 DePin 03:36 AI Gaming 04:18 AI Agent Infra 06:12 AI Memes 07:30 Mid Caps 14:21 Low Caps
29:04

I don’t know how long the AI Agents hype will continue. But I do know that AI crypto as a whole is a long-term narrative, and I have kept this thesis publicly for 2 years already. Today I’ll be showing all the different sub-sectors in AI Crypto not only including AI agents, but also AI blockchains, DePin, AI Gaming, Launchpads, and AI Memes. 01:53 Large Caps 02:33 DePin 03:36 AI Gaming 04:18 AI Agent Infra 06:12 AI Memes 07:30 Mid Caps 14:21 Low Caps

VirtualBacon

119,562 görüntüleme • 1 yıl önce

HERMES Agent just launched a desktop app for managing autonomous AI agents at scale You can now: - Run multiple agents from one interface - Maintain persistent memory - Automate long-running workflows - Keep agents operating continuously
0:34

HERMES Agent just launched a desktop app for managing autonomous AI agents at scale You can now: - Run multiple agents from one interface - Maintain persistent memory - Automate long-running workflows - Keep agents operating continuously

Vaishnavi

57,860 görüntüleme • 13 gün önce

Mayor Sim says he's utilizing artificial intelligence to get stuff done: "I have 11 AI agents running right now doing all my — like a lot of my work in the background."
0:40

Mayor Sim says he's utilizing artificial intelligence to get stuff done: "I have 11 AI agents running right now doing all my — like a lot of my work in the background."

Jarryd Jäger

47,724 görüntüleme • 24 gün önce

Zuck believes there could be more AI agents than people in the world. During our conversation, I asked Mark about his long-term vision of AI and AGI in the future. His response: Zuckerberg: “Our vision is that there should be a lot of different AI out there and AI services, not just one singular AI... I think we're going to live in a world where there are going to be hundreds of millions or billions of different AI agents eventually, probably more AI agents than there are people in the world.“ Zuckerberg added: “Every business in the future, just like they have an email address, a website, and a social media presence today, is going to have an AI agent that their customers can talk to in the future.” His vision really challenges the idea of a singular, dominant, closed AI system. He advocates for "a lot of different AI out there and AI services, not just one singular AI," aiming to democratize AI to foster more widespread innovation. He also believes that this is partly why open-source is "safer than the alternative of closed development".
0:23

Zuck believes there could be more AI agents than people in the world. During our conversation, I asked Mark about his long-term vision of AI and AGI in the future. His response: Zuckerberg: “Our vision is that there should be a lot of different AI out there and AI services, not just one singular AI... I think we're going to live in a world where there are going to be hundreds of millions or billions of different AI agents eventually, probably more AI agents than there are people in the world.“ Zuckerberg added: “Every business in the future, just like they have an email address, a website, and a social media presence today, is going to have an AI agent that their customers can talk to in the future.” His vision really challenges the idea of a singular, dominant, closed AI system. He advocates for "a lot of different AI out there and AI services, not just one singular AI," aiming to democratize AI to foster more widespread innovation. He also believes that this is partly why open-source is "safer than the alternative of closed development".

Rowan Cheung

94,629 görüntüleme • 1 yıl önce

Multi-agent orchestration - Niko spinning 100 AI agents in ultra-fast sandboxes ~ AI at Meta Llama 4 on Groq Inc executing code in Daytona
3:18

Multi-agent orchestration - Niko spinning 100 AI agents in ultra-fast sandboxes ~ AI at Meta Llama 4 on Groq Inc executing code in Daytona

Daytona.io

73,057 görüntüleme • 1 yıl önce

Another example of the multiple TTS parallel pipelines pattern. Here's a voice AI agent that speaks both English and Arabic, using a specific model/voice for each language. These are PlayAI voices. The STT, TTS, and LLM inference is all running on Groq Inc. (The LLM is LLama 4 Maverick.)
1:26

Another example of the multiple TTS parallel pipelines pattern. Here's a voice AI agent that speaks both English and Arabic, using a specific model/voice for each language. These are PlayAI voices. The STT, TTS, and LLM inference is all running on Groq Inc. (The LLM is LLama 4 Maverick.)

kwindla

13,328 görüntüleme • 11 ay önce

Today we’re launching our first homegrown AI model: an open source turn detection model for building voice agents. Instead of relying solely on voice activity detection (VAD), which only considers when a user is speaking, our model also considers what has and is being said in the context of a conversation and predicts when a user is finished expressing their thoughts before the agent responds. Conversations with AI voice agents using this new model flow much more naturally without constant interruptions from the AI— check it out (more videos, details, and code in the thread):
2:36

Today we’re launching our first homegrown AI model: an open source turn detection model for building voice agents. Instead of relying solely on voice activity detection (VAD), which only considers when a user is speaking, our model also considers what has and is being said in the context of a conversation and predicts when a user is finished expressing their thoughts before the agent responds. Conversations with AI voice agents using this new model flow much more naturally without constant interruptions from the AI— check it out (more videos, details, and code in the thread):

LiveKit

126,804 görüntüleme • 1 yıl önce

🤖 HYTOPIA AI Agents Society (Start Building!) 🎮 We've just added the "ai-agents" example to our SDK, which is a starting point of a game with an AI Agent Society! You can use this example as a template to build your own AI agents for a HYTOPIA game:
3:20

🤖 HYTOPIA AI Agents Society (Start Building!) 🎮 We've just added the "ai-agents" example to our SDK, which is a starting point of a game with an AI Agent Society! You can use this example as a template to build your own AI agents for a HYTOPIA game:

HYTOPIA

15,727 görüntüleme • 1 yıl önce

Here is a DigiByte Update video where I talk recent $DGB updates, taproot & and the future of DGB as an AI payment Blockchain for AI agents & the coming autonomous agentic AI economy.
16:57

Here is a DigiByte Update video where I talk recent $DGB updates, taproot & and the future of DGB as an AI payment Blockchain for AI agents & the coming autonomous agentic AI economy.

Jared Tate ©️

11,704 görüntüleme • 1 yıl önce