Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually...

27,086 görüntüleme • 9 ay önce •via X (Twitter)

64 Yorum

kwindla profil fotoğrafı
kwindla9 ay önce

Docs for this in Pipecat: Mark's original PR: The `UserTurnCompletionLLMServiceMixin` was shipped in Pipecat 0.0.101

Alex M. profil fotoğrafı
Alex M.9 ay önce

Agree the stack is converging. Where we still see failures is accents + disfluencies + overlap in real devices/rooms. That’s why we’ve been focusing at Silencio on collecting in‑the‑wild speech across niche dialects—turn detection gets harder the moment you leave clean US‑English

Hrishi @ Evatype.com profil fotoğrafı
Hrishi @ Evatype.com9 ay önce

Interruptions are quite annoying but my biggest gripe is that neither chatgpt nor claude have a voice mode with their most powerful models. I’d rather wait for a good response than get an instant response from a dumb model.

kwindla profil fotoğrafı
kwindla9 ay önce

Agreed. I pretty much always use Gemini 2.5 Flash, GPT-4.1, or Claude 4.5 Sonnet for both my own agents and for enterprise voice AI use cases. The models need to be able to keep track of multi-turn context, do complex instruction following, and make tool calls reliably.

Emile Cairncross profil fotoğrafı
Emile Cairncross9 ay önce

@hrishio Why not gemini flash 3 preview, out of interest?

kwindla profil fotoğrafı
kwindla9 ay önce

@hrishio Right now the Gemini 3 Flash TTFT is ~1200ms. Gemini 2.5 Flash is half that (twice as fast). The latency difference really matters.

Emile Cairncross profil fotoğrafı
Emile Cairncross9 ay önce

@hrishio Fair enough. Didn't realise it was that high. How about groq models if ttft is that important?

kwindla profil fotoğrafı
kwindla9 ay önce

Groq is now owned by NVIDIA, so we'll see if new models land on Groq. We generally need to use SOTA models for most voice use cases, because accurate instruction following and tool calling are so important. There aren't any open weights models that are good enough at both open-ended conversation and tool calling. I'm excited about the upcoming release of Nemotron 3 Super that NVIDIA has said is coming soon, though!

Karim C profil fotoğrafı
Karim C9 ay önce

Yeah, the interruption problem is mostly engineering now: solid VAD + endpointer tuning, plus a “barge-in budget” so it errs on letting the user finish. The real differentiator is how gracefully it recovers when it guessed wrong without making the convo feel brittle.

kwindla profil fotoğrafı
kwindla9 ay önce

> Yeah, the interruption problem is mostly engineering now I think there are a *lot* of things in the AI agents space like this, now: hard problems that are tractable with good, pragmatic, engineering work. The models are going to keep getting better and keep subsuming some of this framework-level work down into just "what the models do." But that takes time, and even as the models get better we generally want to do more than any generation of models can do natively. I'll be very happy when speech-to-speech and STT models handle turn detection entirely internally. But we don't need to wait for that to happen to build *great* voice agents!

Evan profil fotoğrafı
Evan9 ay önce

Incredible work! Pipecat is leading the way here

Andi Marafioti profil fotoğrafı
Andi Marafioti9 ay önce

This is pretty impressive! I was thinking about using a semantic vad for reachy mini but didn’t find anything easy to port, this seems like it would help 🙌

kwindla profil fotoğrafı
kwindla9 ay önce

The Smart Turn native audio model has gotten quite good, and the latest versions are now small enough to run on CPU in pretty much any environment. All of the training data is open source (and the training code, etc). We're also always working on collecting more human data and improving the synthetic data pipeline for the Smart Turn model. (We augment human data with synthetic speech data.) Happy to work on that together if there are gaps you would like to see filled in the training sets!

Andi Marafioti profil fotoğrafı
Andi Marafioti9 ay önce

Just looked into this, quite cool! So you use it on top of silero ? Happy to collab as well, we won’t gather data with reachy but we have compute and expertise 🤗

kwindla profil fotoğrafı
kwindla9 ay önce

Definitely didn't mean that we should gather any data from end users! We're really, really careful about the data in the Smart Turn data sets being clean and properly acquired. We work with a number of the data collection/labeling companies to gather data for specific languages/accents/scenarios, etc. They contribute because of the marketing value of helping a high-profile, useful, open source model. (And sometimes we pay small amounts to fill data set gaps.) The typical configuration of Smart Turn is to segment the audio with Silero set to `min_silence_duration_ms = 200` (~7 non-speech frames). The Smart Turn model sees up to 16 seconds of audio from the current user turn, including the 200ms tail before the Silero EOS event. We trained the model to overfit to a relatively bi-modal COMPLETE/INCOMPLETE response distribution. Unlike some of the other (good) work in this domain from Krisp, Soniox, Deepgram, and others, we do not use the silence duration as input to the training, or try to produce a useful confidence score in the "mid range." The theory is that turn detection is a data poor domain, and the main job of this model is to gate inference calls to a big, expensive LLM. So we're trying not to make a tiny model running on CPU do impossible things. But the trade-offs here are really interesting, and it's fun to think about how to improve this kind of model in various ways.

TigerHix profil fotoğrafı
TigerHix9 ay önce

Wonderful work. Have you tried using a separate smaller LLM for turn completion detection (i.e., outputting the single token) to lower latency and prevent drifting the main LLM’s attention to turn completion detection (instead of say a hard reasoning task in the user message)?

kwindla profil fotoğrafı
kwindla9 ay önce

Yes, we did this with parallel inference pipelines in the past. So you always run the main voice inference but gate output on another model running in parallel making the turn detection decision. There are two issues with doing it that way: 1) you basically need the big model anyway, because small models aren't as good at this classification as you'd think they would be, and 2) running two inference calls has rate limits implications in larger-scale applications. Now that the SOTA models are good enough to do this as an initial, single-token decision in the context of other complex prompting, we've shifted to just doing it this way.

TigerHix profil fotoğrafı
TigerHix9 ay önce

Thank you, that makes sense! 🙏 Will definitely give pipecat a try soon.

Thomas Ip profil fotoğrafı
Thomas Ip8 ay önce

This is amazing work, adding the full weight of the LLM to decide whether to respond or wait is pretty smart. On the other hand, how can we support assistant interrupting the user? This is not useful for voice agents, but for conversational voice chats humans usually respond with 0 latency or even negative latency by interrupting the prior speaker. This seems crucial for a truly natural chat experience. I guess this could be done a similar way to the token tagging approach, except you run it at an interval while the speaker is speaking?

Alex Khomyakov profil fotoğrafı
Alex Khomyakov9 ay önce

wait you actually measured the interruption rate against real people? that's such a good way to validate it. curious what latency you're hitting to get that natural back-and-forth feel

kwindla profil fotoğrafı
kwindla9 ay önce

P50 total voice-to-voice is just under 1500ms. There is a wrinkle I'm still thinking about. The Smart Turn model tends to be conservative about INCOMPLETE classifications if I've paused at least once in the current utterance, so it's a little bit of a bi-modal distribution. ~75% of responses are under 1500ms. 25% are ~3s because of Smart Turn false INCOMPLETE classifications. I'd like to reduce those cases. But, arguably, when I've paused at least once in the middle of an utterance, the model should be more conservative about emitting a COMPLETE. In practice, as long as the model responds very fast for simple utterances, I don't actually notice if the response time is 3s when I've rambled for a while. :-)

Aldrin Jenson profil fotoğrafı
Aldrin Jenson9 ay önce

This is so cool!

sycoms profil fotoğrafı
sycoms9 ay önce

The single-token tag idea is elegant. Latency-aware turn taking is a big deal.

Lina Colucci profil fotoğrafı
Lina Colucci9 ay önce

Wow - this is really good!

iambenslater profil fotoğrafı
iambenslater9 ay önce

Everyone is sleeping on Voice AI. It IS the killer use case for engagement with AI in the most positive and productive of ways.

Trung Le profil fotoğrafı
Trung Le9 ay önce

This is so clever

Shahn profil fotoğrafı
Shahn9 ay önce

Wow.. amazing demo for turn detection. Is this setup ready for production grade voice agents usage at scale. I’m working on a poc using pipecat, but started getting issues with VAD in later for multi turn conversations

kwindla profil fotoğrafı
kwindla9 ay önce

Yes, this setup is being used in production at scale. What are the VAD issues you're having? If it's tied to conversation length, it's definitely something weird, because VAD should basically reset each turn. It could be CPU-related or memory pressure if there's something that runs each turn that doesn't get cleaned up.

Swayam Gupta profil fotoğrafı
Swayam Gupta9 ay önce

woah, its probably the best turn detection I listened to. one quick question tho, you said you used gpt4.1 model, is it also the same for smart turn? As if it is the case it will perform good in multilingual as well!

kwindla profil fotoğrafı
kwindla9 ay önce

The Smart Turn model supports 23 languages! And gpt-4.1 and Gemini 2.5 Flash are both very good in a pretty wide variety of languages, in text mode. So for non-English, the biggest thing is picking a transcription (STT) model that is great in your target language.

David Goodfellow profil fotoğrafı
David Goodfellow8 ay önce

Given your strategic partnership with vapi, will this soon become available OOB far Vapi voice agents?

morgan arthur profil fotoğrafı
morgan arthur9 ay önce

@kwindla if someone wants to attach this promptmixin LLM with smart turn v3.2 how would they go about it?

kwindla profil fotoğrafı
kwindla9 ay önce

Here's the example I ran to record the video:

Diet Coke Dev profil fotoğrafı
Diet Coke Dev9 ay önce

I’ve already solved most of these on iOS and with powerful models.

David Sevenoaks profil fotoğrafı
David Sevenoaks9 ay önce

I’ve been using the realtime api from OAI for a cooking assistant ( with decent results. I’m curious how this compares particularly on production. Are you ruling out OAI realtime due to cost or the fact that the underlying model is not good enough?

Lounes profil fotoğrafı
Lounes9 ay önce

Definitely not

JpK profil fotoğrafı
JpK9 ay önce

This works with all SOTA STT (like Soniox ) and TTS ?

kwindla profil fotoğrafı
kwindla9 ay önce

Yes!

Lukasz Kowejsza profil fotoğrafı
Lukasz Kowejsza9 ay önce

How is your latency when using GPT-4.1 with tts? I am still not happy with LLM + TTS combinations conversation just don’t sound natural enough compared to realtime models. But multi turn context and tool calling is not reliable enough on voice in voice out models

Jsy0980 profil fotoğrafı
Jsy09807 ay önce

Hey @sventy, my AI intent-radar just flagged a highly qualified buyer specifically looking for a local-first voice/meeting tool on Hacker News today. Tried to send you the direct link but your DMs are closed. Where can I drop it for you on the house?

edgar yan profil fotoğrafı
edgar yan9 ay önce

cool! is this with webrtc?

kwindla profil fotoğrafı
kwindla9 ay önce

Yes! This is the Pipecat SmallWebRTCTransport, which is a zero-dependency, peer-to-peer, WebRTC connection between my browser and the Pipecat agent. Pipecat supports SmallWebRTCTransport, telephony (SIP and PSTN), Daily Global Cloud WebRTC, WebSockets of various kinds, local transport in-process on a machine, and some weirder and more specialized network options, too.

edgar yan profil fotoğrafı
edgar yan9 ay önce

nicee! I will dive deeper into this! I was testing this is really neat!

Leonardo Dominguez profil fotoğrafı
Leonardo Dominguez9 ay önce

Is the smart turn taking model multilingual?

kwindla profil fotoğrafı
kwindla9 ay önce

23 languages!

Cyrus Lim profil fotoğrafı
Cyrus Lim9 ay önce

whose voice is this? sounds like Sati from three body problem

kwindla profil fotoğrafı
kwindla9 ay önce

What I did to record the video was just run the `examples/foundational/22-filter-incomplete-turns.py` single-file example from the Pipecat source code repo. The example uses the "British Reading Lady" voice from @cartesia.

Cyrus Lim profil fotoğrafı
Cyrus Lim9 ay önce

@cartesia Thanks! I played with pipecat about 2 years ago when it was early. things are getting better at an accelerating speed. I thought it was her voice.

Nasim 🧠 profil fotoğrafı
Nasim 🧠9 ay önce

Can we use this with Pipecat Flow? I tried to use it, but agent audio output wasn't working for some reason.

Sim profil fotoğrafı
Sim9 ay önce

Tried very impressive, single word responses don't seem to work though e.g. "yep". Is this a prompting issue or is there another way of solving this?

Oussama Sekkat profil fotoğrafı
Oussama Sekkat9 ay önce

Amazing. What do you guys use for VAD? Silero?

kwindla profil fotoğrafı
kwindla9 ay önce

Yes, generally Silero VAD. Works well in both 16khz and 8khz, proven over time, relatively robust to noisy input, efficient on CPU. And for use cases where background speech is an issue, we usually use the @krispHQ VIVA noise reduction models.

Dmitrii Malakhov profil fotoğrafı
Dmitrii Malakhov9 ay önce

ran 28 voice agents, 5k+ sessions. turn detection got solid but curious about CPU overhead running the native audio model alongside the LLM on concurrent calls. we hit 3.2s latency on voicemail detection at 70-150 calls/day, separate classifier helped

kwindla profil fotoğrafı
kwindla9 ay önce

The Smart Turn v3.2 model is very efficient on CPU. We don't have any issues in production. 3.2s latency on voice mail detection sounds high, but of course it depends on exactly what you're doing in that detection step. Voice mail detection is still something we see people struggle with. My general advice is to use a transcript only, and not to try to detect the "beep." Are you using the pipecat voicemail detection classifier?

Dmitrii Malakhov profil fotoğrafı
Dmitrii Malakhov9 ay önce

not using pipecat, rolled our own with a simple transcript-based classifier. the beep detection rabbit hole was a mistake, burned 2 weeks on it. transcript-only approach ended up way cleaner. will check v3.2, didnt know it was that cpu-efficient

kwindla profil fotoğrafı
kwindla9 ay önce

@thewhychain Totally know the feeling. We spent some time on beep detection. It's definitely something you can try a small model to do reliably, in theory. But in practice, there's so much variation, and there's no actual way to get a good enough set of training data.

Dmitrii Malakhov profil fotoğrafı
Dmitrii Malakhov9 ay önce

yeah that's the core of it. what's your ground truth even? 'the call connected' vs 'that was a beep' is ambiguous in 30% of cases. we tried labeling 500 clips, inter-rater agreement was terrible. transcript timestamps just worked.

kwindla profil fotoğrafı
kwindla9 ay önce

The default `voicemail_response_delay` in the Pipecat voicemail detector service is 2s. Which means you do end up at about 3s of delay time when a human says "hello" and then waits. We did a *lot* of testing and it's hard to go below that and not have the bot mis-classify or start talking just because there's a pause in the voicemail message.

Dmitrii Malakhov profil fotoğrafı
Dmitrii Malakhov9 ay önce

makes sense. the pause-in-message case was our biggest misfire too, bot would kick in mid-greeting. we just flagged voicemails as a category and routed to human, stopped trying to handle them gracefully

Mankiw profil fotoğrafı
Mankiw9 ay önce

Can you use this for “instant” translation in real world condition?

Todd Kuehnl profil fotoğrafı
Todd Kuehnl9 ay önce

Todd Kuehnl Nice work, it is trickier then it seems at first, when I added STT to Anvil-OS last year, I approached it the same way but it took some tweaking for sure.

Michael profil fotoğrafı
Michael9 ay önce

Oh this is really cool - been experimenting with my own home rolled solution but this project seems to tick many of the right boxes out of the gate.

Josh Whiton profil fotoğrafı
Josh Whiton8 ay önce

Hey Kwin, this is very clever. But did you get any mileage out of simpler methods like heuristic gates that just match "hold on, one sec, let me think..." or a small classifier model? Trying to think of useful alternatives when models capable of single-token adherence may not be reliable. (at the cost of in-context adjustments ofc).

kwindla profil fotoğrafı
kwindla8 ay önce

I haven't liked any of the results I've gotten from simple heuristics, but could be a skill issue on my part! I wonder if we should just collect a bunch of data and fine-tune a small, fast, text model! :-)

Benzer Videolar

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)

kwindla

40,593 görüntüleme • 5 ay önce

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

336,030 görüntüleme • 2 ay önce

We blew past the Turing test for voice agents about a year ago. I don't think we talk about this enough. We have lots of data from real-world voice agent deployments. In contexts like customer support, people often forget they are talking to a voice agent. This is true even when the system tells people they are talking to an AI agent at the beginning of the session. (Which I think is a best practice that almost all AI agents should adhere to.) So ... when will this happen for realtime, conversational AI video? Well, we're getting very close. Tavus just released a new realtime video model that, at its best, passes the Turing test for conversational video. It's pretty amazing. The model is full-duplex (it can talk and listen at the same time), and does vision and audio input and audio+video output in realtime. Both on the nascent conversational video benchmarks, and in my testing, it's a pretty big step forward. Here's an unedited video of one of my conversations with the new model. I think you can see a few interesting things here. When the model responds as quickly as you expect a human to respond, the experience is magical. The responses are conversational and natural; there's no leaning on stock filler phrases to hit the response time goals. The integration between the video and audio feels natural, which has historically been an uncanny valley thing that most video avatar experiences didn't quite get past. The model can do hand gestures! If you're deep into this stuff, you'll find that a little bit mind-blowing. And, in keeping with all the stuff that Tavus has released, the fact that the model can "see" as well as hear and talk is integral to the experience and a big part of what makes the conversation feel useful and interesting. At the very beginning of the conversation, you can see that the model and I talk over each other. Which is something that happens a lot in human-to-human video call conversations. We both recover from this gracefully, which, again, has been something that rarely worked quite perfectly in previous generations of realtime video systems. Even more than the "when it works perfectly" capabilities, it's probably these recovery and robustness moments that get us all the way to experiences that feel completely natural.

kwindla

40,489 görüntüleme • 1 ay önce

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,573 görüntüleme • 3 yıl önce

Sam Altman made the case for open-source harnesses in July. a month later, someone shipped it, and it's more efficient than most managed harnesses. here is the problem it was aimed at: a large share of your agent's token bill is the model rereading things it already read. that isn't the model's doing. the runtime around it decides what goes into every prompt and how often the model gets called. for example, an agent queries a CRM at step four and gets back 400 rows. those rows get piled up in the conversation history. by step nineteen, the model has to read those rows fifteen times unnecessarily, and every token read is billed at input rates. it happened because your harness assembled that prompt on every turn and kept the rows in it. that gives you two levers: how much context the harness carries forward, and how often it calls the model. there are four practical ways to keep the prompt from growing unnecessarily: → load tool schemas on demand. a server with 100 tools doesn't need to put all 100 into every prompt when the agent only calls two. → offload large results to disk. turn a large response into a short preview and a file path instead of replaying the entire result on every turn. → delegate to subagents. let a subagent spend thirty tool calls in its own context and return one summary to the root agent. → run toolchains in code. one script calls three tools, joins the results, and returns a table instead of three turns each dragging a full response. but reducing context is only half the job. you also need to control how often the model gets called. a good harness should avoid unnecessary planning, verification, and reflection when the work can be completed in fewer steps. TrueFoundry's open-source agent harness, TrueForge, is built around both of those controls. it sits between the model and the tools, deciding what goes into every prompt and when another model call is actually needed. it also breaks token usage down across the harness, skills, instructions, tools, and messages. DevRev's Enterprise-Bench is where this gets tested, on multi-step tasks of the kind where an agent pulls records from one system and reconciles them against another. TrueFoundry ran TrueForge there against Claude Managed Agents, both on the same model, and both finished the same number of tasks. the tie is the part that matters, because it means the gap underneath is not a quality tradeoff. TrueForge reached that score on close to a third of the tokens, with roughly 40% fewer trips back to the model. for the same result, that comes out around 2.7x cheaper than Claude Managed Agents. swapping in an open model made it sharper still. TrueForge with GLM-5.2 scored a little higher than either setup above, and the entire benchmark run cost about $3 at list prices. being open source matters beyond the license here. the model underneath can be swapped without rewriting the agent, and the whole thing can run inside your own environment when the data cannot leave it. all of this comes down to the runtime around the model, the context it carries, the tools it exposes, and how many times it goes back to the model. that is what a production harness actually owns. the full task list, the per-run numbers, and the MIT-licensed code are on GitHub: (don't forget to star 🌟) you can read more about the same in the article quoted below. thanks to the TrueForge team for working with me on this one.

Akshay 🚀

76,998 görüntüleme • 3 ay önce

Chamath: Two terms you need to pay attention to in AI are Prefill and Decode “There's two terms that I think you're going to hear a ton about over these next few years.” “The first term is prefill, and the next is decode.” “What prefill and decode are, are two very distinct ways of how models think, and how a model goes through the process of answering a question that you ask it.” “And so when you send a prompt to AI, what happens is that the model processes it. This is called the reading phase or prefill.” “It reads your entire prompt all at once. And then it does a bunch of math, calculates all these relationships between all the words, and it stores them in temporary memory.” “The problem is that this is really compute bound. So it requires massive brute force. And Nvidia GPUs crush here.” “And their architecture is designed for massive parallel processing, which makes them really amazing at digesting these long prompts.” “So the problem just gets bigger and bigger, Nvidia just completely dominates.” “But the next phase though, this critical phase, the decode phase, is the writing phase, right?” “So the model starts to generate a response, you ask it a question and its response, one token at a time.” “And then to pick the next token to pick the next word, it has to look back at everything it has said already so that it doesn't hallucinate.” “The problem is that this is incredibly memory bandwidth constrained.” “And in our architecture, a long time ago, we made these design decisions from day one.” “And so what we did was we took a very different architectural approach, we took a very conservative process technology. We weren't pushing the boundaries of physics.” “And we used a lot of what's called SRAM. So memory on the chip so that we could do this decode thing as well or better than everybody else.” “And so now when you put these two things together, I just think it's going to create a huge acceleration in the ability for this entire infrastructure layer to get much cheaper and much more valuable, which I suspect then it'll have a lot more developer pull, you'll get a lot more applications being built, billions and billions of more people using it.”

The All-In Podcast

572,793 görüntüleme • 10 ay önce