Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or...

alokbishoyi97's profile picture

Alok Bishoyi

4,938 subscribers

12,633 Aufrufe • vor 3 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.
0:47

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Adam Cohen

25,060 Aufrufe • vor 2 Monaten

BREAKING: Apoorv Agrawal (Apoorv Agrawal) of Altimeter Capital says there are "four seasons" in AI: "They're called OpenAI, Anthropic, SpaceX, & Google." "So if you're a CIO, CEO, if you're about to make a multi-million, billion-dollar decision of which lab to build your intelligence on, plan for the climate, not for the weather." What does that mean? "That's multi-model routing. That's evals. That's thinking about post-train custom models." "You'll have a portion of your workloads going through open source, as Jesse Zhang from Decagon wrote very eloquently. A big portion of their volume goes through open source, ~90% now." "And then the frontier stuff, discovering new use cases, will always go through the most intelligent models." "Planning for that multi-model world, planning to not get stuck in one regime, is the climate."
3:09

BREAKING: Apoorv Agrawal (Apoorv Agrawal) of Altimeter Capital says there are "four seasons" in AI: "They're called OpenAI, Anthropic, SpaceX, & Google." "So if you're a CIO, CEO, if you're about to make a multi-million, billion-dollar decision of which lab to build your intelligence on, plan for the climate, not for the weather." What does that mean? "That's multi-model routing. That's evals. That's thinking about post-train custom models." "You'll have a portion of your workloads going through open source, as Jesse Zhang from Decagon wrote very eloquently. A big portion of their volume goes through open source, ~90% now." "And then the frontier stuff, discovering new use cases, will always go through the most intelligent models." "Planning for that multi-model world, planning to not get stuck in one regime, is the climate."

Molly O’Shea

42,460 Aufrufe • vor 29 Tagen

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀
1:08

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀

Jiaxuan You

75,244 Aufrufe • vor 7 Monaten

Launching today: the world’s first fully optimized LLM routing stack that can be embedded directly into your product, Merge Embedded Routing Stack. Model sovereignty is now table stakes for any AI product. Users and companies are demanding full control over the AI models running in your product. Right now, no company is able to do this without massive overhead. Until today. Merge makes it easy for your customers to use your product exactly the way they want to. With a single API, you can: 1. Allow customers to bring their own keys, manage their own budgets, and set policies like fallback rules, model allowlists, and more 2. Let each customer connect and switch their own models, right inside your product 3. Optimize each customer for cost, performance, or capability Without Merge, you’d be: - Building routing policies, provider integrations, and LLM key management - Giving your users zero model independence - Receiving endless customer support tickets asking “can I use 5.6 Sol?” Products that fight their customers' model strategy will lose to products that fit it. Merge is the only place that allows your users to produce the exact results they want, with the models they have access to. Give your customers the advantage, without the massive overhead costs.
1:14

Launching today: the world’s first fully optimized LLM routing stack that can be embedded directly into your product, Merge Embedded Routing Stack. Model sovereignty is now table stakes for any AI product. Users and companies are demanding full control over the AI models running in your product. Right now, no company is able to do this without massive overhead. Until today. Merge makes it easy for your customers to use your product exactly the way they want to. With a single API, you can: 1. Allow customers to bring their own keys, manage their own budgets, and set policies like fallback rules, model allowlists, and more 2. Let each customer connect and switch their own models, right inside your product 3. Optimize each customer for cost, performance, or capability Without Merge, you’d be: - Building routing policies, provider integrations, and LLM key management - Giving your users zero model independence - Receiving endless customer support tickets asking “can I use 5.6 Sol?” Products that fight their customers' model strategy will lose to products that fit it. Merge is the only place that allows your users to produce the exact results they want, with the models they have access to. Give your customers the advantage, without the massive overhead costs.

Shensi Ding

351,938 Aufrufe • vor 21 Tagen

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.
1:52

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,607 Aufrufe • vor 29 Tagen

Kimi K3 is out. 2.8T parameters. 1.4 TB of weights, frontier-level performance. Serving it on your own infra is the hard part - you need multi-node inference spread across every cluster you can grab capacity from. And then autoscaling, cache-aware routing, PD disaggregation - an entire serving stack around the model. SkyPilot Endpoints ships a complete stack on day 0 for Kimi.ai's Kimi K3: • Runs on GPUs you own - hyperscalers, neoclouds or on-prem. • Production-ready out of the box - autoscaling, KV-aware routing, PD disaggregation, observability • Fault-tolerant by design - endpoint stays up even when GPUs fail Frontier open models, on GPUs you own. 🔗
0:20

Kimi K3 is out. 2.8T parameters. 1.4 TB of weights, frontier-level performance. Serving it on your own infra is the hard part - you need multi-node inference spread across every cluster you can grab capacity from. And then autoscaling, cache-aware routing, PD disaggregation - an entire serving stack around the model. SkyPilot Endpoints ships a complete stack on day 0 for Kimi.ai's Kimi K3: • Runs on GPUs you own - hyperscalers, neoclouds or on-prem. • Production-ready out of the box - autoscaling, KV-aware routing, PD disaggregation, observability • Fault-tolerant by design - endpoint stays up even when GPUs fail Frontier open models, on GPUs you own. 🔗

SkyPilot

14,283 Aufrufe • vor 11 Tagen

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:
0:44

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:

Satya Nadella

907,893 Aufrufe • vor 8 Monaten

Router Chain Mainnet is now LIVE! The wait is over! After years of relentless building, we are proud to announce the launch of Router Chain mainnet. Router Chain is a PoS L1 that allows stateful bridging–enabling truly omnichain dApps that can abstract chains from the users and aggregate liquidity across multiple chains. Key Features of Router Chain: - Maintain states and implement custom business logic directly in the bridging layer. - Flexibility over bridging model, security model, and smart contract platform (EVM or CosmWasm). - Contracts on the Router chain can serve as data aggregation modules for various cross-chain and multi-chain applications. - Rapid and reliable cross-chain message transfers with minimal latency. Connecting over 30+ ecosystems, Router Chain boasts an average block time of 3 seconds, processing over a million transactions worth more than a billion dollars in beta-mainnet through its asset bridge, Nitro. Our journey to decentralize chain abstraction was longer and tougher than anticipated, but the results are redefining Web3 from a fragmented collection to a cohesive ecosystem. Users, participate in the security of the chain now and stake with our introductory staking campaign where you get additional yields: Migrate your old $ROUTE for new ones: Devs, start building omnichain dApps: Discover the full story, explore key features, and understand the power of chain abstraction:
0:34

Router Chain Mainnet is now LIVE! The wait is over! After years of relentless building, we are proud to announce the launch of Router Chain mainnet. Router Chain is a PoS L1 that allows stateful bridging–enabling truly omnichain dApps that can abstract chains from the users and aggregate liquidity across multiple chains. Key Features of Router Chain: - Maintain states and implement custom business logic directly in the bridging layer. - Flexibility over bridging model, security model, and smart contract platform (EVM or CosmWasm). - Contracts on the Router chain can serve as data aggregation modules for various cross-chain and multi-chain applications. - Rapid and reliable cross-chain message transfers with minimal latency. Connecting over 30+ ecosystems, Router Chain boasts an average block time of 3 seconds, processing over a million transactions worth more than a billion dollars in beta-mainnet through its asset bridge, Nitro. Our journey to decentralize chain abstraction was longer and tougher than anticipated, but the results are redefining Web3 from a fragmented collection to a cohesive ecosystem. Users, participate in the security of the chain now and stake with our introductory staking campaign where you get additional yields: Migrate your old $ROUTE for new ones: Devs, start building omnichain dApps: Discover the full story, explore key features, and understand the power of chain abstraction:

Router Protocol

187,957 Aufrufe • vor 2 Jahren

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.
1:24

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,590 Aufrufe • vor 7 Monaten

Chamath is making one of the most important business arguments of 2026. Half of large US companies right now cannot generate returns that exceed their cost of capital, which has normalized back to its long run average of 8 to 11%. Another one in seven companies globally is stuck generating persistent returns between 1 and 5% and most businesses don't have room for error and in this environment walks every frontier AI lab saying the same thing, give us your data, your workflows, your processes and our model will make everything better. And companies by the millions said yes. What they didn't fully account for is what happens on the other side of that door. Every time an employee runs a query through a frontier model API, the prompt goes through external servers, workflows, customer data, pricing logic, internal processes, all of it transmitted through a third party. As Alex Karp said companies are spending on tokens while handing over the exact proprietary advantages that make their business worth owning. Microsoft blocked internal use of Anthropic's Claude Fable 5 but over its 30-day data retention policy and the largest software company in the world decided a frontier model's data handling was too risky for its own employees. A US government action revoked access to another frontier model for foreign nationals overnight. Now here's where the cost math becomes impossible to ignore. Deutsche Bank calculated a roughly 65x cost gap between frontier models like Claude Fable 5 at ~$3.25 per task and open-source alternatives at ~$0.05. For 90% of everyday enterprise tasks, performance is comparable. Open-weight models now match closed frontier systems on core agent tasks at roughly one-tenth the cost, a high-volume deployment that costs $250/day on Claude runs at $12/day on an open-source equivalent. Chamath Palihapitiya tested this directly by running a standard enterprise code migration task through an orchestration layer wrapping an open-source model came in 16.4x cheaper than using a frontier model directly.
5:24

Chamath is making one of the most important business arguments of 2026. Half of large US companies right now cannot generate returns that exceed their cost of capital, which has normalized back to its long run average of 8 to 11%. Another one in seven companies globally is stuck generating persistent returns between 1 and 5% and most businesses don't have room for error and in this environment walks every frontier AI lab saying the same thing, give us your data, your workflows, your processes and our model will make everything better. And companies by the millions said yes. What they didn't fully account for is what happens on the other side of that door. Every time an employee runs a query through a frontier model API, the prompt goes through external servers, workflows, customer data, pricing logic, internal processes, all of it transmitted through a third party. As Alex Karp said companies are spending on tokens while handing over the exact proprietary advantages that make their business worth owning. Microsoft blocked internal use of Anthropic's Claude Fable 5 but over its 30-day data retention policy and the largest software company in the world decided a frontier model's data handling was too risky for its own employees. A US government action revoked access to another frontier model for foreign nationals overnight. Now here's where the cost math becomes impossible to ignore. Deutsche Bank calculated a roughly 65x cost gap between frontier models like Claude Fable 5 at ~$3.25 per task and open-source alternatives at ~$0.05. For 90% of everyday enterprise tasks, performance is comparable. Open-weight models now match closed frontier systems on core agent tasks at roughly one-tenth the cost, a high-volume deployment that costs $250/day on Claude runs at $12/day on an open-source equivalent. Chamath Palihapitiya tested this directly by running a standard enterprise code migration task through an orchestration layer wrapping an open-source model came in 16.4x cheaper than using a frontier model directly.

Milk Road AI

280,929 Aufrufe • vor 1 Monat

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."
2:32

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,486 Aufrufe • vor 2 Monaten

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case. Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few. But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations. OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo. Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.
0:20

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case. Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few. But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations. OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo. Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.

Matthew Lam

120,732 Aufrufe • vor 17 Tagen

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...
2:12

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

kwindla

42,246 Aufrufe • vor 1 Jahr

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!
1:32

New course announcement: Semantic Caching for AI Agents, taught by Tyler Hutcherson and Iliya Zhechev from Redis. Semantic caching can significantly reduce your AI application's inference costs and latency. If someone asks "How do I get a refund?" and another later asks "I want my money back," semantic caching recognizes these mean the same thing so it can use a cached response instead of making another model call. This short course takes you from building your first semantic cache from scratch to implementing production-ready systems using Redis' open-source tools. Skills you'll gain: - Build semantic caches from scratch, then implement them using Redis' SDK with production features - Measure cache performance using hit rate, precision, recall, and latency - Enhance accuracy with threshold tuning, cross-encoders, LLM validation, and fuzzy matching Join and learn to reduce your agentic AI's costs and improve speed!

Andrew Ng

62,497 Aufrufe • vor 8 Monaten

AMD's Anush Elangovan explains why he thinks his company's open source ethos combined with agentic AI superpowers their leverage as a company: Because AMD publishes a lot of technical details about its hardware, when engineers use AI tools, the models already “understand” AMD’s systems and can help write code for them, debug them, or even generate new tools. And that makes developers more productive on AMD hardware without AMD having to do all the work internally. "AMD has had this ethos of open source, which really plays to our advantage. Every frontier model that I use has already seen every bit of AMD source code." "It'll rewrite my spec for me because it's already in the training data. Which you can't get from closed ecosystems." "In fact I built a virtual GPU simulator just based off our public specs, and now I'm running it on the GPU. So now I can run cross-generational GPU simulations on existing hardware." "We have that advantage. And we've run a Dev Day contest where we generated more tokens on AMD — Triton kernels and HIP kernels — than existed on the internet at the time." "So now that's all part of the pre-training data. It's a superpower because now you're open source, and you're agentically accelerating this process."
1:22

AMD's Anush Elangovan explains why he thinks his company's open source ethos combined with agentic AI superpowers their leverage as a company: Because AMD publishes a lot of technical details about its hardware, when engineers use AI tools, the models already “understand” AMD’s systems and can help write code for them, debug them, or even generate new tools. And that makes developers more productive on AMD hardware without AMD having to do all the work internally. "AMD has had this ethos of open source, which really plays to our advantage. Every frontier model that I use has already seen every bit of AMD source code." "It'll rewrite my spec for me because it's already in the training data. Which you can't get from closed ecosystems." "In fact I built a virtual GPU simulator just based off our public specs, and now I'm running it on the GPU. So now I can run cross-generational GPU simulations on existing hardware." "We have that advantage. And we've run a Dev Day contest where we generated more tokens on AMD — Triton kernels and HIP kernels — than existed on the internet at the time." "So now that's all part of the pre-training data. It's a superpower because now you're open source, and you're agentically accelerating this process."

TBPN

34,469 Aufrufe • vor 3 Monaten

Mark Zuckerberg is explaining one of the most misunderstood dynamics in AI and it has direct investment implications (Save this). The concept he's describing is model distillation, and it's one of the most important techniques to emerge in AI over the past year. Here's how it works. You train a massive, enormously expensive model, in Meta's case, Llama 4 Behemoth, a 2 trillion parameter teacher model and then you use that model to teach a much smaller, cheaper model. The smaller model inherits roughly 90 to 95% of the intelligence of the giant while running at 10% of the cost and on a fraction of the compute. Meta already did this with the Llama 4 family and Behemoth serves as the teacher. Llama 4 Scout and Maverick, the publicly released open-source models were distilled from it. Scout runs on a single H100 GPU with a 10 million token context window and outperforms models that cost far more to operate. Maverick, at 17 billion active parameters, rivals DeepSeek V3 in coding at half the parameter count and beats GPT-4o on multimodal benchmarks. Both are completely free for commercial use. What Zuckerberg is pointing at is a structural shift in how AI gets deployed in the real world. Companies aren't taking a frontier model off the shelf and running it as-is but rather taking open-source models, fine-tuning them on their own proprietary data, distilling them into even smaller custom models tailored to their specific use case, and running them on infrastructure they control at a fraction of the cost of a closed frontier API. The investment implication of this is significant and runs in two directions. For Meta specifically, this is a strategic masterstroke. Every company that builds on Llama, fine-tunes it, distills it, or deploys it through their infrastructure is pulling into Meta's orbit while Meta builds the most powerful open teacher model. The ecosystem of companies using it grows and that ecosystem generates commercial activity across Meta's platforms and data services. Meta's AI research benefits from billions of real world deployment signals and it's a flywheel that closed model providers cannot replicate because their strategy requires charging per token, which is now a 65x cost disadvantage against the open-source alternative. For the broader market, distillation changes the economics of inference in a way that has barely been priced in. As intelligence becomes extractable into smaller and cheaper models, the absolute demand for compute doesn't decline but rather it explodes, because now the number of applications that are economically viable expands by orders of magnitude. Every task that was previously too expensive to automate at $3.25 per call becomes viable at $0.05 that means more total token usage, more total GPU utilization, and more demand for the infrastructure companies, the Nebiuses, the GE Vernovas, the Constellation Energies that supply the underlying compute and power.
1:05

Mark Zuckerberg is explaining one of the most misunderstood dynamics in AI and it has direct investment implications (Save this). The concept he's describing is model distillation, and it's one of the most important techniques to emerge in AI over the past year. Here's how it works. You train a massive, enormously expensive model, in Meta's case, Llama 4 Behemoth, a 2 trillion parameter teacher model and then you use that model to teach a much smaller, cheaper model. The smaller model inherits roughly 90 to 95% of the intelligence of the giant while running at 10% of the cost and on a fraction of the compute. Meta already did this with the Llama 4 family and Behemoth serves as the teacher. Llama 4 Scout and Maverick, the publicly released open-source models were distilled from it. Scout runs on a single H100 GPU with a 10 million token context window and outperforms models that cost far more to operate. Maverick, at 17 billion active parameters, rivals DeepSeek V3 in coding at half the parameter count and beats GPT-4o on multimodal benchmarks. Both are completely free for commercial use. What Zuckerberg is pointing at is a structural shift in how AI gets deployed in the real world. Companies aren't taking a frontier model off the shelf and running it as-is but rather taking open-source models, fine-tuning them on their own proprietary data, distilling them into even smaller custom models tailored to their specific use case, and running them on infrastructure they control at a fraction of the cost of a closed frontier API. The investment implication of this is significant and runs in two directions. For Meta specifically, this is a strategic masterstroke. Every company that builds on Llama, fine-tunes it, distills it, or deploys it through their infrastructure is pulling into Meta's orbit while Meta builds the most powerful open teacher model. The ecosystem of companies using it grows and that ecosystem generates commercial activity across Meta's platforms and data services. Meta's AI research benefits from billions of real world deployment signals and it's a flywheel that closed model providers cannot replicate because their strategy requires charging per token, which is now a 65x cost disadvantage against the open-source alternative. For the broader market, distillation changes the economics of inference in a way that has barely been priced in. As intelligence becomes extractable into smaller and cheaper models, the absolute demand for compute doesn't decline but rather it explodes, because now the number of applications that are economically viable expands by orders of magnitude. Every task that was previously too expensive to automate at $3.25 per call becomes viable at $0.05 that means more total token usage, more total GPU utilization, and more demand for the infrastructure companies, the Nebiuses, the GE Vernovas, the Constellation Energies that supply the underlying compute and power.

Milk Road AI

27,908 Aufrufe • vor 1 Monat

🚀Exciting News: The Lit Agent Wallet is Now an elizaOS Plugin! 🚀 We’re thrilled to share that the Lit Agent Wallet—a decentralized system that empowers agents with a private key stored securely across an MPC + TEE network (Lit)—is now available as a plugin for ElizaOS!🎉 This integration brings unparalleled flexibility and security to your decentralized workflows. Whether you're an EOA, a smart account, or a DAO, you can now set tools and policies on-chain that your agents can use, all while ensuring your private keys remain secure and decentralized. 🔒 What Does This Mean for You? >Enhanced Security: Private keys are fragmented and stored across the Lit network, leveraging MPC (Multi-Party Computation) and TEE (Trusted Execution Environment) for maximum security. >On-Chain Control: Set and manage tools, policies, and permissions directly on-chain, giving users and DAOs full control over what your agents can do. >Seamless Integration: As an ElizaOS plugin, the Lit Agent Wallet is now easier than ever to integrate into your existing workflows. Check out the video for a walk through of the setup, configuration, and use cases of the Lit Agent Wallet within ElizaOS. We’ll show you just how easy it is to get started and unlock the full potential of decentralized agents. Get Started Today! Ready to take your agent operations to the next level? Install the Lit Agent Wallet plugin on ElizaOS and experience the future of secure, on-chain agent asset management. 🔗 🔗 Let’s build a more intelligent and decentralized future together! 🌐
3:42

🚀Exciting News: The Lit Agent Wallet is Now an elizaOS Plugin! 🚀 We’re thrilled to share that the Lit Agent Wallet—a decentralized system that empowers agents with a private key stored securely across an MPC + TEE network (Lit)—is now available as a plugin for ElizaOS!🎉 This integration brings unparalleled flexibility and security to your decentralized workflows. Whether you're an EOA, a smart account, or a DAO, you can now set tools and policies on-chain that your agents can use, all while ensuring your private keys remain secure and decentralized. 🔒 What Does This Mean for You? >Enhanced Security: Private keys are fragmented and stored across the Lit network, leveraging MPC (Multi-Party Computation) and TEE (Trusted Execution Environment) for maximum security. >On-Chain Control: Set and manage tools, policies, and permissions directly on-chain, giving users and DAOs full control over what your agents can do. >Seamless Integration: As an ElizaOS plugin, the Lit Agent Wallet is now easier than ever to integrate into your existing workflows. Check out the video for a walk through of the setup, configuration, and use cases of the Lit Agent Wallet within ElizaOS. We’ll show you just how easy it is to get started and unlock the full potential of decentralized agents. Get Started Today! Ready to take your agent operations to the next level? Install the Lit Agent Wallet plugin on ElizaOS and experience the future of secure, on-chain agent asset management. 🔗 🔗 Let’s build a more intelligent and decentralized future together! 🌐

Lit Protocol 🔑

41,989 Aufrufe • vor 1 Jahr

SITUATION EXPLAINED: 70% of frontier model queries could run locally for free. clem 🤗, co-founder and CEO of Hugging Face: "There was an interesting study from Stanford published last year showing that 70% of the queries that people ask to ChatGPT could be accurately answered locally on your laptop. For free." "Most of the AI workloads that people do today with frontier models could be done by models that are cheaper, faster, more customizable, more controllable. And they don't do it because frankly it's a pain to pick the right model." "You're subsidized so you don't have to care because you have your subscription, so you route everything to Einstein. 'Hey Einstein, what's the weather today?'" "In normal life, he would be like, 'I'm not answering your silly question.' But because it's AI and subsidized by the AI labs, all the questions are getting routed to Einstein versus in an ideal world you have different models that are more specialized and better at answering your questions in different domains." "It's possible that routing is gonna redistribute a lot of the value capture from frontier models to a more long tail of models. It's like AI maturing... we are in the first phase where it's very simple, everyone using just one giant model. Now we're moving to the second phase."
3:35

SITUATION EXPLAINED: 70% of frontier model queries could run locally for free. clem 🤗, co-founder and CEO of Hugging Face: "There was an interesting study from Stanford published last year showing that 70% of the queries that people ask to ChatGPT could be accurately answered locally on your laptop. For free." "Most of the AI workloads that people do today with frontier models could be done by models that are cheaper, faster, more customizable, more controllable. And they don't do it because frankly it's a pain to pick the right model." "You're subsidized so you don't have to care because you have your subscription, so you route everything to Einstein. 'Hey Einstein, what's the weather today?'" "In normal life, he would be like, 'I'm not answering your silly question.' But because it's AI and subsidized by the AI labs, all the questions are getting routed to Einstein versus in an ideal world you have different models that are more specialized and better at answering your questions in different domains." "It's possible that routing is gonna redistribute a lot of the value capture from frontier models to a more long tail of models. It's like AI maturing... we are in the first phase where it's very simple, everyone using just one giant model. Now we're moving to the second phase."

MTS

49,884 Aufrufe • vor 1 Monat

If you use Cursor for coding, Claude Code for debugging, and local agents for automation, you almost certainly have a context problem. Every tool acts like an isolated employee. One assistant forgets what the other just learned. The biggest bottleneck in multi-agent workflows right now isn’t reasoning. It’s collective amnesia, combined with zero communication between agents. Memmy just launched a smart fix for this: a local, cross-agent memory layer. Instead of re-pasting your context everywhere, Memmy gives your stack a shared memory hub: → accumulate knowledge once, then use it across Cursor, Codex, and OpenClaw → automatically scan and import your existing agent history, so you don’t have to start from zero → keep your memory local, while model usage depends on your chosen provider or BYOK setup And it isn’t just a memory plugin. It’s a complete personal AI runtime with both MCP and a CLI. Memmy officially launched on Product Hunt, and the team is giving early users experiencing context fragmentation a free trial with 2M GPT tokens. Check it out and support the team with an upvote: → Best part? It’s also open-source for devs who want to build their own agents 🔥 Repo in 🧵↓
3:29

If you use Cursor for coding, Claude Code for debugging, and local agents for automation, you almost certainly have a context problem. Every tool acts like an isolated employee. One assistant forgets what the other just learned. The biggest bottleneck in multi-agent workflows right now isn’t reasoning. It’s collective amnesia, combined with zero communication between agents. Memmy just launched a smart fix for this: a local, cross-agent memory layer. Instead of re-pasting your context everywhere, Memmy gives your stack a shared memory hub: → accumulate knowledge once, then use it across Cursor, Codex, and OpenClaw → automatically scan and import your existing agent history, so you don’t have to start from zero → keep your memory local, while model usage depends on your chosen provider or BYOK setup And it isn’t just a memory plugin. It’s a complete personal AI runtime with both MCP and a CLI. Memmy officially launched on Product Hunt, and the team is giving early users experiencing context fragmentation a free trial with 2M GPT tokens. Check it out and support the team with an upvote: → Best part? It’s also open-source for devs who want to build their own agents 🔥 Repo in 🧵↓

Charly Wargnier

203,456 Aufrufe • vor 7 Tagen