Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or... routing through a classifier. @EVO__HQ can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.show more

Alok Bishoyi

6,009 subscribers

16,327 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie Finanzen

Anya Rossi• Live Now

Private livecam show

44 Kommentare

Profilbild von SKYLINE🥷

SKYLINE🥷vor 3 Monaten

single model workflows are aging fast.

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

we are rolling out early access to EVO Router. Get it here

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

router achieves similar performance of frontier closed source models while costing costing a fraction of it

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

more about our launch here

Profilbild von Mukesh

Mukeshvor 3 Monaten

Congrats!

Profilbild von Boardy

Boardyvor 3 Monaten

@andrewdsouza Alok, this could be a strong fit for the infra and platform teams I know wrestling with model cost and latency. Happy to see who’d be useful to meet.

Profilbild von Lakshmi Narayana

Lakshmi Narayanavor 3 Monaten

Let's go!! I'm done with the generic routers and optimizers that are rerouting based on public benchmarks that have nothing to do with my workloads.

Profilbild von Adarsh

Adarshvor 3 Monaten

WOW

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

🙏🫡

Profilbild von Kshitij Arora

Kshitij Aroravor 3 Monaten

Woah! Can’t wait to try this out, 30-60% reduction is bonkers

Profilbild von Ella | AI Agents Developer

Ella | AI Agents Developervor 3 Monaten

I'd love to see the workloads behind those numbers

Profilbild von Arun Setty Kodavali

Arun Setty Kodavalivor 3 Monaten

Much required!!! When can I get access!?

Profilbild von Saaket Sharma

Saaket Sharmavor 3 Monaten

Lfg!! 🥂🥂

Profilbild von Rahul Parmar

Rahul Parmarvor 3 Monaten

Inference always bites my head, will try!

Profilbild von pH

pHvor 3 Monaten

NotYourAverageRouter™️ 🚀

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

you know it !

Profilbild von Aditya Sharma

Aditya Sharmavor 3 Monaten

with open source catching up to frontier intelligence this is much needed

Profilbild von Ankit

Ankitvor 3 Monaten

congratulations 🥂

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

thanks boss 🙏

Profilbild von Kanak

Kanakvor 3 Monaten

Crazy! Is it based on vllm or something custom inference framework?

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

Hey Kanak, at this moment we mostly do routing orchestration rather than at the inference serving layer. We partner with various inference providers who have the necessary SLAs that meet our customers standards

Profilbild von Sathvik Kollu

Sathvik Kolluvor 3 Monaten

congrats!

Profilbild von Shivam Jaglan

Shivam Jaglanvor 3 Monaten

Much needed!

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

yessir !

Profilbild von Amarnath

Amarnathvor 3 Monaten

Looks promising. Congratulations on the launch Alok.

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

Thank you !!

Profilbild von Yash Khemchandani

Yash Khemchandanivor 3 Monaten

Congratulations!!

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

Thank you !

Profilbild von jay.agent 🤖

jay.agent 🤖vor 3 Monaten

🚀👀

Profilbild von Abhishek Kankani

Abhishek Kankanivor 3 Monaten

This looks awesome @alokbishoyi97! Congrats 🚀

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

Thanks Abhishek !

Profilbild von sandipan

sandipanvor 3 Monaten

Excellent

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

Thank you Sandipan ji 🙏

Profilbild von Vinay Dadi

Vinay Dadivor 3 Monaten

This is super cool. Congratulations

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

thanks vinay !

Profilbild von Gaurav

Gauravvor 3 Monaten

Congratulations on the launch, Alok!

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

Thanks Gaurav!

Profilbild von Abu-Bakr

Abu-Bakrvor 3 Monaten

This is lit

Profilbild von Fajar M Reza

Fajar M Rezavor 3 Monaten

Inference routing matters when production feedback optimizes quality, latency, and cost together.

Profilbild von BLAZE

BLAZEvor 3 Monaten

Wow this is epic!! How long did it take you to make this video?

Profilbild von Alok Bishoyi

Alok Bishoyivor 3 Monaten

hey! A couple of days tbh

Profilbild von jani

janivor 3 Monaten

super cool stuff! congrats 🥂

Profilbild von Biti8

Biti8vor 3 Monaten

$evo

Profilbild von Bharat Jajoo

Bharat Jajoovor 3 Monaten

Congratulations on the launch!

Ähnliche Videos

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Adam Cohen

28,389 Aufrufe • vor 5 Monaten

Introducing Cursor Router, our intelligent model router that selects the right model for the task at hand. Router delivers frontier-quality results at 60% lower cost.

Introducing Cursor Router, our intelligent model router that selects the right model for the task at hand. Router delivers frontier-quality results at 60% lower cost.

Cursor

2,678,471 Aufrufe • vor 3 Monaten

Introducing Smart Routing in Unity AI Gateway. Stop overpaying for AI. Smart Routing matches each coding task to the right model and harness based on what the task needs, so higher-cost models can differentiate on intelligence while lower-cost models differentiate on cost and performance. Match frontier quality and cut task costs by 30%+.

Introducing Smart Routing in Unity AI Gateway. Stop overpaying for AI. Smart Routing matches each coding task to the right model and harness based on what the task needs, so higher-cost models can differentiate on intelligence while lower-cost models differentiate on cost and performance. Match frontier quality and cut task costs by 30%+.

Databricks

126,791 Aufrufe • vor 2 Monaten

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

Akshay 🚀

16,223 Aufrufe • vor 1 Monat

Introducing Entelligence AI Model Router, dynamic turn based routing trained on engineering tasks that cuts costs by over 50% Automatically route every coding turn to the model with the best balance of quality, latency, and cost. Escalate to frontier models when they’re actually needed Balanced mode reduces spend by 46% per session Eco mode helps teams save > 76%

Introducing Entelligence AI Model Router, dynamic turn based routing trained on engineering tasks that cuts costs by over 50% Automatically route every coding turn to the model with the best balance of quality, latency, and cost. Escalate to frontier models when they’re actually needed Balanced mode reduces spend by 46% per session Eco mode helps teams save > 76%

Aiswarya Sankar

87,590 Aufrufe • vor 3 Monaten

The most effective AI systems don't rely on a single model. Frontier models provide state-of-the-art performance for complex tasks, while routers automatically select lightweight, open-source models for simpler jobs to optimize accuracy, latency, and cost. Learn more:

The most effective AI systems don't rely on a single model. Frontier models provide state-of-the-art performance for complex tasks, while routers automatically select lightweight, open-source models for simpler jobs to optimize accuracy, latency, and cost. Learn more:

NVIDIA AI

12,152 Aufrufe • vor 8 Monaten

DigitalOcean Inference Router is now cache-aware. 🔀 Before, switching models mid-conversation meant losing your cache, and the savings and consistency it bought you. Now, Inference Router weighs that cost before switching. Set a model affinity header or routing budget for full control.

DigitalOcean Inference Router is now cache-aware. 🔀 Before, switching models mid-conversation meant losing your cache, and the savings and consistency it bought you. Now, Inference Router weighs that cost before switching. Set a model affinity header or routing budget for full control.

DigitalOcean

318,497 Aufrufe • vor 2 Monaten

🚨 Announcing Smart Model Router - Create Your Own Custom Router To Route To Your Favorite LLM - Route to GPT Sol, Muse Spark, Grok 4.5 and Fable - Optimize for cost or performance - Use Fable only for hard coding Once you create a router, you can use it on our Chat, Agent or in an API! Coming on Monday - Support for all CLI clients including Code, Claude Code and others

🚨 Announcing Smart Model Router - Create Your Own Custom Router To Route To Your Favorite LLM - Route to GPT Sol, Muse Spark, Grok 4.5 and Fable - Optimize for cost or performance - Use Fable only for hard coding Once you create a router, you can use it on our Chat, Agent or in an API! Coming on Monday - Support for all CLI clients including Code, Claude Code and others

Bindu Reddy

2,572,380 Aufrufe • vor 4 Monaten

Introducing model routing to Factory. Factory Router picks the right model for every task, automatically. Maintain frontier performance while cutting costs by 25%.

Introducing model routing to Factory. Factory Router picks the right model for every task, automatically. Maintain frontier performance while cutting costs by 25%.

Factory

2,273,088 Aufrufe • vor 5 Monaten

AI has two sides: the applications that need intelligence, and the computers that provide it. OptimAI Router brings them closer together. For developers: access a growing range of open AI models through one connection. For node operators: turn your computer into a potential source of AI inference, contributing computing resources to real workloads across the network. One side brings the demand. The other brings the capacity. OptimAI connects them. We're building toward an AI network where developers can access diverse models, more computing resources can participate, and intelligence isn't tied to a single provider. Build with open models. Contribute computing power. Help grow the network. One network. From AI requests to distributed compute. OptimAI Router. Coming soon. ⚡

AI has two sides: the applications that need intelligence, and the computers that provide it. OptimAI Router brings them closer together. For developers: access a growing range of open AI models through one connection. For node operators: turn your computer into a potential source of AI inference, contributing computing resources to real workloads across the network. One side brings the demand. The other brings the capacity. OptimAI connects them. We're building toward an AI network where developers can access diverse models, more computing resources can participate, and intelligence isn't tied to a single provider. Build with open models. Contribute computing power. Help grow the network. One network. From AI requests to distributed compute. OptimAI Router. Coming soon. ⚡

OptimAI Network

15,091 Aufrufe • vor 1 Monat

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Veeral Patel

1,432,108 Aufrufe • vor 2 Monaten

Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing. It reads each request, classifies it as simple, medium, or complex, and sends it to the model assigned to that tier. You have two choices: 1. Send every request to the free heuristic classifier to score signals such as technical vocabulary, code, prompt length, and multi-step reasoning. 2. Send the request to an LLM classifier when its difficulty requires a more nuanced judgment. The beauty of using routing is that nothing changes in your code. You still call a single endpoint model, but routing works behind the scenes to pair every request with the best possible model. TrueFoundry ran several experiments with two different setups: 1. Send every request to Claude Opus 2. Send every request to a router with Haiku, Sonnet, and Opus The first experiment ran 550 deterministically graded academic prompts through every setup. Auto Routing was 69% cheaper while retaining 98% of the baseline quality. The second experiment ran three production-shaped workloads through every setup, using user chats, developer chats, and long agent tasks. Auto Routing was 80% cheaper. Thanks to the TrueFoundry team for partnering with me on this post.

Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing. It reads each request, classifies it as simple, medium, or complex, and sends it to the model assigned to that tier. You have two choices: 1. Send every request to the free heuristic classifier to score signals such as technical vocabulary, code, prompt length, and multi-step reasoning. 2. Send the request to an LLM classifier when its difficulty requires a more nuanced judgment. The beauty of using routing is that nothing changes in your code. You still call a single endpoint model, but routing works behind the scenes to pair every request with the best possible model. TrueFoundry ran several experiments with two different setups: 1. Send every request to Claude Opus 2. Send every request to a router with Haiku, Sonnet, and Opus The first experiment ran 550 deterministically graded academic prompts through every setup. Auto Routing was 69% cheaper while retaining 98% of the baseline quality. The second experiment ran three production-shaped workloads through every setup, using user chats, developer chats, and long agent tasks. Auto Routing was 80% cheaper. Thanks to the TrueFoundry team for partnering with me on this post.

Santiago

15,543 Aufrufe • vor 2 Monaten

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and TypeSafe AI The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. Here's how it works 👇🏻

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and TypeSafe AI The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. Here's how it works 👇🏻

OpenRouter

1,334,435 Aufrufe • vor 1 Monat

Inference, training, routing, governance, and sandboxes are the building blocks of every AI product today. We’ve learned a lot from our customers about what it takes to build that system well: OpenEvidence trains models that outperform the closed frontier, then serves them on Dedicated Inference. Datadog, Inc. post-trains on Loops. Opendoor routes every AI request across the company through Grid. This week we have three launches expanding our inference platform: - Grid: gateway, intelligent router, and guardrails for every model your company uses (launching tomorrow) - Sandboxes: stateful, VM-isolated environments for agents that scale to thousands of instances in seconds (launching Thursday) - Loops: serverless SFT and RL on open-weight models, deployable straight to Dedicated Inference (GA Friday) Amir Haghighat, sai, Paul Sinaï, and Charlie O'Neill break down what we're launching. You might notice the visuals have a new look as well.

Inference, training, routing, governance, and sandboxes are the building blocks of every AI product today. We’ve learned a lot from our customers about what it takes to build that system well: OpenEvidence trains models that outperform the closed frontier, then serves them on Dedicated Inference. Datadog, Inc. post-trains on Loops. Opendoor routes every AI request across the company through Grid. This week we have three launches expanding our inference platform: - Grid: gateway, intelligent router, and guardrails for every model your company uses (launching tomorrow) - Sandboxes: stateful, VM-isolated environments for agents that scale to thousands of instances in seconds (launching Thursday) - Loops: serverless SFT and RL on open-weight models, deployable straight to Dedicated Inference (GA Friday) Amir Haghighat, sai, Paul Sinaï, and Charlie O'Neill break down what we're launching. You might notice the visuals have a new look as well.

Baseten

37,826 Aufrufe • vor 7 Tagen

Introducing the Morph Model Router. It chooses the best model for each task in under 50ms. Keep frontier performance while lowering latency and cost. Available today in our API

Introducing the Morph Model Router. It chooses the best model for each task in under 50ms. Keep frontier performance while lowering latency and cost. Available today in our API

Morph

20,281 Aufrufe • vor 5 Monaten

Introducing NVIDIA Nemotron 3 Ultra. A frontier smart open model built for long-running agents that need to plan, reason, use tools and keep working across complex coding, research and enterprise workflows. Up to 5x faster inference and up to 30% lower cost for agentic tasks. Learn more:

Introducing NVIDIA Nemotron 3 Ultra. A frontier smart open model built for long-running agents that need to plan, reason, use tools and keep working across complex coding, research and enterprise workflows. Up to 5x faster inference and up to 30% lower cost for agentic tasks. Learn more:

NVIDIA

230,981 Aufrufe • vor 5 Monaten

Inference providers hate this man... Your LLM deployment can deliver 30 to 50% better performance after initial deployment. If your current inference provider isn't learning your workload and tuning the stack continuously after deployment, you need another provider. Wafer's continual inference agents do that work. Wafer's continual inference agents profile your production workload to identify where scheduling, decoding, or GPU memory pressure limits performance. The agents benchmark changes to batching, serving engines, and GPU kernels against your workload. Each candidate must preserve correctness and meet your reliability targets before it ships. The agents keep profiling as traffic changes, identifying new bottlenecks and further opportunities to improve performance per dollar. Watch emilio andere explain how that work continues after deployment in his conversation with theCUBE.

wafer

42,840 Aufrufe • vor 1 Monat

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

19,009 Aufrufe • vor 4 Monaten

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.

Runway

62,291 Aufrufe • vor 3 Monaten

🚨 Frontier Model Router - Mix And Match All the Frontier LLMs Our most popular custom router is our frontier-model router that combines the best LLMs for all your tasks GPT 5.6 sol - data analysis Fable / Opus 4.8 - coding Flash 3.5 - summarization Grok 4.5 - real-time Pick your favorite LLMs for all your tasks. As of today the custom routers can be used in ANY harness - Abacus AI, Claude, Codex, Open-Code....

🚨 Frontier Model Router - Mix And Match All the Frontier LLMs Our most popular custom router is our frontier-model router that combines the best LLMs for all your tasks GPT 5.6 sol - data analysis Fable / Opus 4.8 - coding Flash 3.5 - summarization Grok 4.5 - real-time Pick your favorite LLMs for all your tasks. As of today the custom routers can be used in ANY harness - Abacus AI, Claude, Codex, Open-Code....

Bindu Reddy

4,592,609 Aufrufe • vor 3 Monaten