Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or... routing through a classifier. @EVO__HQ can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.show more

Alok Bishoyi

6,009 subscribers

16,327 просмотров • 3 месяцев назад •via X (Twitter)

Наука и технологии Финансы

Anya Rossi• Live Now

Private livecam show

Комментарии: 44

Фото профиля SKYLINE🥷

SKYLINE🥷3 месяцев назад

single model workflows are aging fast.

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

we are rolling out early access to EVO Router. Get it here

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

router achieves similar performance of frontier closed source models while costing costing a fraction of it

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

more about our launch here

Фото профиля Mukesh

Mukesh3 месяцев назад

Congrats!

Фото профиля Boardy

Boardy3 месяцев назад

@andrewdsouza Alok, this could be a strong fit for the infra and platform teams I know wrestling with model cost and latency. Happy to see who’d be useful to meet.

Фото профиля Lakshmi Narayana

Lakshmi Narayana3 месяцев назад

Let's go!! I'm done with the generic routers and optimizers that are rerouting based on public benchmarks that have nothing to do with my workloads.

Фото профиля Adarsh

Adarsh3 месяцев назад

WOW

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

🙏🫡

Фото профиля Kshitij Arora

Kshitij Arora3 месяцев назад

Woah! Can’t wait to try this out, 30-60% reduction is bonkers

Фото профиля Ella | AI Agents Developer

Ella | AI Agents Developer3 месяцев назад

I'd love to see the workloads behind those numbers

Фото профиля Arun Setty Kodavali

Arun Setty Kodavali3 месяцев назад

Much required!!! When can I get access!?

Фото профиля Saaket Sharma

Saaket Sharma3 месяцев назад

Lfg!! 🥂🥂

Фото профиля Rahul Parmar

Rahul Parmar3 месяцев назад

Inference always bites my head, will try!

Фото профиля pH

pH3 месяцев назад

NotYourAverageRouter™️ 🚀

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

you know it !

Фото профиля Aditya Sharma

Aditya Sharma3 месяцев назад

with open source catching up to frontier intelligence this is much needed

Фото профиля Ankit

Ankit3 месяцев назад

congratulations 🥂

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

thanks boss 🙏

Фото профиля Kanak

Kanak3 месяцев назад

Crazy! Is it based on vllm or something custom inference framework?

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

Hey Kanak, at this moment we mostly do routing orchestration rather than at the inference serving layer. We partner with various inference providers who have the necessary SLAs that meet our customers standards

Фото профиля Sathvik Kollu

Sathvik Kollu3 месяцев назад

congrats!

Фото профиля Shivam Jaglan

Shivam Jaglan3 месяцев назад

Much needed!

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

yessir !

Фото профиля Amarnath

Amarnath3 месяцев назад

Looks promising. Congratulations on the launch Alok.

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

Thank you !!

Фото профиля Yash Khemchandani

Yash Khemchandani3 месяцев назад

Congratulations!!

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

Thank you !

Фото профиля jay.agent 🤖

jay.agent 🤖3 месяцев назад

🚀👀

Фото профиля Abhishek Kankani

Abhishek Kankani3 месяцев назад

This looks awesome @alokbishoyi97! Congrats 🚀

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

Thanks Abhishek !

Фото профиля sandipan

sandipan3 месяцев назад

Excellent

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

Thank you Sandipan ji 🙏

Фото профиля Vinay Dadi

Vinay Dadi3 месяцев назад

This is super cool. Congratulations

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

thanks vinay !

Фото профиля Gaurav

Gaurav3 месяцев назад

Congratulations on the launch, Alok!

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

Thanks Gaurav!

Фото профиля Abu-Bakr

Abu-Bakr3 месяцев назад

This is lit

Фото профиля Fajar M Reza

Fajar M Reza3 месяцев назад

Inference routing matters when production feedback optimizes quality, latency, and cost together.

Фото профиля BLAZE

BLAZE3 месяцев назад

Wow this is epic!! How long did it take you to make this video?

Фото профиля Alok Bishoyi

Alok Bishoyi3 месяцев назад

hey! A couple of days tbh

Фото профиля jani

jani3 месяцев назад

super cool stuff! congrats 🥂

Фото профиля Biti8

Biti83 месяцев назад

$evo

Фото профиля Bharat Jajoo

Bharat Jajoo3 месяцев назад

Congratulations on the launch!

Похожие видео

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Adam Cohen

28,409 просмотров • 5 месяцев назад

Introducing Cursor Router, our intelligent model router that selects the right model for the task at hand. Router delivers frontier-quality results at 60% lower cost.

Introducing Cursor Router, our intelligent model router that selects the right model for the task at hand. Router delivers frontier-quality results at 60% lower cost.

Cursor

2,679,541 просмотров • 3 месяцев назад

Introducing Smart Routing in Unity AI Gateway. Stop overpaying for AI. Smart Routing matches each coding task to the right model and harness based on what the task needs, so higher-cost models can differentiate on intelligence while lower-cost models differentiate on cost and performance. Match frontier quality and cut task costs by 30%+.

Introducing Smart Routing in Unity AI Gateway. Stop overpaying for AI. Smart Routing matches each coding task to the right model and harness based on what the task needs, so higher-cost models can differentiate on intelligence while lower-cost models differentiate on cost and performance. Match frontier quality and cut task costs by 30%+.

Databricks

126,791 просмотров • 3 месяцев назад

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

Akshay 🚀

16,223 просмотров • 1 месяц назад

Introducing Entelligence AI Model Router, dynamic turn based routing trained on engineering tasks that cuts costs by over 50% Automatically route every coding turn to the model with the best balance of quality, latency, and cost. Escalate to frontier models when they’re actually needed Balanced mode reduces spend by 46% per session Eco mode helps teams save > 76%

Introducing Entelligence AI Model Router, dynamic turn based routing trained on engineering tasks that cuts costs by over 50% Automatically route every coding turn to the model with the best balance of quality, latency, and cost. Escalate to frontier models when they’re actually needed Balanced mode reduces spend by 46% per session Eco mode helps teams save > 76%

Aiswarya Sankar

87,590 просмотров • 4 месяцев назад

The most effective AI systems don't rely on a single model. Frontier models provide state-of-the-art performance for complex tasks, while routers automatically select lightweight, open-source models for simpler jobs to optimize accuracy, latency, and cost. Learn more:

The most effective AI systems don't rely on a single model. Frontier models provide state-of-the-art performance for complex tasks, while routers automatically select lightweight, open-source models for simpler jobs to optimize accuracy, latency, and cost. Learn more:

NVIDIA AI

12,152 просмотров • 8 месяцев назад

DigitalOcean Inference Router is now cache-aware. 🔀 Before, switching models mid-conversation meant losing your cache, and the savings and consistency it bought you. Now, Inference Router weighs that cost before switching. Set a model affinity header or routing budget for full control.

DigitalOcean Inference Router is now cache-aware. 🔀 Before, switching models mid-conversation meant losing your cache, and the savings and consistency it bought you. Now, Inference Router weighs that cost before switching. Set a model affinity header or routing budget for full control.

DigitalOcean

318,497 просмотров • 2 месяцев назад

🚨 Announcing Smart Model Router - Create Your Own Custom Router To Route To Your Favorite LLM - Route to GPT Sol, Muse Spark, Grok 4.5 and Fable - Optimize for cost or performance - Use Fable only for hard coding Once you create a router, you can use it on our Chat, Agent or in an API! Coming on Monday - Support for all CLI clients including Code, Claude Code and others

🚨 Announcing Smart Model Router - Create Your Own Custom Router To Route To Your Favorite LLM - Route to GPT Sol, Muse Spark, Grok 4.5 and Fable - Optimize for cost or performance - Use Fable only for hard coding Once you create a router, you can use it on our Chat, Agent or in an API! Coming on Monday - Support for all CLI clients including Code, Claude Code and others

Bindu Reddy

2,572,380 просмотров • 4 месяцев назад

Introducing model routing to Factory. Factory Router picks the right model for every task, automatically. Maintain frontier performance while cutting costs by 25%.

Introducing model routing to Factory. Factory Router picks the right model for every task, automatically. Maintain frontier performance while cutting costs by 25%.

Factory

2,273,630 просмотров • 5 месяцев назад

AI has two sides: the applications that need intelligence, and the computers that provide it. OptimAI Router brings them closer together. For developers: access a growing range of open AI models through one connection. For node operators: turn your computer into a potential source of AI inference, contributing computing resources to real workloads across the network. One side brings the demand. The other brings the capacity. OptimAI connects them. We're building toward an AI network where developers can access diverse models, more computing resources can participate, and intelligence isn't tied to a single provider. Build with open models. Contribute computing power. Help grow the network. One network. From AI requests to distributed compute. OptimAI Router. Coming soon. ⚡

AI has two sides: the applications that need intelligence, and the computers that provide it. OptimAI Router brings them closer together. For developers: access a growing range of open AI models through one connection. For node operators: turn your computer into a potential source of AI inference, contributing computing resources to real workloads across the network. One side brings the demand. The other brings the capacity. OptimAI connects them. We're building toward an AI network where developers can access diverse models, more computing resources can participate, and intelligence isn't tied to a single provider. Build with open models. Contribute computing power. Help grow the network. One network. From AI requests to distributed compute. OptimAI Router. Coming soon. ⚡

OptimAI Network

15,139 просмотров • 1 месяц назад

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Veeral Patel

1,432,352 просмотров • 2 месяцев назад

Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing. It reads each request, classifies it as simple, medium, or complex, and sends it to the model assigned to that tier. You have two choices: 1. Send every request to the free heuristic classifier to score signals such as technical vocabulary, code, prompt length, and multi-step reasoning. 2. Send the request to an LLM classifier when its difficulty requires a more nuanced judgment. The beauty of using routing is that nothing changes in your code. You still call a single endpoint model, but routing works behind the scenes to pair every request with the best possible model. TrueFoundry ran several experiments with two different setups: 1. Send every request to Claude Opus 2. Send every request to a router with Haiku, Sonnet, and Opus The first experiment ran 550 deterministically graded academic prompts through every setup. Auto Routing was 69% cheaper while retaining 98% of the baseline quality. The second experiment ran three production-shaped workloads through every setup, using user chats, developer chats, and long agent tasks. Auto Routing was 80% cheaper. Thanks to the TrueFoundry team for partnering with me on this post.

Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing. It reads each request, classifies it as simple, medium, or complex, and sends it to the model assigned to that tier. You have two choices: 1. Send every request to the free heuristic classifier to score signals such as technical vocabulary, code, prompt length, and multi-step reasoning. 2. Send the request to an LLM classifier when its difficulty requires a more nuanced judgment. The beauty of using routing is that nothing changes in your code. You still call a single endpoint model, but routing works behind the scenes to pair every request with the best possible model. TrueFoundry ran several experiments with two different setups: 1. Send every request to Claude Opus 2. Send every request to a router with Haiku, Sonnet, and Opus The first experiment ran 550 deterministically graded academic prompts through every setup. Auto Routing was 69% cheaper while retaining 98% of the baseline quality. The second experiment ran three production-shaped workloads through every setup, using user chats, developer chats, and long agent tasks. Auto Routing was 80% cheaper. Thanks to the TrueFoundry team for partnering with me on this post.

Santiago

15,543 просмотров • 2 месяцев назад

Inference, training, routing, governance, and sandboxes are the building blocks of every AI product today. We’ve learned a lot from our customers about what it takes to build that system well: OpenEvidence trains models that outperform the closed frontier, then serves them on Dedicated Inference. Datadog, Inc. post-trains on Loops. Opendoor routes every AI request across the company through Grid. This week we have three launches expanding our inference platform: - Grid: gateway, intelligent router, and guardrails for every model your company uses (launching tomorrow) - Sandboxes: stateful, VM-isolated environments for agents that scale to thousands of instances in seconds (launching Thursday) - Loops: serverless SFT and RL on open-weight models, deployable straight to Dedicated Inference (GA Friday) Amir Haghighat, sai, Paul Sinaï, and Charlie O'Neill break down what we're launching. You might notice the visuals have a new look as well.

Inference, training, routing, governance, and sandboxes are the building blocks of every AI product today. We’ve learned a lot from our customers about what it takes to build that system well: OpenEvidence trains models that outperform the closed frontier, then serves them on Dedicated Inference. Datadog, Inc. post-trains on Loops. Opendoor routes every AI request across the company through Grid. This week we have three launches expanding our inference platform: - Grid: gateway, intelligent router, and guardrails for every model your company uses (launching tomorrow) - Sandboxes: stateful, VM-isolated environments for agents that scale to thousands of instances in seconds (launching Thursday) - Loops: serverless SFT and RL on open-weight models, deployable straight to Dedicated Inference (GA Friday) Amir Haghighat, sai, Paul Sinaï, and Charlie O'Neill break down what we're launching. You might notice the visuals have a new look as well.

Baseten

40,425 просмотров • 11 дней назад

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and TypeSafe AI The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. Here's how it works 👇🏻

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and TypeSafe AI The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. Here's how it works 👇🏻

OpenRouter

1,334,841 просмотров • 1 месяц назад

Introducing the Morph Model Router. It chooses the best model for each task in under 50ms. Keep frontier performance while lowering latency and cost. Available today in our API

Introducing the Morph Model Router. It chooses the best model for each task in under 50ms. Keep frontier performance while lowering latency and cost. Available today in our API

Morph

20,281 просмотров • 5 месяцев назад

Introducing NVIDIA Nemotron 3 Ultra. A frontier smart open model built for long-running agents that need to plan, reason, use tools and keep working across complex coding, research and enterprise workflows. Up to 5x faster inference and up to 30% lower cost for agentic tasks. Learn more:

Introducing NVIDIA Nemotron 3 Ultra. A frontier smart open model built for long-running agents that need to plan, reason, use tools and keep working across complex coding, research and enterprise workflows. Up to 5x faster inference and up to 30% lower cost for agentic tasks. Learn more:

NVIDIA

230,981 просмотров • 5 месяцев назад

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

19,017 просмотров • 4 месяцев назад

Inference providers hate this man... Your LLM deployment can deliver 30 to 50% better performance after initial deployment. If your current inference provider isn't learning your workload and tuning the stack continuously after deployment, you need another provider. Wafer's continual inference agents do that work. Wafer's continual inference agents profile your production workload to identify where scheduling, decoding, or GPU memory pressure limits performance. The agents benchmark changes to batching, serving engines, and GPU kernels against your workload. Each candidate must preserve correctness and meet your reliability targets before it ships. The agents keep profiling as traffic changes, identifying new bottlenecks and further opportunities to improve performance per dollar. Watch emilio andere explain how that work continues after deployment in his conversation with theCUBE.

wafer

42,840 просмотров • 1 месяц назад

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.

Runway

62,291 просмотров • 3 месяцев назад

We evaluated Nexus on 7 benchmarks and 1,222 tasks. It defines the Pareto frontier. Mix open and closed models. Get better intelligence for 4x lower cost than a single frontier model. The smartest way to extend your Claude, ChatGPT, or Codex plan, starting at $8/mo.

We evaluated Nexus on 7 benchmarks and 1,222 tasks. It defines the Pareto frontier. Mix open and closed models. Get better intelligence for 4x lower cost than a single frontier model. The smartest way to extend your Claude, ChatGPT, or Codex plan, starting at $8/mo.

Fireworks

592,004 просмотров • 7 дней назад