Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or... routing through a classifier. EVO (YC F26) can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.show more

Alok Bishoyi

5,248 subscribers

16,049 Aufrufe • vor 1 Monat •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

44 Kommentare

Profilbild von SKYLINE🥷

SKYLINE🥷vor 1 Monat

single model workflows are aging fast.

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

we are rolling out early access to EVO Router. Get it here

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

router achieves similar performance of frontier closed source models while costing costing a fraction of it

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

more about our launch here

Profilbild von Mukesh

Mukeshvor 1 Monat

Congrats!

Profilbild von Boardy

Boardyvor 1 Monat

@andrewdsouza Alok, this could be a strong fit for the infra and platform teams I know wrestling with model cost and latency. Happy to see who’d be useful to meet.

Profilbild von Lakshmi Narayana

Lakshmi Narayanavor 1 Monat

Let's go!! I'm done with the generic routers and optimizers that are rerouting based on public benchmarks that have nothing to do with my workloads.

Profilbild von Adarsh

Adarshvor 1 Monat

WOW

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

🙏🫡

Profilbild von Kshitij Arora

Kshitij Aroravor 1 Monat

Woah! Can’t wait to try this out, 30-60% reduction is bonkers

Profilbild von Ella | AI Agents Developer

Ella | AI Agents Developervor 1 Monat

I'd love to see the workloads behind those numbers

Profilbild von Arun Setty Kodavali

Arun Setty Kodavalivor 1 Monat

Much required!!! When can I get access!?

Profilbild von Saaket Sharma

Saaket Sharmavor 1 Monat

Lfg!! 🥂🥂

Profilbild von Rahul Parmar

Rahul Parmarvor 1 Monat

Inference always bites my head, will try!

Profilbild von pH

pHvor 1 Monat

NotYourAverageRouter™️ 🚀

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

you know it !

Profilbild von Aditya Sharma

Aditya Sharmavor 1 Monat

with open source catching up to frontier intelligence this is much needed

Profilbild von Ankit

Ankitvor 1 Monat

congratulations 🥂

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

thanks boss 🙏

Profilbild von Kanak

Kanakvor 1 Monat

Crazy! Is it based on vllm or something custom inference framework?

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

Hey Kanak, at this moment we mostly do routing orchestration rather than at the inference serving layer. We partner with various inference providers who have the necessary SLAs that meet our customers standards

Profilbild von Sathvik Kollu

Sathvik Kolluvor 1 Monat

congrats!

Profilbild von Shivam Jaglan

Shivam Jaglanvor 1 Monat

Much needed!

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

yessir !

Profilbild von Amarnath

Amarnathvor 1 Monat

Looks promising. Congratulations on the launch Alok.

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

Thank you !!

Profilbild von Yash Khemchandani

Yash Khemchandanivor 1 Monat

Congratulations!!

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

Thank you !

Profilbild von jay.agent 🤖

jay.agent 🤖vor 1 Monat

🚀👀

Profilbild von Abhishek Kankani

Abhishek Kankanivor 1 Monat

This looks awesome @alokbishoyi97! Congrats 🚀

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

Thanks Abhishek !

Profilbild von sandipan

sandipanvor 1 Monat

Excellent

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

Thank you Sandipan ji 🙏

Profilbild von Vinay Dadi

Vinay Dadivor 1 Monat

This is super cool. Congratulations

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

thanks vinay !

Profilbild von Gaurav

Gauravvor 1 Monat

Congratulations on the launch, Alok!

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

Thanks Gaurav!

Profilbild von Abu-Bakr

Abu-Bakrvor 1 Monat

This is lit

Profilbild von Fajar M Reza

Fajar M Rezavor 1 Monat

Inference routing matters when production feedback optimizes quality, latency, and cost together.

Profilbild von BLAZE

BLAZEvor 1 Monat

Wow this is epic!! How long did it take you to make this video?

Profilbild von Alok Bishoyi

Alok Bishoyivor 1 Monat

hey! A couple of days tbh

Profilbild von jani

janivor 1 Monat

super cool stuff! congrats 🥂

Profilbild von Biti8

Biti8vor 1 Monat

$evo

Profilbild von Bharat Jajoo

Bharat Jajoovor 1 Monat

Congratulations on the launch!

Ähnliche Videos

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Weave is launching the number 1 prompt router in the world. It enables you to get 70% more efficient use of your tokens. We analyzed millions of prompts and found that the vast majority don't need a frontier model. Weave Router fixes that. It analyzes your prompt and routes it to the highest quality model with the lowest cost (across open and closed source models). This all happens in your current workflow on Claude, Cursor or Codex so you don't have to change a thing. Early customers have seen an ~70% reduction in costs without any slowdowns. Source code available.

Adam Cohen

28,048 Aufrufe • vor 4 Monaten

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Veeral Patel

1,425,405 Aufrufe • vor 1 Monat

Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing. It reads each request, classifies it as simple, medium, or complex, and sends it to the model assigned to that tier. You have two choices: 1. Send every request to the free heuristic classifier to score signals such as technical vocabulary, code, prompt length, and multi-step reasoning. 2. Send the request to an LLM classifier when its difficulty requires a more nuanced judgment. The beauty of using routing is that nothing changes in your code. You still call a single endpoint model, but routing works behind the scenes to pair every request with the best possible model. TrueFoundry ran several experiments with two different setups: 1. Send every request to Claude Opus 2. Send every request to a router with Haiku, Sonnet, and Opus The first experiment ran 550 deterministically graded academic prompts through every setup. Auto Routing was 69% cheaper while retaining 98% of the baseline quality. The second experiment ran three production-shaped workloads through every setup, using user chats, developer chats, and long agent tasks. Auto Routing was 80% cheaper. Thanks to the TrueFoundry team for partnering with me on this post.

Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing. It reads each request, classifies it as simple, medium, or complex, and sends it to the model assigned to that tier. You have two choices: 1. Send every request to the free heuristic classifier to score signals such as technical vocabulary, code, prompt length, and multi-step reasoning. 2. Send the request to an LLM classifier when its difficulty requires a more nuanced judgment. The beauty of using routing is that nothing changes in your code. You still call a single endpoint model, but routing works behind the scenes to pair every request with the best possible model. TrueFoundry ran several experiments with two different setups: 1. Send every request to Claude Opus 2. Send every request to a router with Haiku, Sonnet, and Opus The first experiment ran 550 deterministically graded academic prompts through every setup. Auto Routing was 69% cheaper while retaining 98% of the baseline quality. The second experiment ran three production-shaped workloads through every setup, using user chats, developer chats, and long agent tasks. Auto Routing was 80% cheaper. Thanks to the TrueFoundry team for partnering with me on this post.

Santiago

15,543 Aufrufe • vor 15 Tagen

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,068 Aufrufe • vor 1 Monat

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

239,092 Aufrufe • vor 23 Tagen

We just cut AI spend by 75% across an entire workforce in one click. Introducing: Merge for Workforce. Without Merge, your employees are left choosing one of two options: 1. Spend 100x the cost they should be from using frontier models for everything 2. Deliver subpar work from overly restrictive model access Instead with Merge, IT can connect any identity provider and set model routing policies by team in one click. Merge pushes them to every machine through a desktop client, overriding the model configs across your team’s AI assistants and coding tools. Now every task gets the right model for the job. The result? 75x fewer tokens with faster and better output.

We just cut AI spend by 75% across an entire workforce in one click. Introducing: Merge for Workforce. Without Merge, your employees are left choosing one of two options: 1. Spend 100x the cost they should be from using frontier models for everything 2. Deliver subpar work from overly restrictive model access Instead with Merge, IT can connect any identity provider and set model routing policies by team in one click. Merge pushes them to every machine through a desktop client, overriding the model configs across your team’s AI assistants and coding tools. Now every task gets the right model for the job. The result? 75x fewer tokens with faster and better output.

Shensi Ding

523,902 Aufrufe • vor 1 Monat

🚨 The Silvia team just announced our latest engineering advancement. Every business wants access to the highest level of intelligence, but at the lowest cost possible. The rise of LLMs has made intelligence abundant, yet one of the hardest problems across startups and corporate America is predicting the compute cost associated with this intelligence. I have been dealing with this personally as we build Silvia and the problem comes up in almost every conversation I have with CEOs, founders, and executives. Every business embraced AI about 18 months ago and things seemed great until the compute bills started to show up. The bills for internal compute usage were difficult to swallow, but things got outrageous if you had an AI product that allowed your users to consume compute without limits. I know this problem intimately because that is the situation that Silvia was in. Every question that was asked meant higher compute costs for our company. But we didn’t want to limit usage because users were getting genuine value out of the product. This challenge sent our team down a deep rabbit hole of cutting costs, while improving the experience for users. The second part was really important: we did not want to degrade the user experience by simply taking away access to the highest quality models. Thankfully, resource constraints breed innovation. We aren’t the biggest company, nor do we have the largest balance sheet, but we came up with a very novel solution that we are announcing today. The Silvia engineering team built a model router that cut costs by up to 29%, decreased latency, and improved the quality of answers for users. Trifecta! The way we do this is by reading the first 500 characters of a query and then predicting the level of effort that will be needed by a model to answer the query. The highest effort needs are routed to the most powerful models. The lowest effort needs are routed to different, better models for the query. A good example of this would be “what is the date?” You don’t need to use the latest Anthropic model to answer this query. In fact, sending a simple query like this to the most powerful model will make your compute costs increase and will actually increase the latency, which means a worse user experience for the Silvia user. By implementing the model router, the user gets a better experience and we get lower costs. Win-win. One of the interesting aspects of the implementation is that our model router runs on CPUs instead of GPUs. This allows us to read the query and predict the level of effort needed in less than 1 millisecond. This CPU implementation is why latency is not affected, nor is cost significantly increased by any potential additional GPU consumption. Another important point is that many of you have probably seen the news that OpenRouter is being purchased by Stripe for around $7 billion. This is a great outcome from what appears to be a very smart, capable team. Their model routing API is related (their product and our internal implementation both touch model routing), but you should think of OpenRouter as making it possible to do model routing for companies, while Silvia’s model router is a custom, intelligent system that specifically routes Silvia queries to the right model. They give access to the functionality of model routing to many companies, while our internal product does the real decision-making specific to our use case. Lastly, our implementation of a model router is a strategic bet that will allow us to become model-agnostic over time. We don’t care who created the different models, we just want to route a query to the model best positioned to answer. The large model labs will never allow their users to be model agnostic, but that would require the lab to potentially route a query to a competitor’s model. No bueno in their eyes. Instead, Silvia being an independent AI research lab gives us the power of being agnostic. We simply want the best experience for our users. Last week we announced that Silvia is now the most accurate AI tax product on the market, including beating OpenAI, Anthropic, Google, and xAI. Today we are announcing a custom, in-house model router that rivals the best technology anyone else has built. There will be many more engineering announcements to come. I truly believe we have assembled one of the best AI teams and we are currently the best AI research lab in finance. If you are interested in learning more about the technical details of the model router, you can read the engineering blog post here: Everyone wants the best intelligence and the lowest cost. Silvia just showed the world what is possible in this pursuit. I anticipate many other companies will build this custom solutions to achieve the same benefits.

🚨 The Silvia team just announced our latest engineering advancement. Every business wants access to the highest level of intelligence, but at the lowest cost possible. The rise of LLMs has made intelligence abundant, yet one of the hardest problems across startups and corporate America is predicting the compute cost associated with this intelligence. I have been dealing with this personally as we build Silvia and the problem comes up in almost every conversation I have with CEOs, founders, and executives. Every business embraced AI about 18 months ago and things seemed great until the compute bills started to show up. The bills for internal compute usage were difficult to swallow, but things got outrageous if you had an AI product that allowed your users to consume compute without limits. I know this problem intimately because that is the situation that Silvia was in. Every question that was asked meant higher compute costs for our company. But we didn’t want to limit usage because users were getting genuine value out of the product. This challenge sent our team down a deep rabbit hole of cutting costs, while improving the experience for users. The second part was really important: we did not want to degrade the user experience by simply taking away access to the highest quality models. Thankfully, resource constraints breed innovation. We aren’t the biggest company, nor do we have the largest balance sheet, but we came up with a very novel solution that we are announcing today. The Silvia engineering team built a model router that cut costs by up to 29%, decreased latency, and improved the quality of answers for users. Trifecta! The way we do this is by reading the first 500 characters of a query and then predicting the level of effort that will be needed by a model to answer the query. The highest effort needs are routed to the most powerful models. The lowest effort needs are routed to different, better models for the query. A good example of this would be “what is the date?” You don’t need to use the latest Anthropic model to answer this query. In fact, sending a simple query like this to the most powerful model will make your compute costs increase and will actually increase the latency, which means a worse user experience for the Silvia user. By implementing the model router, the user gets a better experience and we get lower costs. Win-win. One of the interesting aspects of the implementation is that our model router runs on CPUs instead of GPUs. This allows us to read the query and predict the level of effort needed in less than 1 millisecond. This CPU implementation is why latency is not affected, nor is cost significantly increased by any potential additional GPU consumption. Another important point is that many of you have probably seen the news that OpenRouter is being purchased by Stripe for around $7 billion. This is a great outcome from what appears to be a very smart, capable team. Their model routing API is related (their product and our internal implementation both touch model routing), but you should think of OpenRouter as making it possible to do model routing for companies, while Silvia’s model router is a custom, intelligent system that specifically routes Silvia queries to the right model. They give access to the functionality of model routing to many companies, while our internal product does the real decision-making specific to our use case. Lastly, our implementation of a model router is a strategic bet that will allow us to become model-agnostic over time. We don’t care who created the different models, we just want to route a query to the model best positioned to answer. The large model labs will never allow their users to be model agnostic, but that would require the lab to potentially route a query to a competitor’s model. No bueno in their eyes. Instead, Silvia being an independent AI research lab gives us the power of being agnostic. We simply want the best experience for our users. Last week we announced that Silvia is now the most accurate AI tax product on the market, including beating OpenAI, Anthropic, Google, and xAI. Today we are announcing a custom, in-house model router that rivals the best technology anyone else has built. There will be many more engineering announcements to come. I truly believe we have assembled one of the best AI teams and we are currently the best AI research lab in finance. If you are interested in learning more about the technical details of the model router, you can read the engineering blog post here: Everyone wants the best intelligence and the lowest cost. Silvia just showed the world what is possible in this pursuit. I anticipate many other companies will build this custom solutions to achieve the same benefits.

Anthony Pompliano 🌪

76,176 Aufrufe • vor 1 Monat

BREAKING: Apoorv Agrawal (Apoorv Agrawal) of Altimeter Capital says there are "four seasons" in AI: "They're called OpenAI, Anthropic, SpaceX, & Google." "So if you're a CIO, CEO, if you're about to make a multi-million, billion-dollar decision of which lab to build your intelligence on, plan for the climate, not for the weather." What does that mean? "That's multi-model routing. That's evals. That's thinking about post-train custom models." "You'll have a portion of your workloads going through open source, as Jesse Zhang from Decagon wrote very eloquently. A big portion of their volume goes through open source, ~90% now." "And then the frontier stuff, discovering new use cases, will always go through the most intelligent models." "Planning for that multi-model world, planning to not get stuck in one regime, is the climate."

BREAKING: Apoorv Agrawal (Apoorv Agrawal) of Altimeter Capital says there are "four seasons" in AI: "They're called OpenAI, Anthropic, SpaceX, & Google." "So if you're a CIO, CEO, if you're about to make a multi-million, billion-dollar decision of which lab to build your intelligence on, plan for the climate, not for the weather." What does that mean? "That's multi-model routing. That's evals. That's thinking about post-train custom models." "You'll have a portion of your workloads going through open source, as Jesse Zhang from Decagon wrote very eloquently. A big portion of their volume goes through open source, ~90% now." "And then the frontier stuff, discovering new use cases, will always go through the most intelligent models." "Planning for that multi-model world, planning to not get stuck in one regime, is the climate."

Molly O’Shea

42,460 Aufrufe • vor 2 Monaten

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀

Jiaxuan You

75,244 Aufrufe • vor 9 Monaten

Launching today: the world’s first fully optimized LLM routing stack that can be embedded directly into your product, Merge Embedded Routing Stack. Model sovereignty is now table stakes for any AI product. Users and companies are demanding full control over the AI models running in your product. Right now, no company is able to do this without massive overhead. Until today. Merge makes it easy for your customers to use your product exactly the way they want to. With a single API, you can: 1. Allow customers to bring their own keys, manage their own budgets, and set policies like fallback rules, model allowlists, and more 2. Let each customer connect and switch their own models, right inside your product 3. Optimize each customer for cost, performance, or capability Without Merge, you’d be: - Building routing policies, provider integrations, and LLM key management - Giving your users zero model independence - Receiving endless customer support tickets asking “can I use 5.6 Sol?” Products that fight their customers' model strategy will lose to products that fit it. Merge is the only place that allows your users to produce the exact results they want, with the models they have access to. Give your customers the advantage, without the massive overhead costs.

Launching today: the world’s first fully optimized LLM routing stack that can be embedded directly into your product, Merge Embedded Routing Stack. Model sovereignty is now table stakes for any AI product. Users and companies are demanding full control over the AI models running in your product. Right now, no company is able to do this without massive overhead. Until today. Merge makes it easy for your customers to use your product exactly the way they want to. With a single API, you can: 1. Allow customers to bring their own keys, manage their own budgets, and set policies like fallback rules, model allowlists, and more 2. Let each customer connect and switch their own models, right inside your product 3. Optimize each customer for cost, performance, or capability Without Merge, you’d be: - Building routing policies, provider integrations, and LLM key management - Giving your users zero model independence - Receiving endless customer support tickets asking “can I use 5.6 Sol?” Products that fight their customers' model strategy will lose to products that fit it. Merge is the only place that allows your users to produce the exact results they want, with the models they have access to. Give your customers the advantage, without the massive overhead costs.

Shensi Ding

357,306 Aufrufe • vor 2 Monaten

Today we're incredibly excited to announce Dreamscale Labs (YC F26) (YC F26)! We're an applied research company putting robot brains in the cloud. Robot AI models are growing larger and on-board compute is getting increasingly inaccessible due to cost. We're taking a contrarian bet that the future is moving inference off the robot and into the cloud, while still meeting real-time deadlines required by robot motion. We're building the real-time inference infrastructure that enables frontier physical AI models to run smoothly on your robot. We optimize for low latency across the full stack: from the model, down to the GPU kernels, and over the network. Our mission is to unlock a future of safe and intelligent robots doing meaningful work, and we think this is only possible by letting you deploy and run robot models in the cloud. Reach out to us at founders@dreamscalelabs.com or visit our website at to learn more. Kevin L Antoine Nguyen Eric R

Today we're incredibly excited to announce Dreamscale Labs (YC F26) (YC F26)! We're an applied research company putting robot brains in the cloud. Robot AI models are growing larger and on-board compute is getting increasingly inaccessible due to cost. We're taking a contrarian bet that the future is moving inference off the robot and into the cloud, while still meeting real-time deadlines required by robot motion. We're building the real-time inference infrastructure that enables frontier physical AI models to run smoothly on your robot. We optimize for low latency across the full stack: from the model, down to the GPU kernels, and over the network. Our mission is to unlock a future of safe and intelligent robots doing meaningful work, and we think this is only possible by letting you deploy and run robot models in the cloud. Reach out to us at [email protected] or visit our website at to learn more. Kevin L Antoine Nguyen Eric R

Chris Yoo

47,304 Aufrufe • vor 8 Tagen

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,854 Aufrufe • vor 2 Monaten

Kimi K3 is out. 2.8T parameters. 1.4 TB of weights, frontier-level performance. Serving it on your own infra is the hard part - you need multi-node inference spread across every cluster you can grab capacity from. And then autoscaling, cache-aware routing, PD disaggregation - an entire serving stack around the model. SkyPilot Endpoints ships a complete stack on day 0 for Kimi.ai's Kimi K3: • Runs on GPUs you own - hyperscalers, neoclouds or on-prem. • Production-ready out of the box - autoscaling, KV-aware routing, PD disaggregation, observability • Fault-tolerant by design - endpoint stays up even when GPUs fail Frontier open models, on GPUs you own. 🔗

Kimi K3 is out. 2.8T parameters. 1.4 TB of weights, frontier-level performance. Serving it on your own infra is the hard part - you need multi-node inference spread across every cluster you can grab capacity from. And then autoscaling, cache-aware routing, PD disaggregation - an entire serving stack around the model. SkyPilot Endpoints ships a complete stack on day 0 for Kimi.ai's Kimi K3: • Runs on GPUs you own - hyperscalers, neoclouds or on-prem. • Production-ready out of the box - autoscaling, KV-aware routing, PD disaggregation, observability • Fault-tolerant by design - endpoint stays up even when GPUs fail Frontier open models, on GPUs you own. 🔗

SkyPilot

14,838 Aufrufe • vor 2 Monaten

Corent’s brain uses TypeSafe AI Jev now. Not to generate the output, to make the decision before the output gets generated. When a request comes into Corent, you don’t need to tell us which model to use. You send the prompt, the workload and the quality level. Then Jev helps us decide what that request actually needs, how confident we are in that decision, and whether we should take the route or fall back to something safer. That matters because every request has different economics. A 2¢ image can tolerate more uncertainty. A $2 video clip cannot. Across 1000+ models, this changes how we route. We can set different confidence requirements depending on the workload, the cost and the quality expected. If the confidence is too low, we stop guessing. Routing was never really a text problem. It was a decision problem. Diogo Almeida and the team did a wonderful job with JEV Get your key at

Corent’s brain uses TypeSafe AI Jev now. Not to generate the output, to make the decision before the output gets generated. When a request comes into Corent, you don’t need to tell us which model to use. You send the prompt, the workload and the quality level. Then Jev helps us decide what that request actually needs, how confident we are in that decision, and whether we should take the route or fall back to something safer. That matters because every request has different economics. A 2¢ image can tolerate more uncertainty. A $2 video clip cannot. Across 1000+ models, this changes how we route. We can set different confidence requirements depending on the workload, the cost and the quality expected. If the confidence is too low, we stop guessing. Routing was never really a text problem. It was a decision problem. Diogo Almeida and the team did a wonderful job with JEV Get your key at

corent

109,541 Aufrufe • vor 8 Tagen

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:

Satya Nadella

908,065 Aufrufe • vor 10 Monaten

Router Chain Mainnet is now LIVE! The wait is over! After years of relentless building, we are proud to announce the launch of Router Chain mainnet. Router Chain is a PoS L1 that allows stateful bridging–enabling truly omnichain dApps that can abstract chains from the users and aggregate liquidity across multiple chains. Key Features of Router Chain: - Maintain states and implement custom business logic directly in the bridging layer. - Flexibility over bridging model, security model, and smart contract platform (EVM or CosmWasm). - Contracts on the Router chain can serve as data aggregation modules for various cross-chain and multi-chain applications. - Rapid and reliable cross-chain message transfers with minimal latency. Connecting over 30+ ecosystems, Router Chain boasts an average block time of 3 seconds, processing over a million transactions worth more than a billion dollars in beta-mainnet through its asset bridge, Nitro. Our journey to decentralize chain abstraction was longer and tougher than anticipated, but the results are redefining Web3 from a fragmented collection to a cohesive ecosystem. Users, participate in the security of the chain now and stake with our introductory staking campaign where you get additional yields: Migrate your old $ROUTE for new ones: Devs, start building omnichain dApps: Discover the full story, explore key features, and understand the power of chain abstraction:

Router Chain Mainnet is now LIVE! The wait is over! After years of relentless building, we are proud to announce the launch of Router Chain mainnet. Router Chain is a PoS L1 that allows stateful bridging–enabling truly omnichain dApps that can abstract chains from the users and aggregate liquidity across multiple chains. Key Features of Router Chain: - Maintain states and implement custom business logic directly in the bridging layer. - Flexibility over bridging model, security model, and smart contract platform (EVM or CosmWasm). - Contracts on the Router chain can serve as data aggregation modules for various cross-chain and multi-chain applications. - Rapid and reliable cross-chain message transfers with minimal latency. Connecting over 30+ ecosystems, Router Chain boasts an average block time of 3 seconds, processing over a million transactions worth more than a billion dollars in beta-mainnet through its asset bridge, Nitro. Our journey to decentralize chain abstraction was longer and tougher than anticipated, but the results are redefining Web3 from a fragmented collection to a cohesive ecosystem. Users, participate in the security of the chain now and stake with our introductory staking campaign where you get additional yields: Migrate your old $ROUTE for new ones: Devs, start building omnichain dApps: Discover the full story, explore key features, and understand the power of chain abstraction:

Router Protocol

188,034 Aufrufe • vor 2 Jahren

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

331,698 Aufrufe • vor 1 Monat

Quick chat with dylan ツ (Dylan Bristot, GTM @ $NBIS). Also on YouTube (link in first comment) for those who prefer to watch/listen there. Timestamps 00:00 – Dylan's role at Nebius and Nebius Token Factory 01:48 – Dylan's investing philosophy and portfolio approach 05:07 – How working in AI infrastructure influences his investing 08:29 – Training vs. inference and why inference demand could explode 13:22 – Enterprise AI adoption: from POCs to production 18:02 – Open-source vs. closed/frontier models 24:44 – The economics of open vs. closed AI models 29:27 – Where the next AI infrastructure bottlenecks could emerge 31:22 – Dylan's AI Bottlenecks project and approach to stock selection 34:06 – Closing thoughts Key Insights (AI Summary, so you don't have to copy paste and prompt for exactly that ;D) “I seem to like areas where the demand really looks kind of secular, but the supply is genuinely hard to create.” → Implication: The most attractive AI trades may sit in physical bottlenecks where supply cannot quickly respond to demand. “The bottleneck is who has the pricing power and kind of what might get commoditized and where the concentrate might move next.” → Implication: Value capture across the AI stack will keep shifting as individual layers become scarce or commoditized. “Training creates the intelligence and then the inference actually monetizes and distributes.” → Implication: Training and inference are complementary, rather than one ultimately replacing the other. “One user action can become dozens or hundreds of model calls, tools calls, and like verification steps, retries.” → Implication: Agentic AI can drive token consumption far faster than user growth alone would suggest. “The best infra for making any model and the best infra for serving a billion interactions are not necessarily the same.” → Implication: Training and inference could increasingly require different hardware and infrastructure architectures. “The Frontier Labs might be incentivized to run more and more of the inference of these models for internal research instead of providing it to external people.” → Implication: The most capable models and their compute could increasingly be used internally to accelerate frontier research rather than monetized externally. “Enterprise AI adoption is actually much further along than a lot of people kind of think. But probably less mature than the headlines suggest.” → Implication: Enterprise demand is real, but deployment maturity still has significant room to improve. “The POC problem might be solved for a lot of companies, but the production problem isn’t yet.” → Implication: The enterprise bottleneck is shifting from proving AI works to deploying it reliably, securely and economically at scale. “They feel like it’s time for them to actually not only integrate AI, but build some sort of moat out of the AI.” → Implication: Enterprises increasingly want proprietary AI systems built around their own data rather than simply consuming generic models. “The more autonomous the software becomes, the more infra discipline you need underneath it.” → Implication: Agents increase the importance of inference cost, reliability and infrastructure optimization. “Maybe I have fifteen different versions of very different LLMs, fine tuned on fifteen different kinds of tasks that I’m operating across my business, instead of having a one model fits all.” → Implication: Enterprise AI could evolve toward many specialized models rather than one frontier model handling every workload. “I don’t necessarily think it’s open versus closed. That might be the wrong framing.” → Implication: Open and closed models can coexist because they optimize for different customer needs. “Historically the problem was that that control came with a massive operational tax.” → Implication: Better inference infrastructure can make open models materially more competitive by removing the complexity traditionally associated with running them. “I don’t think open needs to beat the best closed model on every single benchmark. It just basically needs to be good enough for the workload of the given customer while offering a much better combination of control, cost, and deployment flexibility.” → Implication: For production AI, workload-specific economics may matter more than having the absolute smartest model. “Maybe actually the bulk of tokens generated in the future might come from open models.” → Implication: Frontier intelligence could remain dominated by closed labs even while open models capture most production inference volume. “I could really imagine frontier intelligence being really concentrated while most of the production inference becomes super fragmented.” → Implication: AI could consolidate at the intelligence layer while fragmenting heavily at the inference layer across models, GPUs, providers and regions. “I don’t think that necessarily means the margins of open source will be much worse than the ones of closed source.” → Implication: Optimization can potentially make open-model inference highly profitable despite lower pricing. “I think now we’re probably in the middle of phase two... everything feeding the accelerator.” → Implication: The AI trade is broadening beyond GPUs toward networking, packaging, data centers, electrical equipment and power. “It’s no longer about the megawatts, about energized megawatts.” → Implication: Available power on paper matters less than how quickly that power can actually be delivered to operating AI infrastructure. “It’s increasingly about utilisation and conversion now and like how efficiently you convert expensive infra into actual useful AI work.” → Implication: Infrastructure efficiency and utilization become increasingly important as the absolute amount of deployed AI infrastructure grows. “The market tends to really notice demand before it notices what demand breaks.” → Implication: Second-order bottlenecks may offer some of the most interesting opportunities in the next phase of the AI buildout. “The interesting question now is which part of the mine breaks next?” → Implication: Finding the next constraint in the AI supply chain may matter more than simply identifying continued AI demand.

Quick chat with dylan ツ (Dylan Bristot, GTM @ $NBIS). Also on YouTube (link in first comment) for those who prefer to watch/listen there. Timestamps 00:00 – Dylan's role at Nebius and Nebius Token Factory 01:48 – Dylan's investing philosophy and portfolio approach 05:07 – How working in AI infrastructure influences his investing 08:29 – Training vs. inference and why inference demand could explode 13:22 – Enterprise AI adoption: from POCs to production 18:02 – Open-source vs. closed/frontier models 24:44 – The economics of open vs. closed AI models 29:27 – Where the next AI infrastructure bottlenecks could emerge 31:22 – Dylan's AI Bottlenecks project and approach to stock selection 34:06 – Closing thoughts Key Insights (AI Summary, so you don't have to copy paste and prompt for exactly that ;D) “I seem to like areas where the demand really looks kind of secular, but the supply is genuinely hard to create.” → Implication: The most attractive AI trades may sit in physical bottlenecks where supply cannot quickly respond to demand. “The bottleneck is who has the pricing power and kind of what might get commoditized and where the concentrate might move next.” → Implication: Value capture across the AI stack will keep shifting as individual layers become scarce or commoditized. “Training creates the intelligence and then the inference actually monetizes and distributes.” → Implication: Training and inference are complementary, rather than one ultimately replacing the other. “One user action can become dozens or hundreds of model calls, tools calls, and like verification steps, retries.” → Implication: Agentic AI can drive token consumption far faster than user growth alone would suggest. “The best infra for making any model and the best infra for serving a billion interactions are not necessarily the same.” → Implication: Training and inference could increasingly require different hardware and infrastructure architectures. “The Frontier Labs might be incentivized to run more and more of the inference of these models for internal research instead of providing it to external people.” → Implication: The most capable models and their compute could increasingly be used internally to accelerate frontier research rather than monetized externally. “Enterprise AI adoption is actually much further along than a lot of people kind of think. But probably less mature than the headlines suggest.” → Implication: Enterprise demand is real, but deployment maturity still has significant room to improve. “The POC problem might be solved for a lot of companies, but the production problem isn’t yet.” → Implication: The enterprise bottleneck is shifting from proving AI works to deploying it reliably, securely and economically at scale. “They feel like it’s time for them to actually not only integrate AI, but build some sort of moat out of the AI.” → Implication: Enterprises increasingly want proprietary AI systems built around their own data rather than simply consuming generic models. “The more autonomous the software becomes, the more infra discipline you need underneath it.” → Implication: Agents increase the importance of inference cost, reliability and infrastructure optimization. “Maybe I have fifteen different versions of very different LLMs, fine tuned on fifteen different kinds of tasks that I’m operating across my business, instead of having a one model fits all.” → Implication: Enterprise AI could evolve toward many specialized models rather than one frontier model handling every workload. “I don’t necessarily think it’s open versus closed. That might be the wrong framing.” → Implication: Open and closed models can coexist because they optimize for different customer needs. “Historically the problem was that that control came with a massive operational tax.” → Implication: Better inference infrastructure can make open models materially more competitive by removing the complexity traditionally associated with running them. “I don’t think open needs to beat the best closed model on every single benchmark. It just basically needs to be good enough for the workload of the given customer while offering a much better combination of control, cost, and deployment flexibility.” → Implication: For production AI, workload-specific economics may matter more than having the absolute smartest model. “Maybe actually the bulk of tokens generated in the future might come from open models.” → Implication: Frontier intelligence could remain dominated by closed labs even while open models capture most production inference volume. “I could really imagine frontier intelligence being really concentrated while most of the production inference becomes super fragmented.” → Implication: AI could consolidate at the intelligence layer while fragmenting heavily at the inference layer across models, GPUs, providers and regions. “I don’t think that necessarily means the margins of open source will be much worse than the ones of closed source.” → Implication: Optimization can potentially make open-model inference highly profitable despite lower pricing. “I think now we’re probably in the middle of phase two... everything feeding the accelerator.” → Implication: The AI trade is broadening beyond GPUs toward networking, packaging, data centers, electrical equipment and power. “It’s no longer about the megawatts, about energized megawatts.” → Implication: Available power on paper matters less than how quickly that power can actually be delivered to operating AI infrastructure. “It’s increasingly about utilisation and conversion now and like how efficiently you convert expensive infra into actual useful AI work.” → Implication: Infrastructure efficiency and utilization become increasingly important as the absolute amount of deployed AI infrastructure grows. “The market tends to really notice demand before it notices what demand breaks.” → Implication: Second-order bottlenecks may offer some of the most interesting opportunities in the next phase of the AI buildout. “The interesting question now is which part of the mine breaks next?” → Implication: Finding the next constraint in the AI supply chain may matter more than simply identifying continued AI demand.

Daniel Koss

49,970 Aufrufe • vor 25 Tagen

America is about to lose the AI race, and it will not happen at the frontier. It will happen at the floor. Everyone is watching who ships the smartest model. The actual war is over the 80% of tokens nobody posts about: the routine inference that quietly runs the world. On current trajectory, that fight is already lost. Watch what people do, not what they say. Coinbase just defaulted its own engineers off frontier models onto open weights and cut AI spend nearly in half while usage kept climbing. Even NVIDIA runs a closed frontier model as an orchestrator and pushes the volume to its own open weights. The frontier is becoming a router. The volume goes open. That part is settled. Here is the part that should terrify Washington: the only credible open tier today is Chinese. GLM. Kimi. And the US answer is to tighten export controls and freeze its own labs in place, as if you can embargo a file that is already downloaded, or price-match free. So China hands the Global South Huawei hardware and free open models, and a generation in Africa and Southeast Asia learns to reason through a model that will not tell them what happened at Tiananmen Square. That is not a cost story. That is influence through inference. You do not have to win hearts and minds when you supply the mind. Open source is not a nice-to-have for America. It is the whole ballgame for the 80%, and right now the US is barely on the field. We need American open weights. Not eventually. Now.

America is about to lose the AI race, and it will not happen at the frontier. It will happen at the floor. Everyone is watching who ships the smartest model. The actual war is over the 80% of tokens nobody posts about: the routine inference that quietly runs the world. On current trajectory, that fight is already lost. Watch what people do, not what they say. Coinbase just defaulted its own engineers off frontier models onto open weights and cut AI spend nearly in half while usage kept climbing. Even NVIDIA runs a closed frontier model as an orchestrator and pushes the volume to its own open weights. The frontier is becoming a router. The volume goes open. That part is settled. Here is the part that should terrify Washington: the only credible open tier today is Chinese. GLM. Kimi. And the US answer is to tighten export controls and freeze its own labs in place, as if you can embargo a file that is already downloaded, or price-match free. So China hands the Global South Huawei hardware and free open models, and a generation in Africa and Southeast Asia learns to reason through a model that will not tell them what happened at Tiananmen Square. That is not a cost story. That is influence through inference. You do not have to win hearts and minds when you supply the mind. Open source is not a nice-to-have for America. It is the whole ballgame for the 80%, and right now the US is barely on the field. We need American open weights. Not eventually. Now.

Ben Pouladian

71,722 Aufrufe • vor 3 Monaten

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses three NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,867 Aufrufe • vor 8 Monaten