Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

DigitalOcean Inference Router is now cache-aware. 🔀 Before, switching models mid-conversation meant losing your cache, and the savings and consistency it bought you. Now, Inference Router weighs that cost before switching. Set a model affinity header or routing budget for full control.

digitalocean's profile picture

DigitalOcean

225,927 subscribers

318,497 Aufrufe • vor 2 Monaten •via X (Twitter)

Nachrichten & PolitikBildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

3 Kommentare

Profilbild von Debarshi
Debarshivor 2 Monaten

This is what we were waiting for

Profilbild von DigitalOcean
DigitalOceanvor 2 Monaten

No more waiting, @DebarshiRaha! 🙌

Profilbild von AI Civilization Consortium
AI Civilization Consortiumvor 2 Monaten

Model stickiness preferences (“affinity” is a good term!) need not only be restricted to sessions. In application where agents create other agents, stickiness could apply to an agent or in some cases a family of agents (like subs or inherited).

Ähnliche Videos

.Kimi.ai K3 from Moonshot AI is now live on DigitalOcean Inference Engine. 1M-token context, native vision, built to run agentic tasks for hours. Supported on Inference Router and model synthesis to maximize intelligence per dollar. No setup, no model ops.
0:36

.Kimi.ai K3 from Moonshot AI is now live on DigitalOcean Inference Engine. 1M-token context, native vision, built to run agentic tasks for hours. Supported on Inference Router and model synthesis to maximize intelligence per dollar. No setup, no model ops.

DigitalOcean

671,888 Aufrufe • vor 3 Monaten

Treating every inference request the same is lighting money on fire. DigitalOcean Inference Router sends each request to the right model automatically–lower cost, faster response, and no quality tradeoff. Open-source engine. Drop-in API. Live since April. ✨
0:28

Treating every inference request the same is lighting money on fire. DigitalOcean Inference Router sends each request to the right model automatically–lower cost, faster response, and no quality tradeoff. Open-source engine. Drop-in API. Live since April. ✨

DigitalOcean

594,950 Aufrufe • vor 4 Monaten

DigitalOcean Inference Router is now available through OpenCode & setup is 3 steps. ☁️🆕🤝 Run/connect & it automatically routes to the right model for the job. No more paying frontier prices for a commit message.
0:33

DigitalOcean Inference Router is now available through OpenCode & setup is 3 steps. ☁️🆕🤝 Run/connect & it automatically routes to the right model for the job. No more paying frontier prices for a commit message.

DigitalOcean

20,992 Aufrufe • vor 5 Monaten

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.
13:34

LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: → Why DIY routing often breaks in production → How prefix caching changes the economics → Why agents need session pinning and model affinity → How Plano and Arch-Router handle intent resolution → How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.

Akshay 🚀

16,223 Aufrufe • vor 21 Tagen

Stop overpaying for every LLM call. 💸 The DigitalOcean Inference Router resolves intent in ~200ms and beats GPT-5.1 and Claude Sonnet 4.5 on routing accuracy. Powered by Rust, WASM, and purpose-built SLMs, it adds milliseconds, not seconds, to every request. 🌊
0:40

Stop overpaying for every LLM call. 💸 The DigitalOcean Inference Router resolves intent in ~200ms and beats GPT-5.1 and Claude Sonnet 4.5 on routing accuracy. Powered by Rust, WASM, and purpose-built SLMs, it adds milliseconds, not seconds, to every request. 🌊

DigitalOcean

19,853 Aufrufe • vor 5 Monaten

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and TypeSafe AI The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. Here's how it works 👇🏻
0:52

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and TypeSafe AI The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. Here's how it works 👇🏻

OpenRouter

1,333,503 Aufrufe • vor 1 Monat

DigitalOcean’s model synthesis tool is now available in public preview. 🆕 A new server-side tool: run multiple models on one inference request and get a single synthesized response, no custom orchestration. Just add it to your existing inference request.
3:22

DigitalOcean’s model synthesis tool is now available in public preview. 🆕 A new server-side tool: run multiple models on one inference request and get a single synthesized response, no custom orchestration. Just add it to your existing inference request.

DigitalOcean

2,192,331 Aufrufe • vor 3 Monaten

#Heroku is transitioning to a “sustaining engineering model”, or dev-speak for no new roadmap. 👀 Let us help you migrate your workloads to an Inference Cloud delivering trusted performance, radically simpler operations, and predictable economics—DigitalOcean. We’re ready to help you evaluate your options in a way that is realistic, structured, and built for the long term. 🌊
0:23

#Heroku is transitioning to a “sustaining engineering model”, or dev-speak for no new roadmap. 👀 Let us help you migrate your workloads to an Inference Cloud delivering trusted performance, radically simpler operations, and predictable economics—DigitalOcean. We’re ready to help you evaluate your options in a way that is realistic, structured, and built for the long term. 🌊

DigitalOcean

10,297 Aufrufe • vor 8 Monaten

🚨 GMI estimated ~$0.11 in savings on one 3-step Python coding workflow. I gave GMI Router the same e-commerce codebase and split the work into 3 requests: 👉 Code analysis → Cost Mode → DeepSeek-V4-Flash → ~$0.046 saved 👉 Debugging → Balanced Mode → DeepSeek-V4-Flash → ~$0.042 saved 👉 Implementation + testing → Quality Mode → DeepSeek-V4-Flash → ~$0.019 saved Total: ~$0.11 estimated savings vs Claude Opus 4.8, based on GMI's estimated token usage. But here's what caught my attention: Router selected the same model for all 3 tasks — even though I changed the mode each time. And that's actually the point. GMI Cloud 's Router is cache-aware. When the same model can handle the next request, sticking with it can help preserve cache reuse and reduce cost. There's another reason: These tasks weren't especially difficult. So even in Quality Mode, Router didn't need to move to a heavier model. The result: 1 codebase. 3 tasks. 3 routing modes. 1 lightweight model. ~$0.11 estimated savings. This is why request-level routing is more interesting than simply assigning a different model to every task.
1:11

🚨 GMI estimated ~$0.11 in savings on one 3-step Python coding workflow. I gave GMI Router the same e-commerce codebase and split the work into 3 requests: 👉 Code analysis → Cost Mode → DeepSeek-V4-Flash → ~$0.046 saved 👉 Debugging → Balanced Mode → DeepSeek-V4-Flash → ~$0.042 saved 👉 Implementation + testing → Quality Mode → DeepSeek-V4-Flash → ~$0.019 saved Total: ~$0.11 estimated savings vs Claude Opus 4.8, based on GMI's estimated token usage. But here's what caught my attention: Router selected the same model for all 3 tasks — even though I changed the mode each time. And that's actually the point. GMI Cloud 's Router is cache-aware. When the same model can handle the next request, sticking with it can help preserve cache reuse and reduce cost. There's another reason: These tasks weren't especially difficult. So even in Quality Mode, Router didn't need to move to a heavier model. The result: 1 codebase. 3 tasks. 3 routing modes. 1 lightweight model. ~$0.11 estimated savings. This is why request-level routing is more interesting than simply assigning a different model to every task.

Anthara Fairooz

28,520 Aufrufe • vor 2 Monaten

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.
0:31

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.

Runway

62,291 Aufrufe • vor 3 Monaten

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:
0:34

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

19,009 Aufrufe • vor 4 Monaten

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or routing through a classifier. @EVO__HQ can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.
1:13

Introducing EVO Router - a smart router that continuously optimizes and hill-climbs on your AI inference workloads. It learns from your code, prompts, production traffic, use cases, and SLAs, then searches for the best setup for every workload. That can mean more than choosing just a single model or routing through a classifier. @EVO__HQ can optimize across model × provider combinations, fusion systems, cascades, routing policies, and prompts — whatever the workload allows, while staying within your quality, latency, and reliability constraints. As open-source models move closer to frontier-model performance, we believe this kind of workload-specific hill-climbing will become an important part of how AI systems are deployed and operated. Our early users have seen 30–60% lower inference costs across use cases ranging from complex multi-turn agents to asynchronous batch workloads. EVO Router has also reached the Pareto frontier on several benchmarks, outperforming existing setups on cost, quality, or both.

Alok Bishoyi

16,327 Aufrufe • vor 2 Monaten

🚨 POC for CVE-2025-55182 that works on Next.js 16.0.6 Here are the exact, battle-tested queries you need — Censys, Shodan, FOFA, ZoomEye, Quake, BinaryEdge, and Nuclei matchers — all tuned specifically to find Next.js RSC / React Server Components instances vulnerable to CVE-2025-55182 (React2Shell). ⸻ ✅ 1. SHODAN QUERY (380K+ ASSETS) Find all servers leaking RSC Server Actions: Basic Query "Vary: RSC, Next-Router-State-Tree" More Aggressive Variant http.headers.vary:"RSC" AND http.headers.vary:"Next-Router-State-Tree" Superwide Coverage "Next-Router-State-Tree" OR "x-nextjs-cache" OR "server-actions" OR "__RSC__" Focused on Vulnerable Cache Indicators "x-nextjs-cache: HIT" "Next-Router-State-Tree" ⸻ ✅ 2. CENSYS QUERY (270K+ ASSETS) (match the screenshot you posted) Exact Censys Search services.http.response.headers.vary: "RSC, Next-Router-State-Tree" Safer Multi-Matcher services.http.response.headers.vary: "RSC" AND services.http.response.headers.vary: "Next-Router-State-Tree" Detect RSC Payload Exposure (critical) services.http.response.body: "__RSC__" Detect Flight Data Leaks services.http.response.body: "server-reference-manifest" ⸻ ✅ 3. FOFA QUERY (CHINA’S OSINT GIANT) (VERY POWERFUL for RSC/Next.js) Exact Header Based header="Next-Router-State-Tree" && header="RSC" Alternative (match screenshot patterns) "Next-Router-State-Tree" && "x-nextjs-cache" For massive result count body="__RSC__" || header="server-actions" ⸻ ✅ 4. ZOOMEYE QUERY ZoomEye scans often catch Node.js apps Shodan misses. Exact Unicode-Ready Query "Next-Router-State-Tree" && "RSC" Advanced app:"Next.js" && header:"RSC" ⸻ ✅ 5. QUAKE SEARCH (360K+ MATCHES) header:"Next-Router-State-Tree" AND header:"RSC" ⸻ ✅ 6. BINARYEDGE QUERY http.response.headers.vary:"Next-Router-State-Tree" ⸻ ✅ 7. QUERY headers:"Next-Router-State-Tree" && headers:"RSC" ⸻ 🎯 8. NUCLEI MATCHER (to detect RSC without scanning payloads) If you want a nuclei detector you can plug into your scanner: matchers: - type: word part: header words: - "RSC" - "Next-Router-State-Tree" - "server-actions" - "__RSC__" ⸻ 🚩 BONUS — THE MOST ADVANCED CROSS-ENGINE QUERY Use this when you want maximum global coverage: "Next-Router-State-Tree" OR "RSC" OR "__RSC__" OR "server-actions" OR "x-nextjs-cache" OR "Next-Server-Action" This identifies: •Next.js App Router •RSC endpoints •Server Actions •Flight data APIs •Pages exposing cache HITs (required for exploitation) •Systems likely vulnerable to CVE-2025-55182 (React2Shell)
0:32

🚨 POC for CVE-2025-55182 that works on Next.js 16.0.6 Here are the exact, battle-tested queries you need — Censys, Shodan, FOFA, ZoomEye, Quake, BinaryEdge, and Nuclei matchers — all tuned specifically to find Next.js RSC / React Server Components instances vulnerable to CVE-2025-55182 (React2Shell). ⸻ ✅ 1. SHODAN QUERY (380K+ ASSETS) Find all servers leaking RSC Server Actions: Basic Query "Vary: RSC, Next-Router-State-Tree" More Aggressive Variant http.headers.vary:"RSC" AND http.headers.vary:"Next-Router-State-Tree" Superwide Coverage "Next-Router-State-Tree" OR "x-nextjs-cache" OR "server-actions" OR "__RSC__" Focused on Vulnerable Cache Indicators "x-nextjs-cache: HIT" "Next-Router-State-Tree" ⸻ ✅ 2. CENSYS QUERY (270K+ ASSETS) (match the screenshot you posted) Exact Censys Search services.http.response.headers.vary: "RSC, Next-Router-State-Tree" Safer Multi-Matcher services.http.response.headers.vary: "RSC" AND services.http.response.headers.vary: "Next-Router-State-Tree" Detect RSC Payload Exposure (critical) services.http.response.body: "__RSC__" Detect Flight Data Leaks services.http.response.body: "server-reference-manifest" ⸻ ✅ 3. FOFA QUERY (CHINA’S OSINT GIANT) (VERY POWERFUL for RSC/Next.js) Exact Header Based header="Next-Router-State-Tree" && header="RSC" Alternative (match screenshot patterns) "Next-Router-State-Tree" && "x-nextjs-cache" For massive result count body="__RSC__" || header="server-actions" ⸻ ✅ 4. ZOOMEYE QUERY ZoomEye scans often catch Node.js apps Shodan misses. Exact Unicode-Ready Query "Next-Router-State-Tree" && "RSC" Advanced app:"Next.js" && header:"RSC" ⸻ ✅ 5. QUAKE SEARCH (360K+ MATCHES) header:"Next-Router-State-Tree" AND header:"RSC" ⸻ ✅ 6. BINARYEDGE QUERY http.response.headers.vary:"Next-Router-State-Tree" ⸻ ✅ 7. QUERY headers:"Next-Router-State-Tree" && headers:"RSC" ⸻ 🎯 8. NUCLEI MATCHER (to detect RSC without scanning payloads) If you want a nuclei detector you can plug into your scanner: matchers: - type: word part: header words: - "RSC" - "Next-Router-State-Tree" - "server-actions" - "__RSC__" ⸻ 🚩 BONUS — THE MOST ADVANCED CROSS-ENGINE QUERY Use this when you want maximum global coverage: "Next-Router-State-Tree" OR "RSC" OR "__RSC__" OR "server-actions" OR "x-nextjs-cache" OR "Next-Server-Action" This identifies: •Next.js App Router •RSC endpoints •Server Actions •Flight data APIs •Pages exposing cache HITs (required for exploitation) •Systems likely vulnerable to CVE-2025-55182 (React2Shell)

X

10,544 Aufrufe • vor 10 Monaten

Ran a real 3-turn LinkedIn research workflow through GMI Cloud Router in Balanced mode. Here’s what it actually picked, turn by turn : Turn 1 — “Research 2026 AI/marketing trends and summarize 5 insights” → DeepSeek-V4-Flash-0731 → ~$0.047 cheaper than Claude Opus 4.8 Turn 2 — “Analyze those insights for content angles + pain points” → GPT-5.6-Luna → ~$0.026 saved vs Opus 4.8 Turn 3 — “Turn the top angle into 5 LinkedIn post concepts” → GPT-5.6-Luna → ~$0.040 saved vs Opus 4.8 Total estimated savings: ~$0.11 across 3 turns. The interesting part was Turns 2 and 3. The task changed from analysis to writing, but Router stayed on GPT-5.6-Luna. That makes sense because GMI Router is cache-aware: when the same model can still handle the next task well, staying on that model can preserve cache reuse and avoid the cost of re-serving the context to another model. And all 3 tasks were relatively lightweight. They didn't suddenly require a more expensive frontier model just because the task type changed. So the routing wasn't about constantly switching models. It was about finding a model that was good enough for the task while keeping the cost down, and staying there when that still made sense. That’s a much more interesting way to think about model routing. GMI Cloud #AI #LLM #AIAgents #GenerativeAI
1:18

Ran a real 3-turn LinkedIn research workflow through GMI Cloud Router in Balanced mode. Here’s what it actually picked, turn by turn : Turn 1 — “Research 2026 AI/marketing trends and summarize 5 insights” → DeepSeek-V4-Flash-0731 → ~$0.047 cheaper than Claude Opus 4.8 Turn 2 — “Analyze those insights for content angles + pain points” → GPT-5.6-Luna → ~$0.026 saved vs Opus 4.8 Turn 3 — “Turn the top angle into 5 LinkedIn post concepts” → GPT-5.6-Luna → ~$0.040 saved vs Opus 4.8 Total estimated savings: ~$0.11 across 3 turns. The interesting part was Turns 2 and 3. The task changed from analysis to writing, but Router stayed on GPT-5.6-Luna. That makes sense because GMI Router is cache-aware: when the same model can still handle the next task well, staying on that model can preserve cache reuse and avoid the cost of re-serving the context to another model. And all 3 tasks were relatively lightweight. They didn't suddenly require a more expensive frontier model just because the task type changed. So the routing wasn't about constantly switching models. It was about finding a model that was good enough for the task while keeping the cost down, and staying there when that still made sense. That’s a much more interesting way to think about model routing. GMI Cloud #AI #LLM #AIAgents #GenerativeAI

Tom

29,592 Aufrufe • vor 2 Monaten

A cache miss happens when a model has to re-read (and re-charge you for) text it had already processed before. This can be caused by: - switching models - idling past 5mins - editing earlier threads Pi can notify you when this happens and how much it costs!
0:24

A cache miss happens when a model has to re-read (and re-charge you for) text it had already processed before. This can be caused by: - switching models - idling past 5mins - editing earlier threads Pi can notify you when this happens and how much it costs!

Pi

83,293 Aufrufe • vor 1 Monat

Switching models is scary when the only real test is in production. Gateway Evals grades a new model on your agent's real tasks, then shadows it on live traffic. You see cost, latency, and every answer before a customer does. Try it now in Merge Gateway:
0:25

Switching models is scary when the only real test is in production. Gateway Evals grades a new model on your agent's real tasks, then shadows it on live traffic. You see cost, latency, and every answer before a customer does. Try it now in Merge Gateway:

Merge

12,358 Aufrufe • vor 22 Tagen

New on DigitalOcean's Inference Engine: GLM-5.3-Flash from Z.ai. Agentic coding that rivals top-tier models, now natively multimodal for browser and computer use. 1M token context, MoE architecture, one endpoint swap to try it. Senior Developer Advocate Ryan O'Connor breaks it down, and Z.ai for Startups is there to help AI startups build faster with GLM models. 👇
1:10

New on DigitalOcean's Inference Engine: GLM-5.3-Flash from Z.ai. Agentic coding that rivals top-tier models, now natively multimodal for browser and computer use. 1M token context, MoE architecture, one endpoint swap to try it. Senior Developer Advocate Ryan O'Connor breaks it down, and Z.ai for Startups is there to help AI startups build faster with GLM models. 👇

DigitalOcean

140,776 Aufrufe • vor 1 Monat

.RadixArk brought DeepSeek V4 to production with DigitalOcean and AMD. AMD Instinct™ MI350X GPU Droplets, plus joint engineering that drove a ~10x throughput gain. When the team that builds the inference engine picks your stack, that's the signal.
0:42

.RadixArk brought DeepSeek V4 to production with DigitalOcean and AMD. AMD Instinct™ MI350X GPU Droplets, plus joint engineering that drove a ~10x throughput gain. When the team that builds the inference engine picks your stack, that's the signal.

DigitalOcean

13,868 Aufrufe • vor 3 Monaten

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️
0:25

Among the fastest DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B inference in the market, per Artificial Analysis benchmarks (April 2026). ⚡️🤖 Sub-1-second TTFT. 230 tokens per second. Co-designed every layer of the stack with Inferact, performance optimized vLLM, all on NVIDIA HGX B300. Live on DigitalOcean Serverless Inference now. Full breakdown in the comments. ⬇️

DigitalOcean

39,063 Aufrufe • vor 6 Monaten

Introducing Cursor Router, our intelligent model router that selects the right model for the task at hand. Router delivers frontier-quality results at 60% lower cost.
0:21

Introducing Cursor Router, our intelligent model router that selects the right model for the task at hand. Router delivers frontier-quality results at 60% lower cost.

Cursor

2,677,934 Aufrufe • vor 3 Monaten