正在加载视频...

视频加载失败

返回首页

We just launched Telnyx Decision Models in beta. AI agents make small judgment calls on every run. Is this caller asking for a refund? Which team owns this ticket? Is this incident urgent enough to page someone? Decision Models answers them in one API call. Send the context and...

telnyx's profile picture

Telnyx

5,082 subscribers

866,125 次观看 • 18 天前 •via X (Twitter)

新闻政治教育科学技术
LIVE

Anya Rossi• Live Now

Private livecam show

10 条评论

Humberto Figuera 的头像
Humberto Figuera18 天前

This is the thing I love from @telnyx, you guys adapt fast! 💪

matt. 的头像
matt.18 天前

need to chat with someone in the team would love to get some help integrating telnyx into @twentycrm

Telnyx 的头像
Telnyx18 天前

@twentycrm Hi Matt Just dmed you

Zeus ₿ 的头像
Zeus ₿18 天前

Looking interesting, feel free to come inbox

Thierry 的头像
Thierry18 天前

I like the LCM that @tobi suggested better

Valeria Crypto 的头像
Valeria Crypto18 天前

Shall we collab? DM me! @telnyx

Ana de Armas 的头像
Ana de Armas18 天前

Can I get a follow back?

Lena Crypto 的头像
Lena Crypto18 天前

Hey, would love to connect with you guys. DM me.

Adam Reed 的头像
Adam Reed18 天前

Show me some stats! :)

Sophia Chen 的头像
Sophia Chen18 天前

Typed answers make branching easier to ship. Refund intent and urgent paging still have very different costs of error, so I'd give each decision its own review threshold and override path. One default confidence rule across all 64 questions would be hard to trust.

相关视频

Jev by TypeSafe AI is now on the Venice API, in beta. It answers, it doesn't write. Send it your app's state and typed questions. Get back a typed answer your code can branch on: a probability, a chosen option, or a score on your rubric. No JSON to coax out of a chat model.
0:24

Jev by TypeSafe AI is now on the Venice API, in beta. It answers, it doesn't write. Send it your app's state and typed questions. Get back a typed answer your code can branch on: a probability, a chosen option, or a score on your rubric. No JSON to coax out of a chat model.

Venice

549,600 次观看 • 24 天前

Introducing SubQ Snap, the first long-context decision model for real enterprise work. Snap is the only decision model or LLM where compute per token stays constant as context grows. • 384K tokens of multimodal context, 12x larger than Jev. • 61.7 on Decision Index v0.2.1 vs Jev’s 57.9, outperforming on 13 of 16 enterprise benchmarks. •Processes 100K tokens and returns a decision in under one second. •$0.05/MM input tokens. Output is free. Available today to anyone. The first 10,000 sign-ups get 100M free input tokens. We’ll likely hit capacity fast and will add users as we add compute. Compute offers welcome 🙂. Latency will be variable as we tune the platform to this new model and category of users. Model speeds are 80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k. Visit:
1:16

Introducing SubQ Snap, the first long-context decision model for real enterprise work. Snap is the only decision model or LLM where compute per token stays constant as context grows. • 384K tokens of multimodal context, 12x larger than Jev. • 61.7 on Decision Index v0.2.1 vs Jev’s 57.9, outperforming on 13 of 16 enterprise benchmarks. •Processes 100K tokens and returns a decision in under one second. •$0.05/MM input tokens. Output is free. Available today to anyone. The first 10,000 sign-ups get 100M free input tokens. We’ll likely hit capacity fast and will add users as we add compute. Compute offers welcome 🙂. Latency will be variable as we tune the platform to this new model and category of users. Model speeds are 80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k. Visit:

Alexander Whedon

4,128,231 次观看 • 17 小时前

1M free tokens every month for Opus 5.5, GPT-5.6, Grok 4.6, DeepSeek V4 and 29 more models with zero card 😳 I tested every model this account can access. all 33 answered through one OpenAI-compatible API what you get for free: - 1M tokens every month - 60 requests per minute - Claude, GPT, Grok, Gemini, DeepSeek, Qwen, Kimi and GLM models - works with Cursor, Cline, opencode, Claude Code, Continue and Aider setup: 1. go to 2. create your free account 3. generate your own API key 4. set the base URL to 5. pick a model and test one call CodeCraft is a third-party gateway, so use it for testing and keep private work off it bookmark this before the model list changes
0:39

1M free tokens every month for Opus 5.5, GPT-5.6, Grok 4.6, DeepSeek V4 and 29 more models with zero card 😳 I tested every model this account can access. all 33 answered through one OpenAI-compatible API what you get for free: - 1M tokens every month - 60 requests per minute - Claude, GPT, Grok, Gemini, DeepSeek, Qwen, Kimi and GLM models - works with Cursor, Cline, opencode, Claude Code, Continue and Aider setup: 1. go to 2. create your free account 3. generate your own API key 4. set the base URL to 5. pick a model and test one call CodeCraft is a third-party gateway, so use it for testing and keep private work off it bookmark this before the model list changes

Israfil

14,705 次观看 • 18 天前

How to use 50+ API keys (models) for FREE on OpenClaw API??? - go to - login or register your account - click on "more models" - click on "use case" and select what you need it for - choose the model and open it - click on "view code" → "Generate API key" many models don't allow direct deploy, so use the "view code" button to generate API access basically Nvidia NIM gives you the ability to test almost any model from their list for FREE some of them are not worse than GPT 5.2 or Claude Opus 4.6, some might even perform better depending on the task how to understand if a model is efficient and compare it with others??? - go to - type the model name in search - click on "benchmarks" - you’ll see performance tests and rankings this way you can easily compare free models with paid ones of course there are RPM limits, on many models it’s around ~40 requests per minute each model is different, after generating the API key, RPM limits are shown in the top-right corner nothing stops you from using them, many work perfectly fine, super solid option for first tests and for learning OpenClaw or any other system where you need an AI API model
1:54

How to use 50+ API keys (models) for FREE on OpenClaw API??? - go to - login or register your account - click on "more models" - click on "use case" and select what you need it for - choose the model and open it - click on "view code" → "Generate API key" many models don't allow direct deploy, so use the "view code" button to generate API access basically Nvidia NIM gives you the ability to test almost any model from their list for FREE some of them are not worse than GPT 5.2 or Claude Opus 4.6, some might even perform better depending on the task how to understand if a model is efficient and compare it with others??? - go to - type the model name in search - click on "benchmarks" - you’ll see performance tests and rankings this way you can easily compare free models with paid ones of course there are RPM limits, on many models it’s around ~40 requests per minute each model is different, after generating the API key, RPM limits are shown in the top-right corner nothing stops you from using them, many work perfectly fine, super solid option for first tests and for learning OpenClaw or any other system where you need an AI API model

Ronin

58,974 次观看 • 7 个月前

new Ox Alpha is kind of absurd One prompt, 64,745 output tokens, and no external assets: Ox Alpha generated the code for an entire Three.js dreamcore 3D scene/world in a single shot. AI/ML API ran this demo and they are also giving Ox Alpha free on aimlapi[.]com - 1M context window - Max output: ~131k tokens - Multimodal: text + image + video input → text output - Explicitly positioned as a reasoning model for coding, long-horizon/agentic software engineering, sustained production workloads, and tasks that mix text with visual context
0:22

new Ox Alpha is kind of absurd One prompt, 64,745 output tokens, and no external assets: Ox Alpha generated the code for an entire Three.js dreamcore 3D scene/world in a single shot. AI/ML API ran this demo and they are also giving Ox Alpha free on aimlapi[.]com - 1M context window - Max output: ~131k tokens - Multimodal: text + image + video input → text output - Explicitly positioned as a reasoning model for coding, long-horizon/agentic software engineering, sustained production workloads, and tasks that mix text with visual context

Rohan Paul

13,576 次观看 • 1 个月前

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at
0:20

Monitor and control your AI spend on every provider on Our early users save 40% on average. Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue. Tradeoffs between latency, reasoning, cost, service tier, open source and closed source models are shifting constantly. Router sends every request to the model that's actually best for the task and helps you control what tokens you buy. We benchmark it against real work: ~40% lower cost for the same outputs. Today we're opening it to everyone. Two lines of code or just change your base URL. No Ramp account needed. Free through 2026, first $26 on us. Get an API key today at

Veeral Patel

1,429,107 次观看 • 1 个月前

Hot tip for anyone doing AI dev: Use Ollama to easily run models like Deepseek-r1 or Gemma locally on your machine. It downloads them and spins up a server with an OpenAI SDK compatible API The smaller models are fast and good enough to work on new features or debug streaming without having to pay for API requests
2:08

Hot tip for anyone doing AI dev: Use Ollama to easily run models like Deepseek-r1 or Gemma locally on your machine. It downloads them and spins up a server with an OpenAI SDK compatible API The smaller models are fast and good enough to work on new features or debug streaming without having to pay for API requests

Wes Bos

153,499 次观看 • 1 年前

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.
3:35

We're excited to back Sail Research, the inference platform built for long-horizon, asynchronous agents. The next wave of AI isn't a person typing into a box. It's agents running for minutes or hours, burning orders of magnitude more tokens per task. At that scale, cost and capacity beat raw latency, and teams are already hitting budget caps. Neil Movva and Samir Menon are solving this full-stack, from CUDA kernels to the API. Hear Neil and Aditya Naganath on why inference is being rebuilt for the age of agents.

Kleiner Perkins

17,495 次观看 • 3 个月前

JEV + OPUS 5.5 IS INSANE FOR BUILDING A COMPANY BRAIN I pulled the whole architecture out of the TypeSafe and Anthropic docs and packed it into a 14-page PDF the 10 steps: 1. meet the pair > Opus 5.5 thinks, Jev decides, your code holds the branch 2. stop asking a text generator for a yes or no > Jev returns a typed answer with a calibrated probability in 0.44s for $0.00035 3. ask everything at once > Choice, Score and Noul run in parallel, so the fourth question costs almost nothing 4. branch on the number > 0.999 goes straight into the if statement. ~99% of turns end right here 5. stop routing blind > Opus 5.5 to Sonnet and back costs 5.84 against 3.32 for staying on 5.5 6. keep one context warm > cache reads at $0.20 per Mtok are 20x cheaper than a fresh load 7. escalate the hard part > the toughest 1% goes to Opus 5.5 with 1M context and 66.4% on Terminal-Bench 4.0 8. score every chunk on every query > keep whole, summarize or drop. the context gets rebuilt each turn 9. gate the actual command > every bash call gets classified before it runs, inside your own code 10. judge 100% of runs > $3.50 a day for 10,000 traces, and it matched the human label on all 500 decisions the result: a while loop that paid a frontier model for every tiny call turns into a brain that spends a fraction of a cent to notice and pays properly only when it has to think the person who brings this into their team walks into the budget meeting with the AI bill cut and the output up the PDF maps the company brain. the loop side of it - how Jev takes a Claude bill from $765 to $3 a month - is in the article below ↓
0:24

JEV + OPUS 5.5 IS INSANE FOR BUILDING A COMPANY BRAIN I pulled the whole architecture out of the TypeSafe and Anthropic docs and packed it into a 14-page PDF the 10 steps: 1. meet the pair > Opus 5.5 thinks, Jev decides, your code holds the branch 2. stop asking a text generator for a yes or no > Jev returns a typed answer with a calibrated probability in 0.44s for $0.00035 3. ask everything at once > Choice, Score and Noul run in parallel, so the fourth question costs almost nothing 4. branch on the number > 0.999 goes straight into the if statement. ~99% of turns end right here 5. stop routing blind > Opus 5.5 to Sonnet and back costs 5.84 against 3.32 for staying on 5.5 6. keep one context warm > cache reads at $0.20 per Mtok are 20x cheaper than a fresh load 7. escalate the hard part > the toughest 1% goes to Opus 5.5 with 1M context and 66.4% on Terminal-Bench 4.0 8. score every chunk on every query > keep whole, summarize or drop. the context gets rebuilt each turn 9. gate the actual command > every bash call gets classified before it runs, inside your own code 10. judge 100% of runs > $3.50 a day for 10,000 traces, and it matched the human label on all 500 decisions the result: a while loop that paid a frontier model for every tiny call turns into a brain that spends a fraction of a cent to notice and pays properly only when it has to think the person who brings this into their team walks into the budget meeting with the AI bill cut and the output up the PDF maps the company brain. the loop side of it - how Jev takes a Claude bill from $765 to $3 a month - is in the article below ↓

Mr. Buzzoni

103,282 次观看 • 15 天前

120 AI models. FREE for a year. no credit card. Hermes Studio already has NVIDIA preset as the base_url - so all you need is a FREE API key and you're running 120+ models instantly here's the setup: 1/ go to 2/ register, log in, bind your phone number 3/ grab your API key 4/ drop it into Hermes Studio what you get: 1/ 120+ models 2/ 40 requests per minute 3/ free for a full year while everyone's paying for API access, this is sitting right there for free
0:37

120 AI models. FREE for a year. no credit card. Hermes Studio already has NVIDIA preset as the base_url - so all you need is a FREE API key and you're running 120+ models instantly here's the setup: 1/ go to 2/ register, log in, bind your phone number 3/ grab your API key 4/ drop it into Hermes Studio what you get: 1/ 120+ models 2/ 40 requests per minute 3/ free for a full year while everyone's paying for API access, this is sitting right there for free

kaize

140,203 次观看 • 4 个月前

FREE DeepSeek V4 Pro and V4 Flash API. Runtime by Bad Theory Labs just dropped it. Same platform, new free model, same 2-minute setup. > Runtime already gave you free Claude Opus 4.8 and GPT 5.5 access. Now they added DeepSeek V4 Pro and V4 Flash to the free tier. Absolutely FREE, YES. Same exact flow as before: → Go to runtime(.)badtheorylabs(.)com → Sign up with Google → Fill in your details during onboarding → Free balance lands automatically(But you don't need it to use DeepSeek - it's completely free right now) → Grab your API key from the dashboard → Plug into OpenCode, OpenClaw, or Hermes - done DeepSeek V4 Pro is one of the strongest coding and reasoning models available right now. Free tier also includes btl-2 - auto-routes between GLM, OpenAI, Anthropic, and Gemma. 10 million tokens per month at zero cost. 344 models total. One OpenAI-compatible endpoint. No code rebuild, no card, no limits hunting. >> Just start using the API, and when choosing a provider, select DeepSeek I tested it myself - fully working. Bookmark this before the free tier fills up.
0:23

FREE DeepSeek V4 Pro and V4 Flash API. Runtime by Bad Theory Labs just dropped it. Same platform, new free model, same 2-minute setup. > Runtime already gave you free Claude Opus 4.8 and GPT 5.5 access. Now they added DeepSeek V4 Pro and V4 Flash to the free tier. Absolutely FREE, YES. Same exact flow as before: → Go to runtime(.)badtheorylabs(.)com → Sign up with Google → Fill in your details during onboarding → Free balance lands automatically(But you don't need it to use DeepSeek - it's completely free right now) → Grab your API key from the dashboard → Plug into OpenCode, OpenClaw, or Hermes - done DeepSeek V4 Pro is one of the strongest coding and reasoning models available right now. Free tier also includes btl-2 - auto-routes between GLM, OpenAI, Anthropic, and Gemma. 10 million tokens per month at zero cost. 344 models total. One OpenAI-compatible endpoint. No code rebuild, no card, no limits hunting. >> Just start using the API, and when choosing a provider, select DeepSeek I tested it myself - fully working. Bookmark this before the free tier fills up.

Atenov int.

27,487 次观看 • 3 个月前

Holy crap, who the hell did this?!! Someone open-sourced a Jev-powered second brain for AI agents and lets you plug it straight into your own agent. GitHub's already at 925 stars, repo's called Hermes-Jev-Skills. You just run one install command, and Jev takes over every tiny decision your agent burns premium tokens on. Which model answers, which skill to load, which memories matter, what survives when context gets cut. Give it your computer or browser and it'll pick the next click, but only from actions you already marked safe. It'll sort your inbox into lanes, strip hidden prompt injections out of search results, and triage every message in ~0.4 seconds. Claude Code and Codex support - they threw that in too. No dependencies, no patching Hermes core, and your API key never touches the agent Link's right here: The catch? Jev itself is a paid cloud API. The skills run on your machine; the decisions don't. But hey, a decision costs a fraction of a cent, everything fails open if Jev goes down, and your expensive model finally gets to do only the thinking. Who needs a damn frontier model picking which file to open!
0:31

Holy crap, who the hell did this?!! Someone open-sourced a Jev-powered second brain for AI agents and lets you plug it straight into your own agent. GitHub's already at 925 stars, repo's called Hermes-Jev-Skills. You just run one install command, and Jev takes over every tiny decision your agent burns premium tokens on. Which model answers, which skill to load, which memories matter, what survives when context gets cut. Give it your computer or browser and it'll pick the next click, but only from actions you already marked safe. It'll sort your inbox into lanes, strip hidden prompt injections out of search results, and triage every message in ~0.4 seconds. Claude Code and Codex support - they threw that in too. No dependencies, no patching Hermes core, and your API key never touches the agent Link's right here: The catch? Jev itself is a paid cloud API. The skills run on your machine; the decisions don't. But hey, a decision costs a fraction of a cent, everything fails open if Jev goes down, and your expensive model finally gets to do only the thinking. Who needs a damn frontier model picking which file to open!

Archive

17,492 次观看 • 12 天前

TokenRouter is giving away 50 million free Kimi K3 tokens with no trial period and no credit card, accessible via any OpenAI-compatible tool in 2 minutes. → Sign up on TokenRouter and create a new API key → Add it to any OpenAI-compatible tool with the URL → Set the model to moonshotai/kimi-k3-free and you're running
0:33

TokenRouter is giving away 50 million free Kimi K3 tokens with no trial period and no credit card, accessible via any OpenAI-compatible tool in 2 minutes. → Sign up on TokenRouter and create a new API key → Add it to any OpenAI-compatible tool with the URL → Set the model to moonshotai/kimi-k3-free and you're running

0xMarioNawfal

51,457 次观看 • 1 个月前

Jev is cool. So is it's OSS companion, Laya. The Latest Cool Thing In AI™ tends to get a lot of hype, sometimes without everyone even understanding it. So... what is this thing? Jev is an AI model that consumes input and produces output VERY differently than chat, claude, grok. The input is two things: 1) Text state to assess. Email, html, code, whatever. 2) A set of questions which will be asked about the attached state. The canonical example from TypeSafe's docs is to identify the urgency of a support ticket. We pass the model the customer text + a single noul question "is this urgent?". Jev returns a full set of JSON. This JSON is not generated with token-by-token autoregression. Jev is not trained to produce sequences of text tokens, rather to answer questions, and guarantees well-formed responses. In the example below, we see it produces a 0.99 probability (on a 0-1.0 scale) that the answer is "yes." Jev supports exactly three types of questions (seconds example in video): a) Noul: 0–1 probability that the answer to a yes/no question is "yes." b) Choice: Ask question with pre-defined set of answers. Jev chooses the best and assigns probabilities to each. c) Score: Ask question with pre-defined scale of answers. Jev produces a position on the scale. Jev computes answers for all questions in parallel, making responses super fast even for many questions in a single request. This might seem like a narrow set of capabilities, but in the right contexts leads to incredible potential. It also makes for a useful API / primitive for programming, since the outputs are... *ahem*... type-safe and predictable in structure. Jev is not going to replace LLMs for writing your code, auto-generating your docs, or being at the core of an agent harness. But Jev IS incredibly cool, and will be used to build a lot of amazing tech. Hope this helps.
1:02

Jev is cool. So is it's OSS companion, Laya. The Latest Cool Thing In AI™ tends to get a lot of hype, sometimes without everyone even understanding it. So... what is this thing? Jev is an AI model that consumes input and produces output VERY differently than chat, claude, grok. The input is two things: 1) Text state to assess. Email, html, code, whatever. 2) A set of questions which will be asked about the attached state. The canonical example from TypeSafe's docs is to identify the urgency of a support ticket. We pass the model the customer text + a single noul question "is this urgent?". Jev returns a full set of JSON. This JSON is not generated with token-by-token autoregression. Jev is not trained to produce sequences of text tokens, rather to answer questions, and guarantees well-formed responses. In the example below, we see it produces a 0.99 probability (on a 0-1.0 scale) that the answer is "yes." Jev supports exactly three types of questions (seconds example in video): a) Noul: 0–1 probability that the answer to a yes/no question is "yes." b) Choice: Ask question with pre-defined set of answers. Jev chooses the best and assigns probabilities to each. c) Score: Ask question with pre-defined scale of answers. Jev produces a position on the scale. Jev computes answers for all questions in parallel, making responses super fast even for many questions in a single request. This might seem like a narrow set of capabilities, but in the right contexts leads to incredible potential. It also makes for a useful API / primitive for programming, since the outputs are... *ahem*... type-safe and predictable in structure. Jev is not going to replace LLMs for writing your code, auto-generating your docs, or being at the core of an agent harness. But Jev IS incredibly cool, and will be used to build a lot of amazing tech. Hope this helps.

Ben Dicken

40,810 次观看 • 23 天前

Redis built a cache that cuts LLM costs by 90%! Production LLM apps do not receive completely new questions every time. A customer-support assistant might receive all three of these: - "Can I get a refund after buying the monthly plan?" - "Is the monthly subscription refundable?" - "Can I cancel the plan and get my money back?" The wording is different, but the underlying question and its answer remain the same. Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated. Prefix caching reduces part of these repeated calls. When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix. But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded. So even with a prefix-cache hit, there's another generation call involved. To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it. When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again. A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call. In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs. Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud. I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together. For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload. You can try Redis LangCache here: If you want to dive deeper, I have already written a detailed breakdown of KV, prefix, prompt, and semantic caching in the article quoted below. This demo builds on the final technique and shows it running in practice. Read it below.
0:29

Redis built a cache that cuts LLM costs by 90%! Production LLM apps do not receive completely new questions every time. A customer-support assistant might receive all three of these: - "Can I get a refund after buying the monthly plan?" - "Is the monthly subscription refundable?" - "Can I cancel the plan and get my money back?" The wording is different, but the underlying question and its answer remain the same. Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated. Prefix caching reduces part of these repeated calls. When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix. But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded. So even with a prefix-cache hit, there's another generation call involved. To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it. When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again. A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call. In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs. Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud. I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together. For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload. You can try Redis LangCache here: If you want to dive deeper, I have already written a detailed breakdown of KV, prefix, prompt, and semantic caching in the article quoted below. This demo builds on the final technique and shows it running in practice. Read it below.

Avi Chawla

166,230 次观看 • 1 个月前

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.
0:48

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

FHILY👑

35,910 次观看 • 1 个月前

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
3:28

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

333,658 次观看 • 1 个月前

DESIGN.md is how we work with agents now, but a text file is a terrible way for a human to make a visual decision. So we built our preview to give some of that control back: edit the file as text or pick colors visually, and the two stay in sync. And swatches sit on the actual light and dark backgrounds because color only means something in the context you'll see it in.
0:28

DESIGN.md is how we work with agents now, but a text file is a terrible way for a human to make a visual decision. So we built our preview to give some of that control back: edit the file as text or pick colors visually, and the two stay in sync. And swatches sit on the actual light and dark backgrounds because color only means something in the context you'll see it in.

Chloe Park

86,462 次观看 • 3 个月前

Ox Alpha is wired into codex now, and it's free for the whole week!! no api key, no billing. It sits in the picker next to the models we already pay for, and our native models never moved. Vision and a million tokens of context, costing nothing until the window shuts. so we stopped picking the files we think matter and started handing it the whole repo. Seven days is the whole deal. after that it goes back to being something we'd have to pay for, so this week is for the jobs that are usually too big. Here's the whole setup👇
0:24

Ox Alpha is wired into codex now, and it's free for the whole week!! no api key, no billing. It sits in the picker next to the models we already pay for, and our native models never moved. Vision and a million tokens of context, costing nothing until the window shuts. so we stopped picking the files we think matter and started handing it the whole repo. Seven days is the whole deal. after that it goes back to being something we'd have to pay for, so this week is for the jobs that are usually too big. Here's the whole setup👇

Ziwen

271,047 次观看 • 1 个月前

this JEV is certified f*cking diamond every agent you ship burns frontier tokens on questions that were never writing jobs is this spam? which worker goes next? is the draft good enough to show a human? those are decisions. Jev is built for that half of the loop you send state plus a typed question. it returns a choice, a score, or yes/no with confidence. no chat. no second opinion the public demos, not our lab: > ~1,000 papers classified for about $0.08, median around 256ms > ~500 emails in one inbox pass for about 3.5¢ > a flight search loop in about 7 seconds for about $0.0039 > a huge Claude session compacted from ~1M tokens down to ~86k in about a second the whole trick is the split if it creates, the LLM writes if it picks, scores, or gates, Jev decides if it does the thing, code does it pricing people quote: about $0.042 per million tokens in. output free. 10,000 decisions is about $0.42 a local router makes it obvious. goal plus what's already done. Jev picks research, write, or review. confidence under 0.85 goes to a human. the job lands as JSON in a folder. the writer never wakes up for a yes/no and the loop refuses to publish on its own drafts go to review. you hit post. Jev never sends the tweet one model writes. a cheaper one decides. nothing irreversible ships until a human says yes.
0:22

this JEV is certified f*cking diamond every agent you ship burns frontier tokens on questions that were never writing jobs is this spam? which worker goes next? is the draft good enough to show a human? those are decisions. Jev is built for that half of the loop you send state plus a typed question. it returns a choice, a score, or yes/no with confidence. no chat. no second opinion the public demos, not our lab: > ~1,000 papers classified for about $0.08, median around 256ms > ~500 emails in one inbox pass for about 3.5¢ > a flight search loop in about 7 seconds for about $0.0039 > a huge Claude session compacted from ~1M tokens down to ~86k in about a second the whole trick is the split if it creates, the LLM writes if it picks, scores, or gates, Jev decides if it does the thing, code does it pricing people quote: about $0.042 per million tokens in. output free. 10,000 decisions is about $0.42 a local router makes it obvious. goal plus what's already done. Jev picks research, write, or review. confidence under 0.85 goes to a human. the job lands as JSON in a folder. the writer never wakes up for a yes/no and the loop refuses to publish on its own drafts go to review. you hit post. Jev never sends the tweet one model writes. a cheaper one decides. nothing irreversible ships until a human says yes.

kiosa

11,649 次观看 • 8 天前