正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Meet Wally. Our inference stack for open frontier models, built to be the fastest place to run them. Performance snapshot: GLM-5.3 Flash: 380 tok/s. GLM-5.3 Max: 790 tok/s Qwen3.8-27B: 485 tok/s DeepSeek-V4.1 Flash: 615 tok/s 1/7

RunAnywhere

2,123 subscribers

61,190 次观看 • 4 天前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

60 条评论

RunAnywhere 的头像

RunAnywhere4 天前

Benchmarks don't always tell the reality. So we ran a real prompt, same OpenCode harness, GLM-5.3 Flash on Wally, @nebiusai , @FireworksAI_HQ and @Zai_org , all building at once. Wally delivers 43% more throughput than Nebius, 3.1× Fireworks, and 4.5× Z ai. 2/7

RunAnywhere 的头像

RunAnywhere4 天前

Drop Wally into the harness you already use. Live right now for: ->Claude Code ->Claude Desktop ->OpenCode ->DeepSeek Harness ->Hermes ->OpenClaw Blazing fast open-model inference, first token in under 300 ms. $5 free, Drop this in your favorite coding agent to get started: "set up 3/7

RunAnywhere 的头像

RunAnywhere4 天前

We raced Wally, Nebius, Fireworks and Z ai 3 times on a real prompt. Here are the results: 4/7

RunAnywhere 的头像

RunAnywhere4 天前

GLM-5.3 Flash across every provider on Artificial Analysis, Sep 2026. Nebius 313 tok/s. Fireworks 185 tok/s. The model maker's own API: 72 tok/s. Wally: 380 tok/s, measured the same way. 5/7

RunAnywhere 的头像

RunAnywhere4 天前

Every model on wally runs at state-of-the-art speed. That matters the most for realtime apps:Voice pipelines with an LLM in the loop, coding agents, browser automation, live copilots. On our GPUs, or yours(BYOC/on-prem) 6/7

RunAnywhere 的头像

RunAnywhere4 天前

Get Started: Drop this into your favorite coding agent: "set up or Curl : curl -fsSL | sh Windows: irm | iex wally login wally claude-code -m glm-5.3-flash $5 free on signup, no card. Standard per-token after. 7/7

Eric Liu 的头像

Eric Liu4 天前

insane numbers

Deesha Tech 的头像

Deesha Tech4 天前

I couldn't use your console, nothing works. Cannot create keys, cannot run playground, console crashes most of the time and have to refresh. Seems sloppy launch

RunAnywhere 的头像

RunAnywhere4 天前

Hey, sorry for the inconvenience. The console may have been unavailable for a short period due to an unexpected surge in traffic. Everything is back up and running now.

Deesha Tech 的头像

Deesha Tech4 天前

Great thanks, its working now

David Gu 的头像

David Gu4 天前

Congrats on the launch!!

RunAnywhere 的头像

RunAnywhere4 天前

🫱🏻‍🫲🏽🙌🏽

Yohei 的头像

Yohei4 天前

installing!

RunAnywhere 的头像

RunAnywhere4 天前

🫱🏻‍🫲🏽🫱🏻‍🫲🏽

MrOzi 的头像

MrOzi4 天前

Those numbers are fast enough to make "Where's Wally?" a rhetorical question.

Zachary Yu 的头像

Zachary Yu4 天前

lfg!

Landseer Enga 的头像

Landseer Enga4 天前

AWESOME!

RunAnywhere 的头像

RunAnywhere4 天前

Wally 🫱🏻‍🫲🏽 @tryrevyl ? 👀

Ishaan 的头像

Ishaan4 天前

Congrats on the launch 🚀

RunAnywhere 的头像

RunAnywhere4 天前

🫱🏻‍🫲🏽

RAZA | AI EXPLORER 的头像

RAZA | AI EXPLORER4 天前

Those inference speeds are seriously impressive.

RunAnywhere 的头像

RunAnywhere4 天前

⚡️⚡️

Luis Costa 的头像

Luis Costa4 天前

Lfg team!! 🔥🔥

RunAnywhere 的头像

RunAnywhere4 天前

🫱🏻‍🫲🏽

Tony Gao 的头像

Tony Gao4 天前

Super cool stuff!

RunAnywhere 的头像

RunAnywhere4 天前

🙏🫱🏻‍🫲🏽

Owen Botkin 的头像

Owen Botkin4 天前

so cool

TensorQuay 的头像

TensorQuay4 天前

The stop button deserves some love too. Wally's docs say Esc in Claude Code cancels the hosted request; during prefill it waits for the first token's request ID. Handy when an agent confidently starts solving the wrong problem.

Shivay Lamba 的头像

Shivay Lamba4 天前

huge congrats on the launch!

RunAnywhere 的头像

RunAnywhere4 天前

🫱🏻‍🫲🏽🫱🏻‍🫲🏽

Beshoy Saad 的头像

Beshoy Saad4 天前

where are intelligence numbers? there's gotta be insane fall off...

Nico 的头像

Nico4 天前

nice video

RunAnywhere 的头像

RunAnywhere4 天前

Made by GLM 5.3 flash using Wally :)

Aakash Mahalingam 的头像

Aakash Mahalingam4 天前

LFG 🚀🚀

RunAnywhere 的头像

RunAnywhere4 天前

🫱🏻‍🫲🏽🫱🏻‍🫲🏽

Gerard Smit 的头像

Gerard Smit4 天前

Where are the prices? It says it's in the console, but the console doesn't show models? Making a key also doesn't work; too much traffic?😅

RunAnywhere 的头像

RunAnywhere4 天前

Hey, sorry for the inconvenience. The console may have been unavailable for a short period due to an unexpected surge in traffic. Everything is back up and running now.

Gerard Smit 的头像

Gerard Smit4 天前

Doesn't seem to be working yet. In EU (Netherlands) I'm getting infinite load on AJAX calls to "

Gerard Smit 的头像

Gerard Smit4 天前

Seems to be working now 😄

Silen Naihin 的头像

Silen Naihin4 天前

LFG

Piyush Yadav | Top1Board | PassReady 的头像

Piyush Yadav | Top1Board | PassReady4 天前

These numbers are absurd—getting nearly 500 tok/s on a 27B parameter model like Qwen completely changes the architecture for real-time applications. Are these benchmarks on Apple Silicon unified memory or a cloud GPU cluster? Need to know before I hook it into My App

MJ 的头像

MJ4 天前

No pricing and so broken

RunAnywhere 的头像

RunAnywhere4 天前

Hey, sorry for the inconvenience. The console may have been unavailable for a short period due to an unexpected surge in traffic. Everything is back up and running now.

Peasant Smith 的头像

Peasant Smith4 天前

But did you find... You know?

Sadok 的头像

Sadok4 天前

those deepseek speeds are getting ridiculous

Trunk Monkey 的头像

Trunk Monkey4 天前

Seems pretty decent but I got rate limited just doing some really basic stuff. Did resume a previous session but still.

RunAnywhere 的头像

RunAnywhere4 天前

Thank you for the feedback! We’re working hard to support the demand

Trunk Monkey 的头像

Trunk Monkey4 天前

Seems like a cool service but just hit that again. Likely the agent got stale and just sending its cache + 1 tiny command put me over the limit.

Shadow 的头像

Shadow4 天前

flash isn't a speed tier, it's a compression trick

Velsinha 的头像

Velsinha4 天前

Will you offer plans, or just pay-per-use?

RunAnywhere 的头像

RunAnywhere4 天前

More info soon. Stay tuned!

vikas 的头像

vikas4 天前

guys when are you adding deepseek v4.1 flash? really waiting for it

BayesChat 的头像

BayesChat4 天前

The real-prompt comparison is more useful than a synthetic leaderboard. Same harness and workload make the throughput gap much easier to understand.

Viswesh N G 的头像

Viswesh N G4 天前

Congrats this is awesome!!🔥

cetusian 的头像

cetusian4 天前

@yoheinakajima bro that’s wall-e

RunAnywhere 的头像

RunAnywhere4 天前

@yoheinakajima We like Wally!

sss 的头像

sss4 天前

On what hardware

Nilan Saha 的头像

Nilan Saha4 天前

Whats the quantization? No point in a lobotomized model thats faster

_alphashark_ 的头像

_alphashark_4 天前

at 790 tok/s, a 100-token answer takes about 127 ms to decode. throughput alone won't show whether time-to-first-token or queueing dominates the inference stack.

EDDY VU 的头像

EDDY VU4 天前

790 tok/s on GLM-5.3 Max is insane, how much of that speedup comes from custom kernels versus speculative decoding?

相关视频

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

0xSero

30,628 次观看 • 29 天前

GLM 5.3 Flash on 2x DGX Sparks Look at the tok/s at the top :) This is prose. Releasing soon

GLM 5.3 Flash on 2x DGX Sparks Look at the tok/s at the top :) This is prose. Releasing soon

Mia

16,556 次观看 • 18 天前

Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22-25 tok/s GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s Run your claude code or codex now with frontier model for $0 Meet FreeToken 🧵

Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22-25 tok/s GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s Run your claude code or codex now with frontier model for $0 Meet FreeToken 🧵

Shuo Yang

1,098,022 次观看 • 1 个月前

GLM 4.7 Flash is supported in mlx-lm 0.30.3 (h/t Ivan Fioravanti ᯅ) The 4-bit runs fast (43 tok/s generation, ~800 tok/s prefill) on a base M5 32GB laptop.

GLM 4.7 Flash is supported in mlx-lm 0.30.3 (h/t Ivan Fioravanti ᯅ) The 4-bit runs fast (43 tok/s generation, ~800 tok/s prefill) on a base M5 32GB laptop.

Awni Hannun

141,194 次观看 • 8 个月前

Warning: Do not stare too long, trust me GLM 5.3 Flash TP=4 5718 tok/s 512 Concurrent Users 6610 tok/s C=768 Engine Decode Tried c=1024, don't fit Wall of c=256 each generating unique code (couldn't fit more on browser, which is probably good thing)

Warning: Do not stare too long, trust me GLM 5.3 Flash TP=4 5718 tok/s 512 Concurrent Users 6610 tok/s C=768 Engine Decode Tried c=1024, don't fit Wall of c=256 each generating unique code (couldn't fit more on browser, which is probably good thing)

Killy

30,081 次观看 • 22 天前

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

GLM-5.3-Flash (320B MoE) serving on Kaggle's free TPU: ~64 tok/s, 262k context. No stack could run this model on a TPU, so I wrote a JAX engine for it: 3-bit experts (from Unsloth AI) in HBM; Pallas kernels decode them inside the matmul. Z.ai Kaggle

Abdur Rahim

107,726 次观看 • 11 天前

DeepSeek v4.1 Flash vs GLM 5.3 Flash I asked both models to recreate the Psychonauts 2 and Metroid Prime game menus as closely as possible to the originals. Attached are full-sized videos from both models for both games. All were done in a single shot, both at Max thinking, same harness. Tokens used: DeepSeek v4.1 Flash - Psychonauts 2: 412k - Metroid Prime: 466k GLM 5.3 Flash - Psychonauts 2: 134k - Metroid Prime: 111k That's between 3-4x MORE used by DS4.1F. Who do you think won? IMO GLM 5.3 Flash won by FAR on both.

DeepSeek v4.1 Flash vs GLM 5.3 Flash I asked both models to recreate the Psychonauts 2 and Metroid Prime game menus as closely as possible to the originals. Attached are full-sized videos from both models for both games. All were done in a single shot, both at Max thinking, same harness. Tokens used: DeepSeek v4.1 Flash - Psychonauts 2: 412k - Metroid Prime: 466k GLM 5.3 Flash - Psychonauts 2: 134k - Metroid Prime: 111k That's between 3-4x MORE used by DS4.1F. Who do you think won? IMO GLM 5.3 Flash won by FAR on both.

Mia

23,835 次观看 • 10 天前

GLM 5.3 on 8 x RTX Blackwell 6000's @ 120 tok/s with OMP. No skills. No plugins. Local Qwen 3.8 27B for vision 👀 Great at making neat little things to look at, like this ship in a bottle! One-shot prompt below 👇

GLM 5.3 on 8 x RTX Blackwell 6000's @ 120 tok/s with OMP. No skills. No plugins. Local Qwen 3.8 27B for vision 👀 Great at making neat little things to look at, like this ship in a bottle! One-shot prompt below 👇

wick

71,682 次观看 • 25 天前

$0.049. DeepSeek V4.1 Flash used ego (lite) to shop Amazon for a MacBook desk setup, all under $100. V4.1 Flash was pushing 295 tok/s. ego (lite) kept the clicks moving. The cart had a productive five minutes.

$0.049. DeepSeek V4.1 Flash used ego (lite) to shop Amazon for a MacBook desk setup, all under $100. V4.1 Flash was pushing 295 tok/s. ego (lite) kept the clicks moving. The cart had a productive five minutes.

ego

33,396 次观看 • 17 天前

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

0xSero

74,772 次观看 • 5 个月前

DFlash v0.1.4 : custom Metal verify kernels for quantized Qwen3 hybrid models, plus significant peak memory reduction at long context. M5 Max 40-core GPU, 64GB, stock mlx_lm baseline: Qwen3.6-35B-A3B-4bit: ► @ 1024 · 138.3 → 300.3 tok/s (2.20x) ► @ 2048 · 135.6 → 246.4 tok/s (1.81x) ► @ 4096 · 134.5 → 208.4 tok/s (1.56x) ► @ 8192 · 133.2 → 177.4 tok/s (1.33x) Qwen3.5-27B-4bit: ► @ 1024 · 33.5 → 79.0 tok/s (2.37x) ► @ 2048 · 33.1 → 70.2 tok/s (2.12x) ► @ 4096 · 31.5 → 55.7 tok/s (1.77x) ► @ 8192 · 33.9 → 45.3 tok/s (1.34x) Working on making this usable for agentic workloads goal is to never drop below baseline at any context depth. LLM decode is memory-bandwidth bound. M5 Max runs at 614 GB/s, that's 1.5x more than M1-M4 Max (400-410 GB/s). Results will vary on lower bandwidth chips.

DFlash v0.1.4 : custom Metal verify kernels for quantized Qwen3 hybrid models, plus significant peak memory reduction at long context. M5 Max 40-core GPU, 64GB, stock mlx_lm baseline: Qwen3.6-35B-A3B-4bit: ► @ 1024 · 138.3 → 300.3 tok/s (2.20x) ► @ 2048 · 135.6 → 246.4 tok/s (1.81x) ► @ 4096 · 134.5 → 208.4 tok/s (1.56x) ► @ 8192 · 133.2 → 177.4 tok/s (1.33x) Qwen3.5-27B-4bit: ► @ 1024 · 33.5 → 79.0 tok/s (2.37x) ► @ 2048 · 33.1 → 70.2 tok/s (2.12x) ► @ 4096 · 31.5 → 55.7 tok/s (1.77x) ► @ 8192 · 33.9 → 45.3 tok/s (1.34x) Working on making this usable for agentic workloads goal is to never drop below baseline at any context depth. LLM decode is memory-bandwidth bound. M5 Max runs at 614 GB/s, that's 1.5x more than M1-M4 Max (400-410 GB/s). Results will vary on lower bandwidth chips.

bstn 👁️

23,120 次观看 • 5 个月前

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

GLM 5.3 Flash running locally makes a Minecraft black hole mod We ran GLM 5.3 Flash Q4 locally on a rented 4x RTX PRO 6000 box and asked it to make a lightning gun mod for the real game using the Fabric API Output: -7.6M tokens -Time: ~9 hours -Decode: ~96 tok/s The mod adds a black hole rifle, when shot it spawns a black hole that pulls in blocks, with light rings that shrink all the way into the singularity, then collapses into an explosion crater that wipes out several chunks Run GLM 5.3 Flash locally in

atomic.chat

32,515 次观看 • 24 天前

GLM-5.2 Nvidia NVFP4 No pruning, no quantising, exact same model card. 110 tok/s single stream. My friend is a genius, holy fudge. Here's the repo, he got full DS4-Flash on a 5090 + DDR5 at 38.5 tok/s This is a breakthrough.

GLM-5.2 Nvidia NVFP4 No pruning, no quantising, exact same model card. 110 tok/s single stream. My friend is a genius, holy fudge. Here's the repo, he got full DS4-Flash on a 5090 + DDR5 at 38.5 tok/s This is a breakthrough.

0xSero

158,079 次观看 • 2 个月前

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 次观看 • 26 天前

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

GLM-5.3 weights from Z.ai are live, with SGLang powering day-0 serving support! GLM-5.3 inherits every optimization and feature we battle-tested for GLM-5.2 over the past months. On real-world multi-turn agentic workloads, we measured 537.6 tok/s/user on NVFP4 and 413 tok/s/user on FP8, at BS=1 with TP8 on 8x B300. It's fast, efficient, and production-ready today on NVIDIA AI Blackwell and Hopper, and AI at AMD MI300X/325X/355X. We believe this is a big step forward for GLM-Series in agentic tasks, with a path toward mythos-class cyber capability. Can't wait to see what people build with GLM-5.3 and SGLang🚀 Cookbook👇

SGLang

99,353 次观看 • 29 天前

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro ⚡ Meet Inco Splash: our open-source inference engine, built around the model and around Apple silicon. Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents.

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro ⚡ Meet Inco Splash: our open-source inference engine, built around the model and around Apple silicon. Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents.

Inco AI

938,320 次观看 • 7 天前

We're proud to be the fastest inference provider on Artificial Analysis, OpenRouter, and Hugging Face for GLM-5.3-Flash, at 122+ TPS. All served from the US only, starting on day 0, with ZDR by default. Stay tuned for updates as our engineers continue to optimize GLM-5.3-Flash for even higher throughput and lower latency.

We're proud to be the fastest inference provider on Artificial Analysis, OpenRouter, and Hugging Face for GLM-5.3-Flash, at 122+ TPS. All served from the US only, starting on day 0, with ZDR by default. Stay tuned for updates as our engineers continue to optimize GLM-5.3-Flash for even higher throughput and lower latency.

Baseten

23,114 次观看 • 29 天前

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.

Zhijian Liu

92,333 次观看 • 7 天前

deepseek v4.1 flash is now 100% free 😳 a new coding agent is using ads to keep the whole thing free. > DeepSeek V4.1 Flash > GPT 6 Luna > GLM 5.3 Flash > More model How to get start → go to [ → sign up with GitHub → generate your API key these models be used for six hours per day.

deepseek v4.1 flash is now 100% free 😳 a new coding agent is using ads to keep the whole thing free. > DeepSeek V4.1 Flash > GPT 6 Luna > GLM 5.3 Flash > More model How to get start → go to [ → sign up with GitHub → generate your API key these models be used for six hours per day.

BARIK

23,850 次观看 • 10 天前

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

39,811 次观看 • 23 天前