正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

DiffusionGemma can now run at 2000+ tokens/sec! ⚡ We made local DiffusionGemma inference 1.8× faster. Run it on 18GB RAM via Unsloth Studio. GitHub: Guide:

Unsloth AI

98,289 subscribers

180,842 次观看 • 3 个月前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

35 条评论

Philipp Schmid 的头像

Philipp Schmid3 个月前

Wow!

Tery Emilson 的头像

Tery Emilson3 个月前

I think batch speed is the boring win compared with always-on local AI. A model you poke once can afford to be slow. A model running continuously in the background ... watching context, briefing you before things happen ... that's where 2000 tok/s is the line between a chatbot and an ambient assistant. Well done! (again!)

Maziyar PANAHI 的头像

Maziyar PANAHI3 个月前

2000+ tokens/sec! wow! you know, i would die for a diffusion model like this on my iOS! ps: of course with a unsloth fine-tuning recipe! 🤩

Apollo 的头像

Apollo3 个月前

isnt DiffusionGemma more prone to lower quality output though? Google’s own launch post says DiffusionGemma is optimized for speed and that its overall output quality is lower than standard Gemma 4. Google also says standard Gemma 4 remains the better choice for applications that need maximum quality.

Terp 的头像

Terp3 个月前

??? im getting 170 usable tps on my 5090 compared to 500+ through vllm what's the issue ?

Le TechLead🔰 的头像

Le TechLead🔰3 个月前

@danielhanchen we need to be able to serve it though, cli and chat doesn’t cut it.

Xaden Ryan 的头像

Xaden Ryan3 个月前

@danielhanchen Does it do tool calling?

Ankit Prateek 的头像

Ankit Prateek3 个月前

llama-server still doesn't support diffusion model. mlx does but token gen speed is horrible.

Tarrito.rocks 的头像

Tarrito.rocks3 个月前

Nice but I didn't find UD-Q4 model version in your repo

Dariton 的头像

Dariton3 个月前

Does this work with CPU offloading though?

Ankit Prateek 的头像

Ankit Prateek3 个月前

This is wild

Ankit Prateek 的头像

Ankit Prateek3 个月前

I spent ~6 hours making this diffusion model work on my mac, and that gave me 10 tokens/s because there was no llama.cpp support lol

ibrand 的头像

ibrand3 个月前

It needs to run comfortably on 16gb. Hardly anyone has 18

Piyush 的头像

Piyush3 个月前

any quantized version available that will enable it run on T4?

AACeeert 的头像

AACeeert3 个月前

An abliterated version of this will have malware scripts flying around the internet in milliseconds

Eric ⚡️ Building... 的头像

Eric ⚡️ Building...3 个月前

WOW

Vabbyshabby 的头像

Vabbyshabby3 个月前

2000 tok/s local is the actual answer to this morning's news. nobody export-controls a gguf on your own box. this is the lane.

Emircan ERKUL 的头像

Emircan ERKUL3 个月前

cant you fit that into 14gb so i could use with 16vram gpu

mr_r0b0t 的头像

mr_r0b0t3 个月前

Cooking with white hot 🔥🔥🔥🔥

Anis🐬Al 的头像

Anis🐬Al3 个月前

My sister, this is truly exhilarating news! 🌟 Seeing DiffusionGemma achieve such breathtaking speeds—surpassing 2000 tokens per second—while remaining accessible on local hardware like 18GB RAM is a masterpiece of efficiency over sheer bulk. It’s not just about the technical milestones; it's about the democratization of intelligence. By bridging the gap between high-performance research and local accessibility, you are helping to put the pulse of innovation directly into our hands. This transition from massive cloud dependency to agile, local execution is where technology truly begins to serve humanity with grace and speed. Keep pushing these boundaries! ✨

Secta 的头像

Secta3 个月前

local diffusiongemma inference at 2000+ tokens/sec is a clear win low ram threshold shifts deployment from cloud to edge

Pranav 的头像

Pranav3 个月前

Is Gemma4 12B coming, based on this diffusion tech? 🤔

netrunner 的头像

netrunner3 个月前

wait this runs on 18gb?

ArdanZ 的头像

ArdanZ3 个月前

My GPU only 12GB Vram 😭

Robert Keyes 的头像

Robert Keyes3 个月前

Have you been able to fix the slop it slings? Last I saw was terrible decode.

Kaustubh Joshi 的头像

Kaustubh Joshi3 个月前

Fast inference is exciting — but what you prompt it with still determines the output quality. ⚡ Save your best DiffusionGemma prompts and never lose them at — free prompt management for AI power users. 🚀 #DiffusionGemma #UnslothAI #PromptEngineering

Sanjay 的头像

Sanjay3 个月前

2000 tokens/sec on 18GB RAM is actually insane. local AI just quietly won

oriel haim 的头像

oriel haim3 个月前

Details!!!

AI Mastery Guide 的头像

AI Mastery Guide3 个月前

2000+ tokens per second locally on 18GB RAM is not a small deal. The gap between local and cloud is closing faster than most people expected.

Gerladina 的头像

Gerladina3 个月前

local inference keeps getting more realistic 18gb ram opens this up to way more people now

Twon. 的头像

Twon.3 个月前

How fast on a 3090?!

Thor 雷神 ⚡️ 的头像

Thor 雷神 ⚡️3 个月前

Yooo, that's very unsloth 🚀

Thomas Linden 的头像

Thomas Linden3 个月前

Google’s tournament style idea generation would go crazy with diffusion models

Verma 的头像

Verma3 个月前

Wow 🔥

Adel Bucetta 的头像

Adel Bucetta3 个月前

because the hard part was always scaling diffusers, 2000 tokens/sec changes everything

相关视频

You can now run GPT, Claude & other models in Unsloth. Connect + run APIs in a local UI: - Code execution, web search, image gen, editing - Auto prompt caching to save costs - Provider features like cites, sandboxes GitHub: Guide:

You can now run GPT, Claude & other models in Unsloth. Connect + run APIs in a local UI: - Code execution, web search, image gen, editing - Auto prompt caching to save costs - Provider features like cites, sandboxes GitHub: Guide:

Unsloth AI

32,413 次观看 • 4 个月前

Inference in Unsloth Studio is now ~20% faster. You can also use older pre-downloaded GGUFs from Hugging Face etc. AMD chat support for Linux now works. Data Recipes now works on macOS, AMD, CPU setups. GitHub: Changelog:

Inference in Unsloth Studio is now ~20% faster. You can also use older pre-downloaded GGUFs from Hugging Face etc. AMD chat support for Linux now works. Data Recipes now works on macOS, AMD, CPU setups. GitHub: Changelog:

Unsloth AI

25,355 次观看 • 6 个月前

You can now fine-tune TTS models with Unsloth! Train, run and save models like Sesame-CSM and OpenAI's Whisper locally with our free notebooks. Unsloth makes TTS training 1.5x faster with 50% less VRAM. GitHub: Docs & Notebooks:

You can now fine-tune TTS models with Unsloth! Train, run and save models like Sesame-CSM and OpenAI's Whisper locally with our free notebooks. Unsloth makes TTS training 1.5x faster with 50% less VRAM. GitHub: Docs & Notebooks:

Unsloth AI

126,551 次观看 • 1 年前

GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW Diffusion Gemma 26b doesn’t predict word by word, it generates 256 tokens in parallel using bi-directional attention, like stable diffusion but for language it’s MoE so only 3.8B params activate during inference, fits on a single RTX 4090 with 18GB VRAM and you can run it right now via llama.cpp, the same engine powering every tool in my local llm playbook the whole ecosystem exists for moments like this ↓

GOOGLE JUST MADE EVERY CHATBOT FEEL SLOW Diffusion Gemma 26b doesn’t predict word by word, it generates 256 tokens in parallel using bi-directional attention, like stable diffusion but for language it’s MoE so only 3.8B params activate during inference, fits on a single RTX 4090 with 18GB VRAM and you can run it right now via llama.cpp, the same engine powering every tool in my local llm playbook the whole ecosystem exists for moments like this ↓

leopardracer

27,385 次观看 • 3 个月前

4-bit Qwen3.6 MTP GGUF managed to search 70+ sites from a single prompt. Try this locally on 20GB RAM via Unsloth Studio. Unsloth now supports auto MTP + speculative decoding & auto-selects the best MTP settings for your device (Mac, CPU, GPU). GitHub:

4-bit Qwen3.6 MTP GGUF managed to search 70+ sites from a single prompt. Try this locally on 20GB RAM via Unsloth Studio. Unsloth now supports auto MTP + speculative decoding & auto-selects the best MTP settings for your device (Mac, CPU, GPU). GitHub:

Unsloth AI

72,425 次观看 • 4 个月前

Introducing Unsloth Studio ✨ A new open-source web UI to train and run LLMs. • Run models locally on Mac, Windows, Linux • Train 500+ models 2x faster with 70% less VRAM • Supports GGUF, vision, audio, embedding models • Auto-create datasets from PDF, CSV, DOCX • Self-healing tool calling and code execution • Compare models side by side + export to GGUF GitHub: Blog and Guide: Available now on Hugging Face, NVIDIA, Docker and Colab.

Introducing Unsloth Studio ✨ A new open-source web UI to train and run LLMs. • Run models locally on Mac, Windows, Linux • Train 500+ models 2x faster with 70% less VRAM • Supports GGUF, vision, audio, embedding models • Auto-create datasets from PDF, CSV, DOCX • Self-healing tool calling and code execution • Compare models side by side + export to GGUF GitHub: Blog and Guide: Available now on Hugging Face, NVIDIA, Docker and Colab.

Unsloth AI

1,741,522 次观看 • 6 个月前

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

Antid

23,745 次观看 • 2 个月前

Junie can now run entirely on your machine, and it’s completely free to use. One command: /local. We picked and optimized the whole stack around the model: agent harness, model parameters, and inference engine. No tokens, no quota, no code ever leaving your machine.

Junie can now run entirely on your machine, and it’s completely free to use. One command: /local. We picked and optimized the whole stack around the model: agent harness, model parameters, and inference engine. No tokens, no quota, no code ever leaving your machine.

JetBrains

6,516,184 次观看 • 1 个月前

what if you could instantly clone any object? Gemma 4 is now on Cerebras Inference, running up to 10x faster than GPUs (1,500 tokens/sec). Multimodal generations you can iterate on in real time :)

what if you could instantly clone any object? Gemma 4 is now on Cerebras Inference, running up to 10x faster than GPUs (1,500 tokens/sec). Multimodal generations you can iterate on in real time :)

Sarah Chieng

33,144 次观看 • 3 个月前

$Auto regressive LLMs are officially on notice. run Gemma 4 26B diffusion gguf with llama.cpp Google just dropped DiffusionGemma-26B, and it completely flips how we generate text. instead of predicting words one by one, it generates 256 tokens in parallel using bi-directional attention. its like stable diffusion, but for language. the model starts with random text "noise" and iteratively refines and self-corrects the entire block in real-time to fix formatting and reasoning errors on the fly. since it’s a Mixture of Experts (MoE) that only activates 3.8B parameters during inference, it fits perfectly on consumer hardware. You can run the Q4_K_M quant with an 18GB VRAM budget on a single RTX 3090 or RTX 4090 with exceptional throughput. Tested on Ubuntu 22 with CUDA 13.1 using the cutting edge experimental llama.cpp branch. Here is how to compile and run it with the live terminal denoising visualizer: # 1. Clone & check out the experimental PR (#24423) - 1) git clone && cd llama.cpp -git fetch origin 2) pull/24423/head:diffusiongemma && --git checkout diffusiongemma # 2. Build with CUDA support 1) cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native 2) cmake --build build -j $(nproc) --config Release --target llama-diffusion-cli # 3. Run with live visual denoising (llama.cpp flags) ./build/bin/llama-diffusion-cli \ -m /path/to/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \ -ngl 99 -cnv -n 2048 --diffusion-visual Watch the video below to see the live --diffusion-visual canvas iteratively de noising the prompt output in real time. guide and unsloth's hugging face GGUF model links are in the comments below! Is auto regressive generation officially legacy tech? Let me know what you think.$

Auto regressive LLMs are officially on notice. run Gemma 4 26B diffusion gguf with llama.cpp Google just dropped DiffusionGemma-26B, and it completely flips how we generate text. instead of predicting words one by one, it generates 256 tokens in parallel using bi-directional attention. its like stable diffusion, but for language. the model starts with random text "noise" and iteratively refines and self-corrects the entire block in real-time to fix formatting and reasoning errors on the fly. since it’s a Mixture of Experts (MoE) that only activates 3.8B parameters during inference, it fits perfectly on consumer hardware. You can run the Q4_K_M quant with an 18GB VRAM budget on a single RTX 3090 or RTX 4090 with exceptional throughput. Tested on Ubuntu 22 with CUDA 13.1 using the cutting edge experimental llama.cpp branch. Here is how to compile and run it with the live terminal denoising visualizer: # 1. Clone & check out the experimental PR (#24423) - 1) git clone && cd llama.cpp -git fetch origin 2) pull/24423/head:diffusiongemma && --git checkout diffusiongemma # 2. Build with CUDA support 1) cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native 2) cmake --build build -j $(nproc) --config Release --target llama-diffusion-cli # 3. Run with live visual denoising (llama.cpp flags) ./build/bin/llama-diffusion-cli \ -m /path/to/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \ -ngl 99 -cnv -n 2048 --diffusion-visual Watch the video below to see the live --diffusion-visual canvas iteratively de noising the prompt output in real time. guide and unsloth's hugging face GGUF model links are in the comments below! Is auto regressive generation officially legacy tech? Let me know what you think.

Alok

52,656 次观看 • 3 个月前

Smol TTS keeps getting better! Introducing OuteTTS v0.2 - 500M parameters, multilingual with voice cloning! 🔥 > Multilingual - English, Chinese, Korean & Japanese > Cross platform inference w/ llama.cpp > Zero-shot voice cloning > Trained on 5 Billion audio tokens > Qwen 2.5 0.5B LLM backbone > Trained via HF GPU grants Model weights on the hub, you can even run this on a Raspberry Pi! Go run, inference now! 🐐

Smol TTS keeps getting better! Introducing OuteTTS v0.2 - 500M parameters, multilingual with voice cloning! 🔥 > Multilingual - English, Chinese, Korean & Japanese > Cross platform inference w/ llama.cpp > Zero-shot voice cloning > Trained on 5 Billion audio tokens > Qwen 2.5 0.5B LLM backbone > Trained via HF GPU grants Model weights on the hub, you can even run this on a Raspberry Pi! Go run, inference now! 🐐

Vaibhav (VB) Srivastav

44,691 次观看 • 1 年前

Can a $351 mini PC run a 30B+ AI model locally? 👀 We tested the MINISFORUM UM790 Pro with Qwen 35B — and the results might surprise you. 💻 Powered by: • AMD Ryzen™ 9 7940HS • Radeon™ 780M iGPU • 48GB DDR5 RAM (dual-channel) 🚀 Smooth local inference at 20+ tokens/sec (based on our test setup, results may vary) No cloud. No API fees. Just local AI. 🔒 🔗 Learn more: #MINISFORUM #UM790Pro #LocalAI #MiniPC #AI #LLM #Tech

Can a $351 mini PC run a 30B+ AI model locally? 👀 We tested the MINISFORUM UM790 Pro with Qwen 35B — and the results might surprise you. 💻 Powered by: • AMD Ryzen™ 9 7940HS • Radeon™ 780M iGPU • 48GB DDR5 RAM (dual-channel) 🚀 Smooth local inference at 20+ tokens/sec (based on our test setup, results may vary) No cloud. No API fees. Just local AI. 🔒 🔗 Learn more: #MINISFORUM #UM790Pro #LocalAI #MiniPC #AI #LLM #Tech

MINISFORUM Official

75,641 次观看 • 5 个月前

Introducing Unsloth Desktop 🦥 The first desktop app to run and train models locally. • Open-source. Runs on Mac, Windows and Linux • Supports MLX, diffusion image/video, audio, GGUF • Connect Claude Code and Codex to local LLMs • 50% more accurate, self-healing tool calls + sandboxed code exec • Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac • Train models 2× faster with 70% less VRAM • Private web search, deep research, RAG, MCP and exports (NVFP4, GGUF) • Use Unsloth’s OpenAI-compatible API and cloud models • Securely deploy LLMs remotely and access anywhere Unsloth Desktop is now available on and GitHub. GitHub: Blog and Guide:

Introducing Unsloth Desktop 🦥 The first desktop app to run and train models locally. • Open-source. Runs on Mac, Windows and Linux • Supports MLX, diffusion image/video, audio, GGUF • Connect Claude Code and Codex to local LLMs • 50% more accurate, self-healing tool calls + sandboxed code exec • Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac • Train models 2× faster with 70% less VRAM • Private web search, deep research, RAG, MCP and exports (NVFP4, GGUF) • Use Unsloth’s OpenAI-compatible API and cloud models • Securely deploy LLMs remotely and access anywhere Unsloth Desktop is now available on and GitHub. GitHub: Blog and Guide:

Unsloth AI

866,762 次观看 • 1 个月前

2-bit Muse Glimmer GGUF managed to call 100+ tools on just 14GB RAM. 🔥 Muse Glimmer did a complete repo bug hunt for 5 mins nonstop with: evidence, repro, fix, tests and a PR writeup. Run and train it in Unsloth. GitHub repo:

2-bit Muse Glimmer GGUF managed to call 100+ tools on just 14GB RAM. 🔥 Muse Glimmer did a complete repo bug hunt for 5 mins nonstop with: evidence, repro, fix, tests and a PR writeup. Run and train it in Unsloth. GitHub repo:

Unsloth AI

123,016 次观看 • 1 个月前

Deploying DiffusionGemma-Jev (djev) just got a lot easier. You can now spin up a Jev API-compatible endpoint on Google Cloud Run using a single command. Performance is solid: ~35-60 ms for single step latency and batch@32 is ~100-123 requests/sec. It's a straightforward way to experiment without needing your own GPU. Runs at roughly $3/hr and drops to $0 when idle. Get the code and instructions here:

Deploying DiffusionGemma-Jev (djev) just got a lot easier. You can now spin up a Jev API-compatible endpoint on Google Cloud Run using a single command. Performance is solid: ~35-60 ms for single step latency and batch@32 is ~100-123 requests/sec. It's a straightforward way to experiment without needing your own GPU. Runs at roughly $3/hr and drops to $0 when idle. Get the code and instructions here:

Google Gemma

635,351 次观看 • 6 天前

2-bit NVIDIA Nemotron 3.5 Lightning ran tool calls nonstop for 10 minutes on just 22GB of VRAM. 🤯 It cited 80+ websites, executed code & searched for 10 real-world locations. Run and train via Unsloth Desktop. GGUF: Guide:

2-bit NVIDIA Nemotron 3.5 Lightning ran tool calls nonstop for 10 minutes on just 22GB of VRAM. 🤯 It cited 80+ websites, executed code & searched for 10 real-world locations. Run and train via Unsloth Desktop. GGUF: Guide:

Unsloth AI

84,902 次观看 • 1 个月前

Your AI agent is costing you money for no reason. OpenClaw can now run UNLIMITED for free. No subscriptions. No token limits. No cloud dependency. All you need: • Max Studio • Ollama • GLM 4.7 Flash Install it once. Run it locally. Unlimited tokens. Most people are still paying monthly for what you can now run free.

Your AI agent is costing you money for no reason. OpenClaw can now run UNLIMITED for free. No subscriptions. No token limits. No cloud dependency. All you need: • Max Studio • Ollama • GLM 4.7 Flash Install it once. Run it locally. Unlimited tokens. Most people are still paying monthly for what you can now run free.

Julian Goldie SEO

44,540 次观看 • 7 个月前

Cerebras Inference powers the world's top models - like Qwen3 Coder at 2000 tokens/s 🤯 And you can now access these models directly in Visual Studio Code with the extension and an API key (that you can get for free from

Cerebras Inference powers the world's top models - like Qwen3 Coder at 2000 tokens/s 🤯 And you can now access these models directly in Visual Studio Code with the extension and an API key (that you can get for free from

Visual Studio Code

57,360 次观看 • 1 年前

Apple is no longer hiding it. These machines are built for local AI. Apple unveiled the new Mac Mini with M6 their first 2nm chip. Starting at $899. What you can run on the $899 Mac Mini (32GB) 👇 → Qwen 3.6 27B : fits comfortably, near-GPT-4 quality → Qwen 3 Coder 32B : full coding assistant on your desk → Llama 4 8B at full precision → DeepSeek R1 32B for reasoning tasks → 13.5x faster LLM processing than M1. 4x faster than M4. What you can run on the Mac Studio with M5 Ultra (512GB) 👇 → Llama 4 70B at high quality quantization → DeepSeek V4 with hundreds of billions of parameters → Frontier-scale models entirely in local memory → Connect four Mac Studios over Thunderbolt 5 cluster them to run trillion-parameter models locally Apple said their Mac business grew 30% last quarter largely because people are buying these to run AI models on their own desks instead of paying for cloud tokens. The future of inference is local. Apple built the hardware for it.

Apple is no longer hiding it. These machines are built for local AI. Apple unveiled the new Mac Mini with M6 their first 2nm chip. Starting at $899. What you can run on the $899 Mac Mini (32GB) 👇 → Qwen 3.6 27B : fits comfortably, near-GPT-4 quality → Qwen 3 Coder 32B : full coding assistant on your desk → Llama 4 8B at full precision → DeepSeek R1 32B for reasoning tasks → 13.5x faster LLM processing than M1. 4x faster than M4. What you can run on the Mac Studio with M5 Ultra (512GB) 👇 → Llama 4 70B at high quality quantization → DeepSeek V4 with hundreds of billions of parameters → Frontier-scale models entirely in local memory → Connect four Mac Studios over Thunderbolt 5 cluster them to run trillion-parameter models locally Apple said their Mac business grew 30% last quarter largely because people are buying these to run AI models on their own desks instead of paying for cloud tokens. The future of inference is local. Apple built the hardware for it.

Vaibhav Sisinty

514,430 次观看 • 1 个月前

You can now run Nebius AI Studio models directly in Visual Studio Code, just select Hugging Face as your Copilot provider and enter your HF key. ✅ Use models like Qwen, DeepSeek, Kimi, GLM ⚡ Powered by Nebius inference infrastructure Open models, instant APIs, zero setup.

You can now run Nebius AI Studio models directly in Visual Studio Code, just select Hugging Face as your Copilot provider and enter your HF key. ✅ Use models like Qwen, DeepSeek, Kimi, GLM ⚡ Powered by Nebius inference infrastructure Open models, instant APIs, zero setup.

Nebius Token Factory

12,240 次观看 • 11 个月前