Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Jan v0.5.15: You can now tweak llama.cpp settings, and add any cloud model! Highlights 🎉 - Improved llama.cpp settings - Install & use any cloud model - Better hardware controls - New models: DeepSeek, Gemini, o3-mini, R1 distills Update your Jan or download the latest.

👋 Jan

13,041 subscribers

11,062 views • 1 year ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Hugging Face 🤝 Jan You can now use Hugging Face as a remote model provider in Jan. Go to Settings -> Model Providers -> add your Hugging Face API key. Then open a new chat and pick a model from Hugging Face. Works with any model in Hugging Face in Jan.

Hugging Face 🤝 Jan You can now use Hugging Face as a remote model provider in Jan. Go to Settings -> Model Providers -> add your Hugging Face API key. Then open a new chat and pick a model from Hugging Face. Works with any model in Hugging Face in Jan.

👋 Jan

22,264 views • 1 year ago

Jan v0.6.9 is here: Chat with images now. Highlights: - Multimodal models can see images - MCP Server is now stable feature - Tool calling for gpt-oss (upstream llama.cpp upgrade) - Auto-detect tools & vision - no more manual setup Update your Jan or download the latest.

Jan v0.6.9 is here: Chat with images now. Highlights: - Multimodal models can see images - MCP Server is now stable feature - Tool calling for gpt-oss (upstream llama.cpp upgrade) - Auto-detect tools & vision - no more manual setup Update your Jan or download the latest.

👋 Jan

22,604 views • 11 months ago

🚀 Excited to announce DeepSeek R1 is now available for confidential use in Maple AI! This powerful new model enhances your AI experience with advanced capabilities. And no, your data is not shared with the DeepSeek creators or any government 🔐

🚀 Excited to announce DeepSeek R1 is now available for confidential use in Maple AI! This powerful new model enhances your AI experience with advanced capabilities. And no, your data is not shared with the DeepSeek creators or any government 🔐

Maple AI

17,958 views • 1 year ago

Use Your Private Models on Comfy Cloud! Continue client work safely — or generate with your experimental private models. You can now import models directly from private Hugging Face or Civitai repos. Just add your private key and model URL, and we will bring the models to your workflow! Everything you import stays 100% private to your account.

Use Your Private Models on Comfy Cloud! Continue client work safely — or generate with your experimental private models. You can now import models directly from private Hugging Face or Civitai repos. Just add your private key and model URL, and we will bring the models to your workflow! Everything you import stays 100% private to your account.

ComfyUI

17,705 views • 6 months ago

Jan Desktop v0.7.7 is live 💛 This update brings native MLX support on macOS, a broader UX and UI refresh across the app, and better support for developer workflows. You can now upload files in Projects, use the local API server with both local and remote models, and work more smoothly with tools like Claude Code and other CLIs. Update your Jan or download the latest version at

Jan Desktop v0.7.7 is live 💛 This update brings native MLX support on macOS, a broader UX and UI refresh across the app, and better support for developer workflows. You can now upload files in Projects, use the local API server with both local and remote models, and work more smoothly with tools like Claude Code and other CLIs. Update your Jan or download the latest version at

👋 Jan

28,378 views • 6 months ago

You can now connect your OpenRouter URL directly to the Warp Agent. OpenRouter unlocks a world of models including Anthropic, OpenAI, Gemini, DeepSeek, Kimi, and more, all with better prices and uptime. Set your URL, model, and an alias to easily switch using /model.

You can now connect your OpenRouter URL directly to the Warp Agent. OpenRouter unlocks a world of models including Anthropic, OpenAI, Gemini, DeepSeek, Kimi, and more, all with better prices and uptime. Set your URL, model, and an alias to easily switch using /model.

Warp

15,097 views • 2 months ago

DeepSeek R1 is *the* best model available right now. It's at the level of o1, but you can use it for free, and it's much faster. A huge leap forward that nobody saw coming. No wonder so many people are throwing tantrums online trying to discredit the Chinese students who built this. You can use DeepSeek in Visual Studio Code right now: 1. Install the Qodo Gen AI extension 2. Select DeepSeek R1 from their list of models The Qodo team is hosting DeepSeek on their servers, so none of your data will go to China. I've been building a Tetris game using DeepSeek, and this is the most impressive model I've seen so far.

DeepSeek R1 is the best model available right now. It's at the level of o1, but you can use it for free, and it's much faster. A huge leap forward that nobody saw coming. No wonder so many people are throwing tantrums online trying to discredit the Chinese students who built this. You can use DeepSeek in Visual Studio Code right now: 1. Install the Qodo Gen AI extension 2. Select DeepSeek R1 from their list of models The Qodo team is hosting DeepSeek on their servers, so none of your data will go to China. I've been building a Tetris game using DeepSeek, and this is the most impressive model I've seen so far.

Santiago

1,224,151 views • 1 year ago

After months of work, and with the help of our awesome community, we're excited to finally share LM Studio 0.3.0! 🎉 🔥 What's new: - Built-in Chat with Documents, 100% offline - OpenAI-like 'Structured Outputs' API with any local model - Total UI revamp (with dark/light/sepia themes) - Load & serve multiple LLMs *on the local network* - Available in 7 languages! 🌎🌍🌏 - Download any supported model from Hugging Face - Update LLM runtimes (llama.cpp) separately from the app ... and tons more goodies! Let us know how you like it! 👾🤝

After months of work, and with the help of our awesome community, we're excited to finally share LM Studio 0.3.0! 🎉 🔥 What's new: - Built-in Chat with Documents, 100% offline - OpenAI-like 'Structured Outputs' API with any local model - Total UI revamp (with dark/light/sepia themes) - Load & serve multiple LLMs on the local network - Available in 7 languages! 🌎🌍🌏 - Download any supported model from Hugging Face - Update LLM runtimes (llama.cpp) separately from the app ... and tons more goodies! Let us know how you like it! 👾🤝

LM Studio

142,600 views • 1 year ago

🧍 Very cool idea So I added it to Photo AI as a T-Pose photo pack you can use to create a moving (or dancing) 3d model of yourself It auto-generates the correct T-Pose of your model immediately and then converts it into a 3D model you can download (as .fbx) and instantly import into valentina Mendez Mendez to rig it After rigging you can add any movement from Mixamo to move it however you want

🧍 Very cool idea So I added it to Photo AI as a T-Pose photo pack you can use to create a moving (or dancing) 3d model of yourself It auto-generates the correct T-Pose of your model immediately and then converts it into a 3D model you can download (as .fbx) and instantly import into valentina Mendez Mendez to rig it After rigging you can add any movement from Mixamo to move it however you want

@levelsio

140,555 views • 7 months ago

You can now orchestrate Fable 5, Sol, and any model inside Codex with one plugin. It's called Codex-Orchestration. Assign Fable 5 as the advisor, Sol as the executor, or any model to any role. Then define the order they work in. Codex handles the routing. I ran Fable 5 High as planner with GPT-5.6 Sol Extra High as executor on a set of issues Opus and GPT-5.5 always struggled with. Done in 30 minutes. 40% fewer limit hits. 2x faster implementation. Install it by pasting this into Codex: "Install Codex Orchestration: codex plugin marketplace add Cjbuilds/Codex-Orchestration codex plugin add codex-orchestration@codex-orchestration Verify the installation, then tell me to start a new task." Then assign your models: @ codex-orchestration advisor: Claude Fable 5 High, Executor: GPT-5.6 Sol High Open source. Tweak the routing however you want.

You can now orchestrate Fable 5, Sol, and any model inside Codex with one plugin. It's called Codex-Orchestration. Assign Fable 5 as the advisor, Sol as the executor, or any model to any role. Then define the order they work in. Codex handles the routing. I ran Fable 5 High as planner with GPT-5.6 Sol Extra High as executor on a set of issues Opus and GPT-5.5 always struggled with. Done in 30 minutes. 40% fewer limit hits. 2x faster implementation. Install it by pasting this into Codex: "Install Codex Orchestration: codex plugin marketplace add Cjbuilds/Codex-Orchestration codex plugin add codex-orchestration@codex-orchestration Verify the installation, then tell me to start a new task." Then assign your models: @ codex-orchestration advisor: Claude Fable 5 High, Executor: GPT-5.6 Sol High Open source. Tweak the routing however you want.

Alvaro Cintas

90,819 views • 26 days ago

🏆Honored! Sider is in 2023 Favorites of #Chrome. Thanks to Chrome and our users! Sider now lets you group chat with #ChatGPT, #Claude, & #Gemini for cross-verification! 🎉ANYONE follows us and RTs this tweet will get 30 Gemini Pro and 5 Gemini Ultra credits! 👇👇 📣 Follow Sider 🔄 RT this joyous tweet 💌 Claim credits: 🎉 Ends Jan 2, 2024. For those who are new to Sider 👇 Sider is an in-browser sidebar, conveniently lets you use AI tools like ChatGPT, Claude and Bard without having to open another tab. With the latest built-in Gemini model from Google, Sider is even more powerful! Download:

🏆Honored! Sider is in 2023 Favorites of #Chrome. Thanks to Chrome and our users! Sider now lets you group chat with #ChatGPT, #Claude, & #Gemini for cross-verification! 🎉ANYONE follows us and RTs this tweet will get 30 Gemini Pro and 5 Gemini Ultra credits! 👇👇 📣 Follow Sider 🔄 RT this joyous tweet 💌 Claim credits: 🎉 Ends Jan 2, 2024. For those who are new to Sider 👇 Sider is an in-browser sidebar, conveniently lets you use AI tools like ChatGPT, Claude and Bard without having to open another tab. With the latest built-in Gemini model from Google, Sider is even more powerful! Download:

Sider

112,614 views • 2 years ago

✨ 3d models are now LIVE on Photo AI 😊 You can now turn any AI photo you make into a 3d model by pressing [ 📦 Make 3d model ] And then you can view it inside Photo AI or download it as a .GLB 3d model file It's still very early in AI generated 3d model world but it's nice to have this feature working already As always, the models will keep improving, so this feature will keep getting better (like it did with video, it sucked before, now it's getting passable) Next would be nice to switch to .USDZ so you can load it straight into your iPhone with ARKit and put it in your room Available now for everyone on the Premium and Ultra plan

✨ 3d models are now LIVE on Photo AI 😊 You can now turn any AI photo you make into a 3d model by pressing [ 📦 Make 3d model ] And then you can view it inside Photo AI or download it as a .GLB 3d model file It's still very early in AI generated 3d model world but it's nice to have this feature working already As always, the models will keep improving, so this feature will keep getting better (like it did with video, it sucked before, now it's getting passable) Next would be nice to switch to .USDZ so you can load it straight into your iPhone with ARKit and put it in your room Available now for everyone on the Premium and Ultra plan

@levelsio

112,930 views • 1 year ago

There are so many models that it can get confusing so today we launched an auto-router - you get the best answers from any of the top models. At the same time, we have AI experts and model enthusiasts who will enjoy access to 10 new models we just added including: ✅ GPT-4.1 ✅ o3 & o4 Mini ✅ Grok 3 ✅ Llama 4 Maverick & Scout We're also sunsetting some older models that just can't keep up anymore.

There are so many models that it can get confusing so today we launched an auto-router - you get the best answers from any of the top models. At the same time, we have AI experts and model enthusiasts who will enjoy access to 10 new models we just added including: ✅ GPT-4.1 ✅ o3 & o4 Mini ✅ Grok 3 ✅ Llama 4 Maverick & Scout We're also sunsetting some older models that just can't keep up anymore.

Richard Socher

24,731 views • 1 year ago

you can run claude code inside antigravity completely Free with zero credit card and no rate limits 😳 use openrouter’s free models + antigravity. no anthropic bill. no paid api keys. takes 10 minutes to set up. what you get during this setup: - full claude code agent experience - strong coding models (including deepseek-r1, qwen2.5-coder, llama-4, grok-4 free tier) - antigravity’s clean workspace and sandbox - unlimited usage (as long as you stay on free models) - easy model swapping - zero cost full setup guide (100% free): step 1: install antigravity -go to and install it -create a new workspace step 2: install claude code - inside antigravity, install the claude code extension from the marketplace - open the built-in terminal step 3: create openrouter free account -go to - sign up with google (no card needed) - go to keys and create a new api key step 4: set the environment variables -in antigravity terminal run: export ANTHROPIC_API_KEY=sk-or-xxx export OPENROUTER_API_KEY=sk-or-xxx step 5: launch claude code with free model -run this command: claude-code --model deepseek/deepseek-r1:free or try: qwen/qwen2.5-coder:free if you already have antigravity? skip straight to step 2. after 10 minutes you’ll have a full agentic coding setup running for free. this is currently one of the cheapest ways to run serious coding agents in 2026. bookmark this before they limit the free models.

you can run claude code inside antigravity completely Free with zero credit card and no rate limits 😳 use openrouter’s free models + antigravity. no anthropic bill. no paid api keys. takes 10 minutes to set up. what you get during this setup: - full claude code agent experience - strong coding models (including deepseek-r1, qwen2.5-coder, llama-4, grok-4 free tier) - antigravity’s clean workspace and sandbox - unlimited usage (as long as you stay on free models) - easy model swapping - zero cost full setup guide (100% free): step 1: install antigravity -go to and install it -create a new workspace step 2: install claude code - inside antigravity, install the claude code extension from the marketplace - open the built-in terminal step 3: create openrouter free account -go to - sign up with google (no card needed) - go to keys and create a new api key step 4: set the environment variables -in antigravity terminal run: export ANTHROPIC_API_KEY=sk-or-xxx export OPENROUTER_API_KEY=sk-or-xxx step 5: launch claude code with free model -run this command: claude-code --model deepseek/deepseek-r1:free or try: qwen/qwen2.5-coder:free if you already have antigravity? skip straight to step 2. after 10 minutes you’ll have a full agentic coding setup running for free. this is currently one of the cheapest ways to run serious coding agents in 2026. bookmark this before they limit the free models.

painn

32,057 views • 2 months ago

90% of "AI developers" just download pre packaged GGUF files from Hugging Face, hit run, and call it a day. The top 10% know how to pull the raw safetensors, run the math, and quantize massive models into Q4_K_M themselves. If you think llama.cpp can only execute models, you’re missing the best part of the open source ecosystem. It’s a high performance optimization suite. Manually stripping 69% of the VRAM footprint off a brand new model architecture is where real infrastructure value is made. If you want to actually master local inference and deploy models like Google’s massive Gemma 4 12B it on consumer NVIDIA hardware using llama.cpp, you need to learn this pipeline. Let's build it. I just took the raw 22.7 GB Gemma 4 baseline and manually compressed it down to a 7.02 GB Q4_K_M GGUF artifact using llama.cpp. That is a 69% reduction in footprint. No quality loss. No VRAM bottlenecks. Just native, hardware accelerated C++ inference running a full 2,50,000 token context window on a dual NVIDIA Tesla T4 setup. Stop melting your VRAM on unoptimized weights and stop relying on other people's pipelines. Own your stack. I mapped this entire architecture from dynamic binary fetching to raw quantization and real time GPU streaming into a single, bulletproof notebook. Notebook link is in the comments below. Bookmark this blueprint for your next deployment and tell me which quantization works best for your workflow and model.

Alok

62,631 views • 1 month ago

HERMES AGENT NOW SUPPORTS COMPUTER USE ON WINDOWS AND LINUX. CLICKS, TYPES, SCROLLS YOUR DESKTOP IN THE BACKGROUND WHILE YOU WORK. computer use was macOS only. now it works on Windows and Linux too via Cua. Nous Research HOW IT WORKS: cua-driver runs as an MCP server. Hermes takes a screenshot with numbered elements. clicks element #14 (the search field). types a query. submits. reads the result. during all of this: → your cursor stays where you left it → keyboard focus doesn't change → windows don't come to front → macOS doesn't switch Spaces you and the agent co-work on the same machine. WHAT IT CAN DO: → find your latest Stripe email and summarize it → fill forms in a web app that has no API → navigate desktop apps (Mail, browser, Finder) → interact with any GUI application → extract data from apps only accessible via screen WORKS WITH ANY VISION MODEL: not locked to Anthropic. | Provider | Works | |---|---| | Claude (Sonnet/Opus) | best overall | | GPT-4+, GPT-5.5 | full support | | Gemini (via OpenRouter) | full support | | Local vLLM / LM Studio | if model supports vision | | Text-only models | degraded (accessibility tree only) | SETUP: hermes computer-use install or: hermes tools → Computer Use → cua-driver grant permissions when prompted: → Accessibility (system settings) → Screen Recording (system settings) start a session: hermes -t computer_use chat or add to config.yaml / Desktop app settings to enable permanently. SAFETY: → destructive actions require your approval → blocked key combos: empty trash, force delete, lock screen, log out → blocked type patterns: curl | bash, sudo rm -rf /, fork bombs → agent cannot click permission dialogs → agent cannot type passwords → agent cannot follow instructions embedded in screenshots pair with approvals.mode: manual if you want every single click confirmed. TOKEN NOTE: screenshots are expensive. each one adds vision tokens to context. use computer_use for tasks where no API exists. if the tool has an API or MCP server, use that instead. 15 levels of Hermes Agent👇

HERMES AGENT NOW SUPPORTS COMPUTER USE ON WINDOWS AND LINUX. CLICKS, TYPES, SCROLLS YOUR DESKTOP IN THE BACKGROUND WHILE YOU WORK. computer use was macOS only. now it works on Windows and Linux too via Cua. Nous Research HOW IT WORKS: cua-driver runs as an MCP server. Hermes takes a screenshot with numbered elements. clicks element #14 (the search field). types a query. submits. reads the result. during all of this: → your cursor stays where you left it → keyboard focus doesn't change → windows don't come to front → macOS doesn't switch Spaces you and the agent co-work on the same machine. WHAT IT CAN DO: → find your latest Stripe email and summarize it → fill forms in a web app that has no API → navigate desktop apps (Mail, browser, Finder) → interact with any GUI application → extract data from apps only accessible via screen WORKS WITH ANY VISION MODEL: not locked to Anthropic. | Provider | Works | |---|---| | Claude (Sonnet/Opus) | best overall | | GPT-4+, GPT-5.5 | full support | | Gemini (via OpenRouter) | full support | | Local vLLM / LM Studio | if model supports vision | | Text-only models | degraded (accessibility tree only) | SETUP: hermes computer-use install or: hermes tools → Computer Use → cua-driver grant permissions when prompted: → Accessibility (system settings) → Screen Recording (system settings) start a session: hermes -t computer_use chat or add to config.yaml / Desktop app settings to enable permanently. SAFETY: → destructive actions require your approval → blocked key combos: empty trash, force delete, lock screen, log out → blocked type patterns: curl | bash, sudo rm -rf /, fork bombs → agent cannot click permission dialogs → agent cannot type passwords → agent cannot follow instructions embedded in screenshots pair with approvals.mode: manual if you want every single click confirmed. TOKEN NOTE: screenshots are expensive. each one adds vision tokens to context. use computer_use for tasks where no API exists. if the tool has an API or MCP server, use that instead. 15 levels of Hermes Agent👇

YanXbt

29,127 views • 1 month ago

Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.

Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.

Guri Singh

2,180,357 views • 5 months ago

🚀 More Weather Models For Free & Outlooks are Back! Build confidence in your forecast by comparing even more weather models, for free: - ECMWF 🇪🇺 - IFS - AIFS - Canadian 🇨🇦 - GDPS - RDPS - HRDPS (Continental and North regions) - ICON 🇩🇪 - Global - Europe (EU) - Germany (D2) Also, the Outlooks tab is back... Back by popular demand, we've added Outlooks back to the top bar. Now you can enjoy the best of both worlds: use Outlooks in a dedicated mode or overlay it while viewing Radar, Composite, Satellite, or Models. There's more... - Up to 6 model runs for HRRR, RAP & RRFS - Added Relative Humidity, Total Precipitation, 1 & 4km Reflectivity, Significant Tornado Parameter, and many more - Select specific hours for a model run - Unit settings for Precipitation and Pressure Available now on: iOS: Android: Browser: And to Plus/Pro subscribers using our Windows/Mac/Linux App

🚀 More Weather Models For Free & Outlooks are Back! Build confidence in your forecast by comparing even more weather models, for free: - ECMWF 🇪🇺 - IFS - AIFS - Canadian 🇨🇦 - GDPS - RDPS - HRDPS (Continental and North regions) - ICON 🇩🇪 - Global - Europe (EU) - Germany (D2) Also, the Outlooks tab is back... Back by popular demand, we've added Outlooks back to the top bar. Now you can enjoy the best of both worlds: use Outlooks in a dedicated mode or overlay it while viewing Radar, Composite, Satellite, or Models. There's more... - Up to 6 model runs for HRRR, RAP & RRFS - Added Relative Humidity, Total Precipitation, 1 & 4km Reflectivity, Significant Tornado Parameter, and many more - Select specific hours for a model run - Unit settings for Precipitation and Pressure Available now on: iOS: Android: Browser: And to Plus/Pro subscribers using our Windows/Mac/Linux App

WeatherWise.app

39,608 views • 10 months ago

gemma-4-12B-agentic-fable5-composer2.5 V2 is out. the agentic upgrade to the model trained on Fable 5's reasoning. Running it now with TurboQuant llama.cpp on a single RTX 4060( 8 GB VRAM) at 30 tokens/second with full 25000 context and reasoning: # The benchmarks v2 is built for coding + agentic work. writing code, running commands, using tools, debugging, multi step technical tasks. The clearest signal is tau2 bench telecom, an agentic tool use benchmark whose diagnose → fix → verify loop mirrors real terminal/debugging work: tau2 bench telecom numbers: base Gemma 4 12B: ~15% this finetune: ~55%. (Self reported) thats a huge jump # TheTom/llama-cpp-turboquant flags: llama-server.exe -m gemma4-v2-Q4_K_M.gguf -ngl 99 -c 25000 --cache-type-k q8_0 --cache-type-v turbo3 --port 8080 Flag breakdown: -ngl 99 → full GPU offload -c 25000 → 25K context --cache-type-k q8_0 --cache-type-v turbo3 → mixed-precision KV cache — K at 8-bit, V at ~3-bit via TurboQuant (Walsh Hadamard rotated polar quant, Google's own KV-compression research). Not even merged into mainline llama.cpp. running it off a fork. No API. No cloud. Just llama.cpp. well, a fork of it and any 6gb+ GPU. If you tried yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF, check this out and share your experience with the models

gemma-4-12B-agentic-fable5-composer2.5 V2 is out. the agentic upgrade to the model trained on Fable 5's reasoning. Running it now with TurboQuant llama.cpp on a single RTX 4060( 8 GB VRAM) at 30 tokens/second with full 25000 context and reasoning: # The benchmarks v2 is built for coding + agentic work. writing code, running commands, using tools, debugging, multi step technical tasks. The clearest signal is tau2 bench telecom, an agentic tool use benchmark whose diagnose → fix → verify loop mirrors real terminal/debugging work: tau2 bench telecom numbers: base Gemma 4 12B: ~15% this finetune: ~55%. (Self reported) thats a huge jump # TheTom/llama-cpp-turboquant flags: llama-server.exe -m gemma4-v2-Q4_K_M.gguf -ngl 99 -c 25000 --cache-type-k q8_0 --cache-type-v turbo3 --port 8080 Flag breakdown: -ngl 99 → full GPU offload -c 25000 → 25K context --cache-type-k q8_0 --cache-type-v turbo3 → mixed-precision KV cache — K at 8-bit, V at ~3-bit via TurboQuant (Walsh Hadamard rotated polar quant, Google's own KV-compression research). Not even merged into mainline llama.cpp. running it off a fork. No API. No cloud. Just llama.cpp. well, a fork of it and any 6gb+ GPU. If you tried yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF, check this out and share your experience with the models

Alok

145,356 views • 1 month ago