Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Google Turbo Quant running Locally in Atomic Chat MacBook Air M4 16 GB Model: QWEN3.5-9B Context window: 50000 Summarising 20000 words in just seconds.. You can do 3x larger context window, processing 3x faster than before!

atomic.chat

11,970 subscribers

435,661 views • 4 months ago •via X (Twitter)

Education News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Qwen just published the 'Thinking' variant of this model 🔥 So you can run a model EVEN MORE powerful than GPT-4o locally! - Still only 3B active parameters - Open source license - 256k context window extendable to 1M - Strong in math, science and coding Details below ↓

Qwen just published the 'Thinking' variant of this model 🔥 So you can run a model EVEN MORE powerful than GPT-4o locally! - Still only 3B active parameters - Open source license - 256k context window extendable to 1M - Strong in math, science and coding Details below ↓

Paul Couvert

106,261 views • 1 year ago

Memory just landed inside Notch Agents. Brand colors. Voice clones. Do-not-say words. Taste. All read before every generation. On-brand ads. Faster. With brand context. Upload your brand context once. Or teach the agent inline as you chat. Each chat is no longer a re-introduction. It's training. Drop your brand URLs and instructions in the comments. We'll spin ads around your context.

Vinay Jain

647,024 views • 2 months ago

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

Lior Alexander

83,432 views • 2 years ago

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,207,042 views • 3 months ago

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,129 views • 2 months ago

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours

Noisy

79,665 views • 1 month ago

WHY IS NO ONE TALKING ABOUT THIS?? Gemma 3n model was one of the best surprises for me. The fact that you can run it on edge devices even with just 2GB of RAM is impressive. A few weeks back, I was on holiday and used the Gemini Live feature a lot. But I kept running into issues whenever I was in a place where the network wasn’t reliable. Gemma 3n : > Multimodal: Supports text and image inputs; video/audio in development. > Context Window: Up to 128K tokens (32K for 1B model). > Multilingual: Trained on 140+ languages. > Privacy: Offline, on-device processing for data security. > Model Sizes: E2B (5B parameters, 2GB RAM) and E4B (10–12B parameters, 3GB RAM). Video credit: Google YT

WHY IS NO ONE TALKING ABOUT THIS?? Gemma 3n model was one of the best surprises for me. The fact that you can run it on edge devices even with just 2GB of RAM is impressive. A few weeks back, I was on holiday and used the Gemini Live feature a lot. But I kept running into issues whenever I was in a place where the network wasn’t reliable. Gemma 3n : > Multimodal: Supports text and image inputs; video/audio in development. > Context Window: Up to 128K tokens (32K for 1B model). > Multilingual: Trained on 140+ languages. > Privacy: Offline, on-device processing for data security. > Model Sizes: E2B (5B parameters, 2GB RAM) and E4B (10–12B parameters, 3GB RAM). Video credit: Google YT

AshutoshShrivastava

210,490 views • 1 year ago

MiniMax-M2.1 running fully local in AWQ-4Bit with full context window (170 GB VRAM w full context) - 1000~ to 16,000~ tps prefill - 100~ tps generation speeds - Opencode It’s doing real work, updating my blog with little steering or specificity. The problem with local LLMs is that they require too much steering, this means baby sitting which I don’t have the time to do MiniMax cracked the cost, intelligence, and speed challenge, I would say this is a top tier model. I run frontier models like Gemini and it just fails to call tools, in this year lol… ——————— I think glm-4.?-air is needed still. We need a viable model at each hardware entry point, a Mac M1 Ultra 192GB? is relatively cheap 5k to be able to run this model at 40 tps is a huge societal unlock. Smaller models can be good but size matters :p

MiniMax-M2.1 running fully local in AWQ-4Bit with full context window (170 GB VRAM w full context) - 1000~ to 16,000~ tps prefill - 100~ tps generation speeds - Opencode It’s doing real work, updating my blog with little steering or specificity. The problem with local LLMs is that they require too much steering, this means baby sitting which I don’t have the time to do MiniMax cracked the cost, intelligence, and speed challenge, I would say this is a top tier model. I run frontier models like Gemini and it just fails to call tools, in this year lol… ——————— I think glm-4.?-air is needed still. We need a viable model at each hardware entry point, a Mac M1 Ultra 192GB? is relatively cheap 5k to be able to run this model at 40 tps is a huge societal unlock. Smaller models can be good but size matters :p

0xSero

23,836 views • 7 months ago

🚨 do you understand what happened to Claude.. Anthropic just shipped Opus 4.8 with something called dynamic workflows. It no longer works alone - it spins up hundreds of agents that argue, verify, and break each other's work until the answer is right. - A dev used it to port Bun from Zig to Rust in 11 days - 750,000 lines of Rust, 99.8 percent of tests passing - Hundreds of agents ran in parallel with two reviewers per file - Fast mode runs the same model 2.5x faster and 3x cheaper than before work you used to plan in quarters now finishes before the weekend

🚨 do you understand what happened to Claude.. Anthropic just shipped Opus 4.8 with something called dynamic workflows. It no longer works alone - it spins up hundreds of agents that argue, verify, and break each other's work until the answer is right. - A dev used it to port Bun from Zig to Rust in 11 days - 750,000 lines of Rust, 99.8 percent of tests passing - Hundreds of agents ran in parallel with two reviewers per file - Fast mode runs the same model 2.5x faster and 3x cheaper than before work you used to plan in quarters now finishes before the weekend

BuBBliK

431,325 views • 2 months ago

first impressions of qwen 3.5 27B dense on a single RTX 3090. 35 tok/s. from 4K all the way to 300K+ context. no speed drop. hermes 4.3 started at 35 and degraded to 15 as context filled. qwen dense holds. MoE held 112 flat. 3x faster but only 3B of 35B active per token. architecture tradeoff. Q4_K_M on 16.7GB. native context 262K. pushed past training limit to 376K before VRAM ceiling on 24GB. tried q8 KV cache at 262K, speed collapsed to 11 tok/s. q4_0 KV is the sweet spot. flash attention mandatory. built in reasoning mode. the model thinks step by step before it answers. full chain of thought surviving Q4 quant. 1,799+ token thinking chains with self correction loops. on a single consumer GPU. gave it one prompt: "build a realtime particle galaxy simulation in one HTML file." 3,340 tokens. 95 seconds. one shot. ran on first load. full reasoning and coding in the video below. optimal config if you want to skip the hours of testing: llama-server -ngl 99 -c 262144 -fa on --cache-type-k q4_0 --cache-type-v q4_0 this is just the warmup. octopus invaders is next: 10 files, 3,400+ lines, zero steering. the prompt hermes quit at 22%. already more impressed than expected. full results coming soon.

first impressions of qwen 3.5 27B dense on a single RTX 3090. 35 tok/s. from 4K all the way to 300K+ context. no speed drop. hermes 4.3 started at 35 and degraded to 15 as context filled. qwen dense holds. MoE held 112 flat. 3x faster but only 3B of 35B active per token. architecture tradeoff. Q4_K_M on 16.7GB. native context 262K. pushed past training limit to 376K before VRAM ceiling on 24GB. tried q8 KV cache at 262K, speed collapsed to 11 tok/s. q4_0 KV is the sweet spot. flash attention mandatory. built in reasoning mode. the model thinks step by step before it answers. full chain of thought surviving Q4 quant. 1,799+ token thinking chains with self correction loops. on a single consumer GPU. gave it one prompt: "build a realtime particle galaxy simulation in one HTML file." 3,340 tokens. 95 seconds. one shot. ran on first load. full reasoning and coding in the video below. optimal config if you want to skip the hours of testing: llama-server -ngl 99 -c 262144 -fa on --cache-type-k q4_0 --cache-type-v q4_0 this is just the warmup. octopus invaders is next: 10 files, 3,400+ lines, zero steering. the prompt hermes quit at 22%. already more impressed than expected. full results coming soon.

Sudo su

120,788 views • 5 months ago

gpt-5.5 just dropped. and it changes EVERYTHING for traders. openai stopped calling it a chat model. they call it an "agent runtime" now. that means one thing: AI that doesn't wait for you - it acts. here's what gpt-5.5 can actually do: → runs multi-step trading workflows autonomously → operates computer interfaces without human input → plans, iterates, executes - in a terminal, not a chat window → 82.7% on Terminal-Bench 2.0 - real command-line task execution the only question is: are you already running one? full FREE Guide here → like + bookmark. this is the wave.

gpt-5.5 just dropped. and it changes EVERYTHING for traders. openai stopped calling it a chat model. they call it an "agent runtime" now. that means one thing: AI that doesn't wait for you - it acts. here's what gpt-5.5 can actually do: → runs multi-step trading workflows autonomously → operates computer interfaces without human input → plans, iterates, executes - in a terminal, not a chat window → 82.7% on Terminal-Bench 2.0 - real command-line task execution the only question is: are you already running one? full FREE Guide here → like + bookmark. this is the wave.

AlphaCartel

56,823 views • 3 months ago

HERMES AGENT CROSSED 140,000 GITHUB STARS IN 3 MONTHS AND JUST BECAME THE MOST USED AGENT IN THE WORLD. Most AI agents forget everything between sessions. Hermes writes its own skills from experience. Next time it runs the skill, improves it, and gets faster. Independent benchmarks show agents with 20+ self-created skills complete similar tasks 40% faster than fresh instances. Qwen 3.6 where the 35B version outperforms last year's 120B models at one third the memory footprint. DGX Spark with 128GB unified memory running everything locally at $0 per month after hardware. The setup takes 30 minutes. LM Studio plus Qwen 3.6 27B for the model server. One install script for Hermes. One config connecting them. Set context window to 65,536 tokens or nothing works. After one month of daily use your skills directory has 20 to 50 learned workflows. Your Hermes is genuinely different from anyone else's.

HERMES AGENT CROSSED 140,000 GITHUB STARS IN 3 MONTHS AND JUST BECAME THE MOST USED AGENT IN THE WORLD. Most AI agents forget everything between sessions. Hermes writes its own skills from experience. Next time it runs the skill, improves it, and gets faster. Independent benchmarks show agents with 20+ self-created skills complete similar tasks 40% faster than fresh instances. Qwen 3.6 where the 35B version outperforms last year's 120B models at one third the memory footprint. DGX Spark with 128GB unified memory running everything locally at $0 per month after hardware. The setup takes 30 minutes. LM Studio plus Qwen 3.6 27B for the model server. One install script for Hermes. One config connecting them. Set context window to 65,536 tokens or nothing works. After one month of daily use your skills directory has 20 to 50 learned workflows. Your Hermes is genuinely different from anyone else's.

Lummox

69,676 views • 2 months ago

$I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.$

I just crammed the updated Gemma 4 26B A4B QAT (MoE) with 180k context into an 8GB RTX 4060 (8 GB VRAM + 16 GB RAM only!!) and optimized the batch size. 23 tokens/sec decode, 300 tokens/sec prefill Yesterday I showed you a Gemma 4 31B dense model running flawlessly on an RTX 4090. Today, we're breaking the VRAM bank on a budget card using Unsloth’s new Gemma 4 26B (A4B) QAT quants. Following Google’s chat template update that boosted agentic benchmarks by +10%, I pushed this model to its absolute limits. Here is how you squeeze 250k context out of 8GB of VRAM. # The Setup & The Optimization - Hardware: Nvidia RTX 4060 (8GB VRAM) + 16GB System RAM - Environment: CUDA 13.0 build of llama.cpp - Model: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf - Prompt: 28,000 tokens of prompt for each run If you read my L2 cache breakdown (attached in replies), you know the 4060’s 24MB cache maxes out at `-b 1024 -ub 1024`. Push past that, and prefill crashes. I locked those flags in for every test below to ensure maximum GEMM throughput. # 1. The Raw Context Push (Unquantized KV Cache) First, I wanted to see how far pure 8GB VRAM + 16GB RAM could stretch without touching the KV cache: - 80k Context: Prefill 385 t/s | Decode 25.5 t/s - 120k Context: Prefill 270 t/s | Decode 24 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 120000 --port 8080 -ub 1024 -b 1024 Without KV quantization, 120k is your hard ceiling. push past that prefill throughput drops off a cliff, making the model practically unusable for large agentic workloads. # 2. The Q8 KV Cache Lifeline To survive 250k context on a budget card, you have to quantize the KV cache. I enabled 8 bit KV cache (`-ctk q8_0 -ctv q8_0`) and re ran: - 180k Context: Prefill 280 t/s | Decode 22.8 t/s - 250k Context: Prefill 115 t/s | Decode 20 t/s llama.cpp flags: .\llama-server -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 180000 --port 8080 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 Result: Q8 KV cache brings 250k context back from the dead. Decode speed stabilizes at a highly usable 20 t/s. You are trading a very small bit amount of reasoning precision for an extra 130,000 tokens of context window. if you own a single rtx 3050, 3060, 3070, 4050, 4060, 5050 or 5060, you must try this model and optimize your batch size for higher prefill. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. What model are you running on your 6GB, 8GB or 12GB cards right now? Let's see your setups.

Alok

36,617 views • 16 days ago

holy shit, i built a better Cluely. introducing: Obscura, a better, more performant Cluely. Obscura helps you during your meeting and interview calls, in real time, with more context and a built-in agent to websearch in less than a second. it is completely INVISIBLE to your screen recorder, screen share and cannot be detected during your google calls or zoom calls no matter what. some functionalities: 1. toggle visibility: so that only you can see it 2. panic kill: if you wanna kill it in a second 3. screenshot ocr: for screenshot capabilities 4. context window: pass any context in .md or .docx file for it to read before your calls and Obscura will understand it 5. performative built-in agent: to search for anything in less than a second. it uses Deepgram for STT model and Anthropic / OpenAI api keys to generate answers. it takes $1 for 20 hours of your calls as compared to cluely's $20/mo or $75/mo plan (but that depends on tokens consumed on both models). releasing a .dmg for people for people to use. access will be given to people who share this with their friends. I will roll out a referral program so that people can share it with others. feel free to give some feedback. thanks to rahat for being on call with me and testing it.

holy shit, i built a better Cluely. introducing: Obscura, a better, more performant Cluely. Obscura helps you during your meeting and interview calls, in real time, with more context and a built-in agent to websearch in less than a second. it is completely INVISIBLE to your screen recorder, screen share and cannot be detected during your google calls or zoom calls no matter what. some functionalities: 1. toggle visibility: so that only you can see it 2. panic kill: if you wanna kill it in a second 3. screenshot ocr: for screenshot capabilities 4. context window: pass any context in .md or .docx file for it to read before your calls and Obscura will understand it 5. performative built-in agent: to search for anything in less than a second. it uses Deepgram for STT model and Anthropic / OpenAI api keys to generate answers. it takes $1 for 20 hours of your calls as compared to cluely's $20/mo or $75/mo plan (but that depends on tokens consumed on both models). releasing a .dmg for people for people to use. access will be given to people who share this with their friends. I will roll out a referral program so that people can share it with others. feel free to give some feedback. thanks to rahat for being on call with me and testing it.

Naruto11.eth

15,760 views • 4 months ago

🚨 THIS GUY JUST TURNED CLAUDE INTO A FULL-BLOWN TRADING TERMINAL BY PASTING ONE URL!! Liquid's MCP connector feeds Claude live pricing, funding rates, open interest, positioning data, and even lets you execute trades, all from the chat window Setup takes ~30 seconds: Settings → Connectors → paste Liquid URL → authorize. That's it It sounds like a gimmick until you see the output: Claude drops a full market breakdown, macro context, risk-off regime, sector rotation, then gives you 5 actionable picks: WTI Oil long, MU dip buy, NASDAQ short, MRVL long, ZEC momentum play You ask 'analyze MU for a long trade' and it pulls live data: $870.94 price, -2.69% 24h, $127M OI, funding at 0.0028%. Then breaks down positioning, smart money is 36.2% short-heavy, meaning squeeze potential if it reverses The wildest part? You can literally submit a 3x leveraged long right from Claude $1,500 size, $500 collateral, market order, one click confirm Most retail traders still pay $50/month for signal groups with less context than a single Claude prompt plugged into live markets Bookmark this. The edge is now one connector URL away

🚨 THIS GUY JUST TURNED CLAUDE INTO A FULL-BLOWN TRADING TERMINAL BY PASTING ONE URL!! Liquid's MCP connector feeds Claude live pricing, funding rates, open interest, positioning data, and even lets you execute trades, all from the chat window Setup takes ~30 seconds: Settings → Connectors → paste Liquid URL → authorize. That's it It sounds like a gimmick until you see the output: Claude drops a full market breakdown, macro context, risk-off regime, sector rotation, then gives you 5 actionable picks: WTI Oil long, MU dip buy, NASDAQ short, MRVL long, ZEC momentum play You ask 'analyze MU for a long trade' and it pulls live data: $870.94 price, -2.69% 24h, $127M OI, funding at 0.0028%. Then breaks down positioning, smart money is 36.2% short-heavy, meaning squeeze potential if it reverses The wildest part? You can literally submit a 3x leveraged long right from Claude $1,500 size, $500 collateral, market order, one click confirm Most retail traders still pay $50/month for signal groups with less context than a single Claude prompt plugged into live markets Bookmark this. The edge is now one connector URL away

cristal

87,506 views • 1 month ago

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!

Alok

42,894 views • 5 days ago

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

kwindla

42,246 views • 1 year ago

Most AI agents are just toys that live in a chat box. They are high on hype, low on utility. You spend more time prompting them than doing actual work. Notion Agent is different. It’s not just a window into an LLM. It’s a layer of intelligence sitting directly on any page on Notion. Check out the attached video. I just asked Notion Agent to write practical examples of each React Hook. Here’s how Notion Agent is helping me build it. • Notion Agent forms a plan: (Search → Read → Decide → Update → Notify). You give it a goal, and it executes multi-step workflows like editing hundreds of pages or updating complex databases. • It can create pages, customize properties, and build entire database views from scratch. • Total Context: It doesn't just search Notion. It pulls from Slack, Google Drive, and the web. It finds the patterns across your entire stack in seconds. Check out Notion Agent here: Appreciate the Notion team for collaborating with me on this deep dive.

Most AI agents are just toys that live in a chat box. They are high on hype, low on utility. You spend more time prompting them than doing actual work. Notion Agent is different. It’s not just a window into an LLM. It’s a layer of intelligence sitting directly on any page on Notion. Check out the attached video. I just asked Notion Agent to write practical examples of each React Hook. Here’s how Notion Agent is helping me build it. • Notion Agent forms a plan: (Search → Read → Decide → Update → Notify). You give it a goal, and it executes multi-step workflows like editing hundreds of pages or updating complex databases. • It can create pages, customize properties, and build entire database views from scratch. • Total Context: It doesn't just search Notion. It pulls from Slack, Google Drive, and the web. It finds the patterns across your entire stack in seconds. Check out Notion Agent here: Appreciate the Notion team for collaborating with me on this deep dive.

Pratham

20,933 views • 6 months ago

Explore state-of-the-art multimodal prompting in our new short course Large Multimodal Model Prompting with Gemini, taught by Erwin Huizenga in collaboration with Google Cloud. One interesting insight from this course: with multimodal models, prompt structure matters significantly. Placing text inputs, such as a patient's medical history, before image inputs, like an X-ray, can enhance the model's ability to contextualize and interpret visual data effectively. In other contexts, such as image captioning, you may get better results by putting the image first. Multimodal models behave differently than text-only LLMs, and effective prompting for models varies depending on the model you’re using. In this course you’ll learn how to effectively prompt Gemini models. Gemini's multimodal capabilities also enable new approaches in AI application development, for example: - The Gemini library handles various video formats (MP4, MOV, MPEG), streamlining applications using these formats. - Large context window (up to 1 million tokens) enables processing of extensive content, like analyzing multiple 50-minute videos simultaneously. - Function calling feature integrates real-time data (e.g., current exchange rates) into model responses. The course demonstrates building multimodal applications with real-world examples including document analyzers that reason across text and graphs simultaneously, video content extractors that find and timestamp specific information from multiple hours of footage, and automated expense report systems processing receipt images while cross-referencing company policies. Sign up here:

Explore state-of-the-art multimodal prompting in our new short course Large Multimodal Model Prompting with Gemini, taught by Erwin Huizenga in collaboration with Google Cloud. One interesting insight from this course: with multimodal models, prompt structure matters significantly. Placing text inputs, such as a patient's medical history, before image inputs, like an X-ray, can enhance the model's ability to contextualize and interpret visual data effectively. In other contexts, such as image captioning, you may get better results by putting the image first. Multimodal models behave differently than text-only LLMs, and effective prompting for models varies depending on the model you’re using. In this course you’ll learn how to effectively prompt Gemini models. Gemini's multimodal capabilities also enable new approaches in AI application development, for example: - The Gemini library handles various video formats (MP4, MOV, MPEG), streamlining applications using these formats. - Large context window (up to 1 million tokens) enables processing of extensive content, like analyzing multiple 50-minute videos simultaneously. - Function calling feature integrates real-time data (e.g., current exchange rates) into model responses. The course demonstrates building multimodal applications with real-world examples including document analyzers that reason across text and graphs simultaneously, video content extractors that find and timestamp specific information from multiple hours of footage, and automated expense report systems processing receipt images while cross-referencing company policies. Sign up here:

Andrew Ng

74,060 views • 1 year ago