Loading video...

Video Failed to Load

Go Home

New TTS banger: Chatterbox Turbo 🤯 Zero-shot model that matches any reference voice with native paralinguistic tags, optimized for low-latency voice agents. ⬇️ Demo available on Hugging Face

victormustar's profile picture

Victor M

25,408 subscribers

41,631 views • 8 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

New Voxtral-4B-TTS is beautiful 💛 it sounds so good that it's whispering at closed models: "your time is over" - Demo available on Hugging Face ⬇️
0:13

New Voxtral-4B-TTS is beautiful 💛 it sounds so good that it's whispering at closed models: "your time is over" - Demo available on Hugging Face ⬇️

Victor M

107,012 views • 4 months ago

llama.cpp isn't just for text LLMs anymore. Pure C++ zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!
0:12

llama.cpp isn't just for text LLMs anymore. Pure C++ zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!

Alok

47,881 views • 8 days ago

Fuck yeah! MaskGCT - New open SoTA Text to Speech model! 🔥 > Zero-shot voice cloning > Emotional TTS > Trained on 100K hours of data > Long form synthesis > Variable speed synthesis > Bilingual - Chinese & English > Available on Hugging Face Fully non-autoregressive architecture: > Stage 1: Predicts semantic tokens from text, using tokens extracted from a speech self-supervised learning (SSL) model > Stage 2: Predicts acoustic tokens conditioned on the semantic tokens. Synthesised: "Would you guys personally like to have a fake fireplace, an electric one, in your house? Or would you rather have a real fireplace? Let me know down below. Okay everybody, that's all for today's video and I hope you guys learned a bunch of furniture vocabulary!" TTS scene keeps getting lit! 🐐
0:15

Fuck yeah! MaskGCT - New open SoTA Text to Speech model! 🔥 > Zero-shot voice cloning > Emotional TTS > Trained on 100K hours of data > Long form synthesis > Variable speed synthesis > Bilingual - Chinese & English > Available on Hugging Face Fully non-autoregressive architecture: > Stage 1: Predicts semantic tokens from text, using tokens extracted from a speech self-supervised learning (SSL) model > Stage 2: Predicts acoustic tokens conditioned on the semantic tokens. Synthesised: "Would you guys personally like to have a fake fireplace, an electric one, in your house? Or would you rather have a real fireplace? Let me know down below. Okay everybody, that's all for today's video and I hope you guys learned a bunch of furniture vocabulary!" TTS scene keeps getting lit! 🐐

Vaibhav (VB) Srivastav

139,105 views • 1 year ago

This free GitHub fork can get you to $5,000/month. Clones any voice for free, writes the script, makes the video. Zero API costs, zero camera. It's MoneyPrinterTurbo Extended - an enhanced fork of the repo that already prints faceless content on autopilot. This version closes the last paid gap: voice. >> What's new in this fork: - Chatterbox TTS built in - completely free voice cloning, no API costs, no rate limits. - Clone any voice from 10-60 seconds of reference audio. - Word-by-word subtitle highlighting that lights up exactly when spoken. - Semantic video matching - pulls clips that actually match the script's meaning instead of random stock footage. Same pipeline as before: topic in, script out, voiceover generated, footage matched, subtitles synced, finished video. The difference is now nothing in the chain costs money. Script generation, voice, footage, editing - all free, all local. // This is the exact setup behind faceless channels pulling $800-3,000+/month once they scale. Run multiple channels in parallel, multiple niches, and the free voice cloning alone removes the biggest recurring cost most people are still paying for. Repo: Open source. MIT license. Bookmark this before you lose the repo.
0:15

This free GitHub fork can get you to $5,000/month. Clones any voice for free, writes the script, makes the video. Zero API costs, zero camera. It's MoneyPrinterTurbo Extended - an enhanced fork of the repo that already prints faceless content on autopilot. This version closes the last paid gap: voice. >> What's new in this fork: - Chatterbox TTS built in - completely free voice cloning, no API costs, no rate limits. - Clone any voice from 10-60 seconds of reference audio. - Word-by-word subtitle highlighting that lights up exactly when spoken. - Semantic video matching - pulls clips that actually match the script's meaning instead of random stock footage. Same pipeline as before: topic in, script out, voiceover generated, footage matched, subtitles synced, finished video. The difference is now nothing in the chain costs money. Script generation, voice, footage, editing - all free, all local. // This is the exact setup behind faceless channels pulling $800-3,000+/month once they scale. Run multiple channels in parallel, multiple niches, and the free voice cloning alone removes the biggest recurring cost most people are still paying for. Repo: Open source. MIT license. Bookmark this before you lose the repo.

Atenov int.

39,712 views • 1 month ago

Big moment for text-to-speech. Qwen just open-sourced a text-to-speech model that lets you clone voices, design new ones, and control speech using natural language. Let me explain what I mean: You can literally tell it "speak in a cheerful tone with slight nervousness," and it actually does that. No complex audio engineering needed. What makes this special: - 3-second voice cloning - Covers 10 languages: English, German, French, and more - Latency as low as 97ms for real-time applications - Supports both streaming and non-streaming generation The model comes in two sizes (0.6B and 1.7B parameters), so you can pick based on your hardware and quality needs. Three modes to work with: 1. Custom Voice: Use pre-built premium voices with instruction-based style control 2. Voice Design: Describe the voice you want in plain English (or Chinese), and the model creates it 3. Voice Clone: Provide a 3-second reference audio and clone that voice The best part? It integrates with vLLM for production deployment and has a simple Python package you can pip install. I've shared a link to the GitHub repo in the next tweet.
0:17

Big moment for text-to-speech. Qwen just open-sourced a text-to-speech model that lets you clone voices, design new ones, and control speech using natural language. Let me explain what I mean: You can literally tell it "speak in a cheerful tone with slight nervousness," and it actually does that. No complex audio engineering needed. What makes this special: - 3-second voice cloning - Covers 10 languages: English, German, French, and more - Latency as low as 97ms for real-time applications - Supports both streaming and non-streaming generation The model comes in two sizes (0.6B and 1.7B parameters), so you can pick based on your hardware and quality needs. Three modes to work with: 1. Custom Voice: Use pre-built premium voices with instruction-based style control 2. Voice Design: Describe the voice you want in plain English (or Chinese), and the model creates it 3. Voice Clone: Provide a 3-second reference audio and clone that voice The best part? It integrates with vLLM for production deployment and has a simple Python package you can pip install. I've shared a link to the GitHub repo in the next tweet.

Akshay 🚀

31,249 views • 6 months ago

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?
0:16

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?

Alok

57,362 views • 5 days ago

My friend who bought a new Model 3 today was convinced that the 3 couldn’t be quieter than his Genesis. He had an injury that basically made his voice the volume of a whisper years ago Talking in the phone in the car is trouble for him because he can’t speak louder than a quiet whisper. I told him to try voice command in the Model 3 and the look in his face as the car picked up every word he said precisely was so fun to see the surprise on his face! He called me on his way home saying this is the best most quietest car he’s ever driven. Before today he has never driven any tesla and now 8 hours later can’t enjoy driving anything else!🔥🔥🔥
0:11

My friend who bought a new Model 3 today was convinced that the 3 couldn’t be quieter than his Genesis. He had an injury that basically made his voice the volume of a whisper years ago Talking in the phone in the car is trouble for him because he can’t speak louder than a quiet whisper. I told him to try voice command in the Model 3 and the look in his face as the car picked up every word he said precisely was so fun to see the surprise on his face! He called me on his way home saying this is the best most quietest car he’s ever driven. Before today he has never driven any tesla and now 8 hours later can’t enjoy driving anything else!🔥🔥🔥

Drake

351,241 views • 1 year ago

Seedance 2 just dropped. This was my first test and it changed everything. Perfect lip sync. Realistic movement. Multi-shot support. Natural voice completely human, zero robotic tone. And the SFX? It nailed the sound effects automatically. Every detail synced perfectly. This is a game-changer. From now on, automation isn't theory it's real. And yes, it's happening on Crafft. With a high-fidelity model like this + the right prompting rules and skills, you can generate endless variations that work every single time. Crafft agents will handle the whole process. One model, infinite outputs. You can try Seedance 2 on CapCut right now. RT + comment "PROMPT" and I'll send you the exact setup (follow so I can DM)
0:13

Seedance 2 just dropped. This was my first test and it changed everything. Perfect lip sync. Realistic movement. Multi-shot support. Natural voice completely human, zero robotic tone. And the SFX? It nailed the sound effects automatically. Every detail synced perfectly. This is a game-changer. From now on, automation isn't theory it's real. And yes, it's happening on Crafft. With a high-fidelity model like this + the right prompting rules and skills, you can generate endless variations that work every single time. Crafft agents will handle the whole process. One model, infinite outputs. You can try Seedance 2 on CapCut right now. RT + comment "PROMPT" and I'll send you the exact setup (follow so I can DM)

Ahmad

31,486 views • 5 months ago

This Kling 3.0 UGC workflow is absolutely insane 🤯 It generates hyper-realistic UGC-style ad videos from a single text prompt — talking heads, unboxings, testimonials, tutorials — in one shot. Perfect for DTC brands and agencies who need to scale UGC without paying $500 per UGC video. Here's the problem: You're either paying creators who take weeks to deliver, or you're stitching together janky 5-second AI clips that look obviously fake. Kling 3.0 solves it: → Write a prompt with character, scene, and dialogue → Get a 10-15 second UGC video with realistic facial expressions → Native voice control — tone, pacing, emotion per line → Multi-shot scenes in one generation (hook → demo → CTA) → Paste negative rules to kill AI artifacts instantly No creator fees. No stitching clips. No uncanny valley. What you get: → 6 copy-paste prompt templates for every UGC format →Voice & dialogue cheat sheet with tone keywords → Multi-shot ad structure (hook → problem → solution → CTA) → Image-to-video workflow for branded product shots → A "where it fails" section so you don't waste credits → ChatGPT template to convert any rough idea into a Kling prompt I built a full prompting guide for UGC ads on TikTok and Meta. Want the guide? > Like this post > Comment "KLING" And I'll send it over (must be following so I can DM)
0:16

This Kling 3.0 UGC workflow is absolutely insane 🤯 It generates hyper-realistic UGC-style ad videos from a single text prompt — talking heads, unboxings, testimonials, tutorials — in one shot. Perfect for DTC brands and agencies who need to scale UGC without paying $500 per UGC video. Here's the problem: You're either paying creators who take weeks to deliver, or you're stitching together janky 5-second AI clips that look obviously fake. Kling 3.0 solves it: → Write a prompt with character, scene, and dialogue → Get a 10-15 second UGC video with realistic facial expressions → Native voice control — tone, pacing, emotion per line → Multi-shot scenes in one generation (hook → demo → CTA) → Paste negative rules to kill AI artifacts instantly No creator fees. No stitching clips. No uncanny valley. What you get: → 6 copy-paste prompt templates for every UGC format →Voice & dialogue cheat sheet with tone keywords → Multi-shot ad structure (hook → problem → solution → CTA) → Image-to-video workflow for branded product shots → A "where it fails" section so you don't waste credits → ChatGPT template to convert any rough idea into a Kling prompt I built a full prompting guide for UGC ads on TikTok and Meta. Want the guide? > Like this post > Comment "KLING" And I'll send it over (must be following so I can DM)

Mike Futia

36,161 views • 6 months ago

Seedance 2.0 Valence-Arousal + FACS Created an example scene to show how valence-arousal and FACS can be used together. Prompt: 15s, cinematic emotional confrontation. Two characters @[chracter sheet ref] stand face-to-face inside a small apartment kitchen late at night. The room is dimly lit by a single warm overhead light and soft city lights leaking through the window. The atmosphere feels emotionally exhausted, tense and painfully intimate, like an argument that has been building for years. Modern cinematic realism, subtle handheld camera movement, shallow depth of field, soft film grain, emotionally restrained acting, realistic silence between dialogue lines. Beat 1: The emotional state remains at high arousal and medium-low valence. Camera: slow handheld side shot circling both characters tight over-the-shoulder close-ups brief eye-level two-shot showing emotional distance FACS Character A: AU4 + AU7 + AU17 Dialogue A: /juː ˈnev.ɚ ˈriː.ə .li lʊkt æt miː/ /juː wɚ ɔːlˌweɪz ˈsʌmˌwɛɹ ɛls/ Voice: tight restrained voice, controlled anger, uneven breathing Character A tries to stay calm while suppressing years of resentment. Beat 2: The emotional state gradually shifts toward very low valence and medium-high arousal. Camera: slow push-in toward Character B extreme close-up on trembling eyes and mouth wide static shot showing silence after the argument peaks FACS Character B: AU1 + AU4 + AU15 + AU25 Dialogue B: /aɪ wəz ˈtɹaɪ.ɪŋ maɪ bɛst/ /aɪ dɪdnt noʊ haʊ tə fɪks ˈɛv.ɹiˌθɪŋ/ Voice: breaking voice, unstable breath support, emotionally collapsing delivery Beat 3: The emotional state remains at very low valence and medium-low arousal. Camera: locked wide shot with silence between them slow close-up on both characters avoiding eye contact subtle rack focus between faces FACS Character A: AU1 + AU15 + AU17 Dialogue A: /ˈmeɪ.bi wiː stɑpt ˈlɪs.ən.ɪŋ ə lɔŋ taɪm əˈgoʊ/ Voice: emotionally exhausted, quieter delivery, fading anger replaced by sadness No exaggerated screaming, no violence, no comedy, no text overlay, no watermark.
0:15

Seedance 2.0 Valence-Arousal + FACS Created an example scene to show how valence-arousal and FACS can be used together. Prompt: 15s, cinematic emotional confrontation. Two characters @[chracter sheet ref] stand face-to-face inside a small apartment kitchen late at night. The room is dimly lit by a single warm overhead light and soft city lights leaking through the window. The atmosphere feels emotionally exhausted, tense and painfully intimate, like an argument that has been building for years. Modern cinematic realism, subtle handheld camera movement, shallow depth of field, soft film grain, emotionally restrained acting, realistic silence between dialogue lines. Beat 1: The emotional state remains at high arousal and medium-low valence. Camera: slow handheld side shot circling both characters tight over-the-shoulder close-ups brief eye-level two-shot showing emotional distance FACS Character A: AU4 + AU7 + AU17 Dialogue A: /juː ˈnev.ɚ ˈriː.ə .li lʊkt æt miː/ /juː wɚ ɔːlˌweɪz ˈsʌmˌwɛɹ ɛls/ Voice: tight restrained voice, controlled anger, uneven breathing Character A tries to stay calm while suppressing years of resentment. Beat 2: The emotional state gradually shifts toward very low valence and medium-high arousal. Camera: slow push-in toward Character B extreme close-up on trembling eyes and mouth wide static shot showing silence after the argument peaks FACS Character B: AU1 + AU4 + AU15 + AU25 Dialogue B: /aɪ wəz ˈtɹaɪ.ɪŋ maɪ bɛst/ /aɪ dɪdnt noʊ haʊ tə fɪks ˈɛv.ɹiˌθɪŋ/ Voice: breaking voice, unstable breath support, emotionally collapsing delivery Beat 3: The emotional state remains at very low valence and medium-low arousal. Camera: locked wide shot with silence between them slow close-up on both characters avoiding eye contact subtle rack focus between faces FACS Character A: AU1 + AU15 + AU17 Dialogue A: /ˈmeɪ.bi wiː stɑpt ˈlɪs.ən.ɪŋ ə lɔŋ taɪm əˈgoʊ/ Voice: emotionally exhausted, quieter delivery, fading anger replaced by sadness No exaggerated screaming, no violence, no comedy, no text overlay, no watermark.

Kōda

23,696 views • 3 months ago

Robot Utility Models (RUMs) enable basic tasks – door opening, drawer opening, object reorientation, etc. – at ~90% accuracy without ANY finetuning (i.e. zero-shot) in unseen new environments. Fully open source!!! models, data, code & hw. We think this is super exciting, why?👇 1. Unlocks many practical home utility tasks that often involve these basic tasks as part of an action chain. “Go get me a fork” involves opening the kitchen door and then opening the cutlery drawer. 2. This works well **zero-shot in unseen and new** environments, which is practically a huge deal. Turn the robot on, and get going. 3. The recipe for building a new model is fairly generic, and we think with a bit more refinement this can be a general recipe to build many more Utility models. More details and access 👇
0:13

Robot Utility Models (RUMs) enable basic tasks – door opening, drawer opening, object reorientation, etc. – at ~90% accuracy without ANY finetuning (i.e. zero-shot) in unseen new environments. Fully open source!!! models, data, code & hw. We think this is super exciting, why?👇 1. Unlocks many practical home utility tasks that often involve these basic tasks as part of an action chain. “Go get me a fork” involves opening the kitchen door and then opening the cutlery drawer. 2. This works well **zero-shot in unseen and new** environments, which is practically a huge deal. Turn the robot on, and get going. 3. The recipe for building a new model is fairly generic, and we think with a bit more refinement this can be a general recipe to build many more Utility models. More details and access 👇

Mahi Shafiullah 🏠🤖

89,535 views • 1 year ago

For new followers: - I'm a long-time investor and builder in this space. - Founding Contributor of Realms.World ☁️. - Co-founder of Dojo. - Builder with the kings at Cartridge. - Starknet (Privacy Arc) class of '21. - Founder and Game Director of ETERNUM HAS MOVED. - Founder of Daydreams.Systems (x402, 8004 agents) My prime purpose for the past three years has been to build onchain infrastructure to enable the next generation of onchain experiences. This is done Starknet (Privacy Arc) as it is the superior VM for building complex applications—this will become clear soon enough. I work up and down the entire stack, from low-level indexing and contracts to GUI design. Nothing is out of scope. I have been pushing on agents for two years, mostly using existing frameworks like , until I came across @ElizaOS_ai in October. As I focused on building agents for ETERNUM HAS MOVED, it became clear that agents playing games require infinite paths to achieve goals. Thus, it's not scalable to hardcode functions—agents need to have total fluidity to take any action or call anything the game requires in any order. And ironically onchain infra is perfect for agent playgrounds because of its open nature. This exploration led me to create Daydreams.Systems (x402, 8004 agents), which focuses on the hardest problems of agents: long time-horizon goals using Hierarchical task networks (HTN). Daydreams agents don't require custom code—they work entirely based on 'sleeves'—which are just markdown files that explain how the agent can interact with the service (API docs, game guides, etc.) My thesis is simple. By focusing on the hardest problem (games), the design of the library will naturally lean towards an optimal structure for any problem an agent could face. We are early in this path and iterating with speed. If you are an onchain app developer or game builder—DM me, I want to know the architecture of your game so we can build sleeves together.
0:10

For new followers: - I'm a long-time investor and builder in this space. - Founding Contributor of Realms.World ☁️. - Co-founder of Dojo. - Builder with the kings at Cartridge. - Starknet (Privacy Arc) class of '21. - Founder and Game Director of ETERNUM HAS MOVED. - Founder of Daydreams.Systems (x402, 8004 agents) My prime purpose for the past three years has been to build onchain infrastructure to enable the next generation of onchain experiences. This is done Starknet (Privacy Arc) as it is the superior VM for building complex applications—this will become clear soon enough. I work up and down the entire stack, from low-level indexing and contracts to GUI design. Nothing is out of scope. I have been pushing on agents for two years, mostly using existing frameworks like , until I came across @ElizaOS_ai in October. As I focused on building agents for ETERNUM HAS MOVED, it became clear that agents playing games require infinite paths to achieve goals. Thus, it's not scalable to hardcode functions—agents need to have total fluidity to take any action or call anything the game requires in any order. And ironically onchain infra is perfect for agent playgrounds because of its open nature. This exploration led me to create Daydreams.Systems (x402, 8004 agents), which focuses on the hardest problems of agents: long time-horizon goals using Hierarchical task networks (HTN). Daydreams agents don't require custom code—they work entirely based on 'sleeves'—which are just markdown files that explain how the agent can interact with the service (API docs, game guides, etc.) My thesis is simple. By focusing on the hardest problem (games), the design of the library will naturally lean towards an optimal structure for any problem an agent could face. We are early in this path and iterating with speed. If you are an onchain app developer or game builder—DM me, I want to know the architecture of your game so we can build sleeves together.

loaf

43,320 views • 1 year ago

As always everyone is blind staring at the progress of LLMs for coding and chat But meanwhile the new SOTA video model Seedance 2.5 has been slowly rolling out and it's really quite exceptional It's made by ByteDance (TikTok) who of course have lots of training data With just a few reference pics, it can get quite close to how you look IRL and you can do quite professional video shots with just a prompt I'd say it's the first video model that's now at the level of image models with the level of character likeness, cracking that in image models also took about 3 years (2022-2025) Generating 15 seconds takes about 4 minutes I put it live now on Photo AI, you can use it under [ Make video ] from the sidebar with just a prompt and your model selected! So you don't need to take an AI photo first and then turn that into a video! Saves lots of time :D It's more expensive than but I kept the credits the same (30 for 1 video) It also works inside the new video editor and you can make changes in your video with [ Magic edit ] in both the main app and the video editor Also a message for my server guy Daniel Lockyer (it can do voice too and you can even submit a voice sample of yourself, but I didn't here)
0:15

As always everyone is blind staring at the progress of LLMs for coding and chat But meanwhile the new SOTA video model Seedance 2.5 has been slowly rolling out and it's really quite exceptional It's made by ByteDance (TikTok) who of course have lots of training data With just a few reference pics, it can get quite close to how you look IRL and you can do quite professional video shots with just a prompt I'd say it's the first video model that's now at the level of image models with the level of character likeness, cracking that in image models also took about 3 years (2022-2025) Generating 15 seconds takes about 4 minutes I put it live now on Photo AI, you can use it under [ Make video ] from the sidebar with just a prompt and your model selected! So you don't need to take an AI photo first and then turn that into a video! Saves lots of time :D It's more expensive than but I kept the credits the same (30 for 1 video) It also works inside the new video editor and you can make changes in your video with [ Magic edit ] in both the main app and the video editor Also a message for my server guy Daniel Lockyer (it can do voice too and you can even submit a voice sample of yourself, but I didn't here)

@levelsio

704,902 views • 5 days ago

This guy built an AI pipeline that generates hyperrealistic fashion models in 47 minutes and now dropshippers pay him $1,400 to clone the entire system. He got tired of watching e-com brands lose $8K per photoshoot when a single product angle changed so he built a 9-node workflow that generates 127 product videos from one Pinterest photo without hiring a single model. Here's the exact breakdown: → Claude writes a 34-parameter JSON brand DNA before any image is touched target psychographics, price anchor, vibe matrix, anti-inspiration blacklist → Pinterest becomes the model source library but you can't just download and animate → Kling 2.6 takes that static JPG and turns it into 5-second video but only after the prompt architecture is locked → Negative prompt node runs 41 exclusion terms: no plastic skin, no CGI glow, no symmetry artifacts, no doll face, no synthetic lighting → That one step kills the "AI look" that tanks engagement by 67% in the first 3 seconds → TikTok Studio uploads 19 videos in one batch with zero manual captioning because the brand voice was pre-programmed in step one → Atlas scrapes Amazon product links and auto-generates a Shopify store with hero images, pricing tiers, scarcity copy, and mobile-optimized checkout in 90 seconds → The store goes live before the first TikTok video finishes processing The key move 94% of people skip: you can't animate the photo before you inject the negative prompt. If you send a raw Pinterest image straight into image-to-video the face morphs into a wax figure. The fabric loses texture. The hands grow extra fingers. The whole thing screams "AI" and your CTR dies. His system runs the exclusion filter first so the model moves like she's shot on an iPhone 15 Pro in natural light. One brand hit 2.6M views on TikTok in 11 days with zero paid ads and converted at 3.7% because the videos looked like organic UGC not polished studio content. Brands now pay him $1,400 for the full pipeline setup + $340/month to keep the store synced with new product drops and seasonal video batches. The entire system runs on $23/month in API costs and one laptop. No photographer. No model agency. No product samples. Just a prompt template, a Pinterest account, and the discipline to filter out the AI artifacts before you render movement.
0:10

This guy built an AI pipeline that generates hyperrealistic fashion models in 47 minutes and now dropshippers pay him $1,400 to clone the entire system. He got tired of watching e-com brands lose $8K per photoshoot when a single product angle changed so he built a 9-node workflow that generates 127 product videos from one Pinterest photo without hiring a single model. Here's the exact breakdown: → Claude writes a 34-parameter JSON brand DNA before any image is touched target psychographics, price anchor, vibe matrix, anti-inspiration blacklist → Pinterest becomes the model source library but you can't just download and animate → Kling 2.6 takes that static JPG and turns it into 5-second video but only after the prompt architecture is locked → Negative prompt node runs 41 exclusion terms: no plastic skin, no CGI glow, no symmetry artifacts, no doll face, no synthetic lighting → That one step kills the "AI look" that tanks engagement by 67% in the first 3 seconds → TikTok Studio uploads 19 videos in one batch with zero manual captioning because the brand voice was pre-programmed in step one → Atlas scrapes Amazon product links and auto-generates a Shopify store with hero images, pricing tiers, scarcity copy, and mobile-optimized checkout in 90 seconds → The store goes live before the first TikTok video finishes processing The key move 94% of people skip: you can't animate the photo before you inject the negative prompt. If you send a raw Pinterest image straight into image-to-video the face morphs into a wax figure. The fabric loses texture. The hands grow extra fingers. The whole thing screams "AI" and your CTR dies. His system runs the exclusion filter first so the model moves like she's shot on an iPhone 15 Pro in natural light. One brand hit 2.6M views on TikTok in 11 days with zero paid ads and converted at 3.7% because the videos looked like organic UGC not polished studio content. Brands now pay him $1,400 for the full pipeline setup + $340/month to keep the store synced with new product drops and seasonal video batches. The entire system runs on $23/month in API costs and one laptop. No photographer. No model agency. No product samples. Just a prompt template, a Pinterest account, and the discipline to filter out the AI artifacts before you render movement.

Shade

536,735 views • 2 months ago

Claude Code + ChatGPT Images 2.0 is f*cking cracked 🤯 I rebuilt my static ad system inside Claude Code on the new ChatGPT Images 2.0 model. One brand name + one URL = 40 production-ready static ads. All inside Claude Code. Perfect for DTC brands and agencies who need high-volume ad creative without briefing a designer or spending hours in Canva. If you're finding winning ad concepts on Meta and manually recreating them one at a time — copying prompts, pasting product details, tweaking aspect ratios, downloading, organizing... This system eliminates the entire loop: → Give Claude a brand name and URL → It researches the brand's fonts, colors, packaging, and photography style → Builds a Brand DNA document from scratch → Fills in 40 proven ad templates (headline, us vs them, testimonial, UGC, review cards, stat callouts) with brand-specific details → Fires every prompt to ChatGPT Images 2.0 with your product photos as reference → Downloads finished ads into organized folders with an HTML gallery No manual prompt filling. No Canva templates. No copy-pasting between tools. What you get: → 40 ad formats filled with your exact brand colors, fonts, and copy → Text that actually renders correctly (the new model handles dense copy, logos, and multi-language callouts cleanly) → Product photos passed as reference so the model matches your real packaging → A reusable system — new brand, new folder, same pipeline Built 100% in Claude Code with ChatGPT Images 2.0. I put together a DIY playbook showing the exact architecture so you can build this yourself in Claude Code. Want it for free? > Like this post > Comment "CHAT" And I'll send it over (must be following so I can DM)
0:16

Claude Code + ChatGPT Images 2.0 is f*cking cracked 🤯 I rebuilt my static ad system inside Claude Code on the new ChatGPT Images 2.0 model. One brand name + one URL = 40 production-ready static ads. All inside Claude Code. Perfect for DTC brands and agencies who need high-volume ad creative without briefing a designer or spending hours in Canva. If you're finding winning ad concepts on Meta and manually recreating them one at a time — copying prompts, pasting product details, tweaking aspect ratios, downloading, organizing... This system eliminates the entire loop: → Give Claude a brand name and URL → It researches the brand's fonts, colors, packaging, and photography style → Builds a Brand DNA document from scratch → Fills in 40 proven ad templates (headline, us vs them, testimonial, UGC, review cards, stat callouts) with brand-specific details → Fires every prompt to ChatGPT Images 2.0 with your product photos as reference → Downloads finished ads into organized folders with an HTML gallery No manual prompt filling. No Canva templates. No copy-pasting between tools. What you get: → 40 ad formats filled with your exact brand colors, fonts, and copy → Text that actually renders correctly (the new model handles dense copy, logos, and multi-language callouts cleanly) → Product photos passed as reference so the model matches your real packaging → A reusable system — new brand, new folder, same pipeline Built 100% in Claude Code with ChatGPT Images 2.0. I put together a DIY playbook showing the exact architecture so you can build this yourself in Claude Code. Want it for free? > Like this post > Comment "CHAT" And I'll send it over (must be following so I can DM)

Mike Futia

189,409 views • 3 months ago

Claude Code + ChatGPT Images 2.0 is f*cking cracked 🤯 I rebuilt my static ad system inside Claude Code on the new ChatGPT Images 2.0 model. One brand name + one URL = 40 production-ready static ads. All inside Claude Code. Perfect for DTC brands and agencies who need high-volume ad creative without briefing a designer or spending hours in Canva. If you're finding winning ad concepts on Meta and manually recreating them one at a time — copying prompts, pasting product details, tweaking aspect ratios, downloading, organizing... This system eliminates the entire loop: → Give Claude a brand name and URL → It researches the brand's fonts, colors, packaging, and photography style → Builds a Brand DNA document from scratch → Fills in 40 proven ad templates (headline, us vs them, testimonial, UGC, review cards, stat callouts) with brand-specific details → Fires every prompt to ChatGPT Images 2.0 with your product photos as reference → Downloads finished ads into organized folders with an HTML gallery No manual prompt filling. No Canva templates. No copy-pasting between tools. What you get: → 40 ad formats filled with your exact brand colors, fonts, and copy → Text that actually renders correctly (the new model handles dense copy, logos, and multi-language callouts cleanly) → Product photos passed as reference so the model matches your real packaging → A reusable system — new brand, new folder, same pipeline Built 100% in Claude Code with ChatGPT Images 2.0. I put together a DIY playbook showing the exact architecture so you can build this yourself in Claude Code. Want it for free? > Like this post > Comment "CHAT" And I'll send it over (must be following @learnwithella so I can DM)
0:16

Claude Code + ChatGPT Images 2.0 is f*cking cracked 🤯 I rebuilt my static ad system inside Claude Code on the new ChatGPT Images 2.0 model. One brand name + one URL = 40 production-ready static ads. All inside Claude Code. Perfect for DTC brands and agencies who need high-volume ad creative without briefing a designer or spending hours in Canva. If you're finding winning ad concepts on Meta and manually recreating them one at a time — copying prompts, pasting product details, tweaking aspect ratios, downloading, organizing... This system eliminates the entire loop: → Give Claude a brand name and URL → It researches the brand's fonts, colors, packaging, and photography style → Builds a Brand DNA document from scratch → Fills in 40 proven ad templates (headline, us vs them, testimonial, UGC, review cards, stat callouts) with brand-specific details → Fires every prompt to ChatGPT Images 2.0 with your product photos as reference → Downloads finished ads into organized folders with an HTML gallery No manual prompt filling. No Canva templates. No copy-pasting between tools. What you get: → 40 ad formats filled with your exact brand colors, fonts, and copy → Text that actually renders correctly (the new model handles dense copy, logos, and multi-language callouts cleanly) → Product photos passed as reference so the model matches your real packaging → A reusable system — new brand, new folder, same pipeline Built 100% in Claude Code with ChatGPT Images 2.0. I put together a DIY playbook showing the exact architecture so you can build this yourself in Claude Code. Want it for free? > Like this post > Comment "CHAT" And I'll send it over (must be following @learnwithella so I can DM)

Ismail Khan

19,800 views • 3 months ago

We are in an insane run of open-weight drops. Every modality, open source is winning. This is what an open source AI summer ☀️ looks like: 🧠 LLMs & Reasoning → DeepSeek-V4-Flash-0731 (my king 👑): 304B MoE refresh, Terminal-Bench 2.1 jumps 61.8→82.7 over the preview, DeepSWE 7.3→54.4. Closes in on Opus-4.8 on Agents' Last Exam (25.2 vs 25.7). MIT. → Muse-Glimmer-30B, from Meta (they are back!!): their first open agentic model. ~29.6B dense + perception encoder, 131k+ context, built to run fully local, no cloud. Apache 2.0. → Liquid AI LFM2.5-2.6B: 2.69B params, 131k context, 220 tok/s on an M5 Max in under 2.5GB RAM. Competitive with models 4x larger on agentic tasks. → inclusionAI Ling-3.0-flash: 124B total, only 5.1B active, ~12% the size of their old 1T flagship Ring-2.6, matches it on key benchmarks. MIT. → inclusionAI Ling-3.0-tiny: 7.9B total, 1.3B active, 86-90 tok/s on an M4 Pro MacBook at ~8GB peak memory. MIT. → NVIDIA Nemotron-3.5-Lightning-30B-A3B: hybrid Mamba-2+MoE+Attention, up to 1M context, runs on a single H100 or DGX Spark, SWE-bench Verified 52.8. → deepgrove maple-preview: 20B-A1B ternary-weight reasoner, 218 tok/s on a Mac mini M4, 5.3GB checkpoint. MIT. → BigBang-v1 (endless-frontier): fine-tuned from Qwen3.6-35B-A3B via a self-evolving generator/critic synthetic-data loop. Lands aggregate performance between DeepSeek V4 Flash (284B) and V4 Pro (1.6T), at 35B. Apache 2.0. 🎬 Video → MiniMax-H3: 33B dense omni model, native stereo audio, up to 2K/15s. 3.6k+ likes already. → Minimax-H3-Turbo (lightx2v): Apache-2.0 turbo distillation of H3 for fast inference. → Lightricks LTX-2.5: image-to-video update, custom Gemma-4-12B text encoder, a markedly stronger distilled model. 🔊 Voice → NVIDIA NemotronLabs VoiceChat-11B: full-duplex speech-to-speech, ~450ms turn-taking, #2 on open VoiceBench, and the first open full-duplex model with live tool-calling mid-conversation. 🛡️ Safety → Mistral Shieldstral-1.0-3B: 3B multimodal guardrail that takes your safety policy as plain text instead of fixed categories. Beats LlamaGuard-4-12B and ShieldGemma-9B on HarmBench (99.4) and ToxicChat (84.1) at a fraction of the size. Apache 2.0.
0:14

We are in an insane run of open-weight drops. Every modality, open source is winning. This is what an open source AI summer ☀️ looks like: 🧠 LLMs & Reasoning → DeepSeek-V4-Flash-0731 (my king 👑): 304B MoE refresh, Terminal-Bench 2.1 jumps 61.8→82.7 over the preview, DeepSWE 7.3→54.4. Closes in on Opus-4.8 on Agents' Last Exam (25.2 vs 25.7). MIT. → Muse-Glimmer-30B, from Meta (they are back!!): their first open agentic model. ~29.6B dense + perception encoder, 131k+ context, built to run fully local, no cloud. Apache 2.0. → Liquid AI LFM2.5-2.6B: 2.69B params, 131k context, 220 tok/s on an M5 Max in under 2.5GB RAM. Competitive with models 4x larger on agentic tasks. → inclusionAI Ling-3.0-flash: 124B total, only 5.1B active, ~12% the size of their old 1T flagship Ring-2.6, matches it on key benchmarks. MIT. → inclusionAI Ling-3.0-tiny: 7.9B total, 1.3B active, 86-90 tok/s on an M4 Pro MacBook at ~8GB peak memory. MIT. → NVIDIA Nemotron-3.5-Lightning-30B-A3B: hybrid Mamba-2+MoE+Attention, up to 1M context, runs on a single H100 or DGX Spark, SWE-bench Verified 52.8. → deepgrove maple-preview: 20B-A1B ternary-weight reasoner, 218 tok/s on a Mac mini M4, 5.3GB checkpoint. MIT. → BigBang-v1 (endless-frontier): fine-tuned from Qwen3.6-35B-A3B via a self-evolving generator/critic synthetic-data loop. Lands aggregate performance between DeepSeek V4 Flash (284B) and V4 Pro (1.6T), at 35B. Apache 2.0. 🎬 Video → MiniMax-H3: 33B dense omni model, native stereo audio, up to 2K/15s. 3.6k+ likes already. → Minimax-H3-Turbo (lightx2v): Apache-2.0 turbo distillation of H3 for fast inference. → Lightricks LTX-2.5: image-to-video update, custom Gemma-4-12B text encoder, a markedly stronger distilled model. 🔊 Voice → NVIDIA NemotronLabs VoiceChat-11B: full-duplex speech-to-speech, ~450ms turn-taking, #2 on open VoiceBench, and the first open full-duplex model with live tool-calling mid-conversation. 🛡️ Safety → Mistral Shieldstral-1.0-3B: 3B multimodal guardrail that takes your safety policy as plain text instead of fixed categories. Beats LlamaGuard-4-12B and ShieldGemma-9B on HarmBench (99.4) and ToxicChat (84.1) at a fraction of the size. Apache 2.0.

Victor M

39,217 views • 1 day ago

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.
0:14

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

max fu

70,450 views • 28 days ago