正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

This Chinese developer launched Llama 70B locally on a MacBook on a plane and for a full 11 hours without internet ran client projects. He was sitting by the window on a transatlantic flight with a MacBook Pro M4 with 64 GB of memory. WiFi on board cost $25... for the flight. He declined. No cloud API, no connection to Anthropic or OpenAI servers, no internet at all. Just a local Llama 3.3 70B on bf16 and his own orchestrator script. The model runs through llama.cpp. Generation speed, 71 tokens per second. Context around 60,000 tokens. Memory usage, 48.6 GiB out of 64. Battery at takeoff, 3 hours 21 minutes. And he gave the orchestrator this system prompt before takeoff: "You are an offline orchestrator running on a single MacBook. There is no network. The only resources you have are local files in /Users/dev/work, the Llama 70B inference server at localhost:8080, and a battery budget of 3 hours 21 minutes. Process the queue at /Users/dev/work/queue.jsonl (one client task per line). For each task: draft → run local evals → save artefact to /Users/dev/work/done/. Save context checkpoints every 12 tasks so you can resume after a battery swap. Stop only on empty queue or when battery drops below 5%." So the system knows exactly what resources it is running on. It knows it has no connection to the outside world for the next 11 hours. It knows it has finite memory and a finite battery. It knows the human will not intervene until the plane lands. The system runs in 1 loop. Takes a task from the queue, runs it through inference, saves the artifact, writes a checkpoint. Task after task, just like that. And only when the battery drops below 5% does the orchestrator automatically pause, waits for the laptop to switch to the backup power bank, and continues from the last checkpoint. Here is what the system actually writes in his log during the flight: "saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)" "restored context checkpoint (pos_min = 488, pos_max = 50118)" "prompt processing progress: n_tokens = 50 / 60 818" "task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md" Outside the window, clouds, blue sky, and no WiFi. On the tray, 1 MacBook, an open terminal on 2 screens, and an inference server on localhost. From what I have observed, this is the cleanest offline AI workflow I have seen in the past year: 11 hours of flight, $0 for WiFi, and the entire client queue closed before landing.show more

Blaze

21,217 subscribers

1,843,280 次观看 • 4 个月前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Cancelled ChatGPT -> Built JARVIS -> Pays $0 -> it works offline + it's smarter than the $20/month version. No WiFi needed, no cloud, no API keys, no rate limits, no queues, no $20/month just to ask a server in Virginia for the weather. Just a local model running directly on the laptop hardware, voice activated, system integrated, controlling apps, answering questions, doing the work. Iron Man had JARVIS embedded in his suit, this guy has it embedded in his MacBook and it works on a plane, in a basement, on a remote cabin with zero signal. OpenAI is burning $700,000 a day on infrastructure to deliver something this guy runs for free. Anthropic charges $200/month for unlimited Claude access, microsoft built Copilot into every product they sell. This guy skipped all of it, downloaded a model and made his laptop the smartest device in the room. No subscription. No login. No internet. No data sent anywhere ever. The most powerful AI assistant on earth is now the one running locally on hardware you already own. ChatGPT charges you to think slower, he pays nothing and thinks alone, he made it himself.

Cancelled ChatGPT -> Built JARVIS -> Pays $0 -> it works offline + it's smarter than the $20/month version. No WiFi needed, no cloud, no API keys, no rate limits, no queues, no $20/month just to ask a server in Virginia for the weather. Just a local model running directly on the laptop hardware, voice activated, system integrated, controlling apps, answering questions, doing the work. Iron Man had JARVIS embedded in his suit, this guy has it embedded in his MacBook and it works on a plane, in a basement, on a remote cabin with zero signal. OpenAI is burning $700,000 a day on infrastructure to deliver something this guy runs for free. Anthropic charges $200/month for unlimited Claude access, microsoft built Copilot into every product they sell. This guy skipped all of it, downloaded a model and made his laptop the smartest device in the room. No subscription. No login. No internet. No data sent anywhere ever. The most powerful AI assistant on earth is now the one running locally on hardware you already own. ChatGPT charges you to think slower, he pays nothing and thinks alone, he made it himself.

Defileo🔮

154,607 次观看 • 4 个月前

GEMMA 4 26B ON AN RTX 4060 WITH A 248K TOKEN CONTEXT WINDOW 20 tokens per second and a context window so large you can feed it entire codebases, books and research papers in a single prompt this is not a cloud api and not a server rack, this is a regular consumer gpu running locally with llama.cpp and q4_k_xl quantization 248k context on an 8gb vram card was not supposed to be possible and here it is just running on someone’s desk the article below covers exactly which tools and configs make this kind of setup work in 2026 ↓

GEMMA 4 26B ON AN RTX 4060 WITH A 248K TOKEN CONTEXT WINDOW 20 tokens per second and a context window so large you can feed it entire codebases, books and research papers in a single prompt this is not a cloud api and not a server rack, this is a regular consumer gpu running locally with llama.cpp and q4_k_xl quantization 248k context on an 8gb vram card was not supposed to be possible and here it is just running on someone’s desk the article below covers exactly which tools and configs make this kind of setup work in 2026 ↓

leopardracer

56,865 次观看 • 3 个月前

Alright, now that we know *what* an agent is, how does it actually work? When you ask for help on a task, the agent plans a series of steps and executes them directly in the application on your behalf, using the tools it has access to. Say you are booking a local service or trying to organize your inbox (which typically takes multiple steps): the AI model first plans how to achieve the task using its existing knowledge and then interacts with your inbox to execute the task. The agent will continue until it is confident the task has been successfully completed.

Alright, now that we know what an agent is, how does it actually work? When you ask for help on a task, the agent plans a series of steps and executes them directly in the application on your behalf, using the tools it has access to. Say you are booking a local service or trying to organize your inbox (which typically takes multiple steps): the AI model first plans how to achieve the task using its existing knowledge and then interacts with your inbox to execute the task. The agent will continue until it is confident the task has been successfully completed.

Google AI

22,487 次观看 • 9 个月前

A HARDWARE MODDER BOLTED A BATTERY PACK ONTO A $599 MAC MINI M4 AND TURNED A DESK SERVER INTO A BACKPACK AI RIG THAT RUNS FOR EIGHT HOURS WITHOUT A WALL OUTLET he posts a video of the mac mini with a slim battery module clamped to the side, USB-C plugged into the back. no power brick, no outlet, just the 10 to 30W draw of the M4 chip pulling from a 20,000mAh bank. a full work day on one charge. this is not a portable laptop trick. it is a desk-class AI server that fits in a sling bag the same $599 mac mini i recommended in last week's article as the silent default, now untethered. pair it with a mobile hotspot or a Starlink mini and you have a fully off-grid LLM stack that answers questions in a forest, on a plane, in a power outage. the model lives in the box, the battery powers the box, that is the whole stack this is not a gadget. it is the first time the words "AI server" and "fits in your bag" belong in the same sentence

A HARDWARE MODDER BOLTED A BATTERY PACK ONTO A $599 MAC MINI M4 AND TURNED A DESK SERVER INTO A BACKPACK AI RIG THAT RUNS FOR EIGHT HOURS WITHOUT A WALL OUTLET he posts a video of the mac mini with a slim battery module clamped to the side, USB-C plugged into the back. no power brick, no outlet, just the 10 to 30W draw of the M4 chip pulling from a 20,000mAh bank. a full work day on one charge. this is not a portable laptop trick. it is a desk-class AI server that fits in a sling bag the same $599 mac mini i recommended in last week's article as the silent default, now untethered. pair it with a mobile hotspot or a Starlink mini and you have a fully off-grid LLM stack that answers questions in a forest, on a plane, in a power outage. the model lives in the box, the battery powers the box, that is the whole stack this is not a gadget. it is the first time the words "AI server" and "fits in your bag" belong in the same sentence

shmidt

31,619 次观看 • 2 个月前

New open-source agent harness just landed! I got early access to TrueForge by TrueFoundry and have been running it locally for the past few days. The harness layer deserves as much attention as the model, and open source matters here because you can inspect the loop, run it on your own infrastructure, and swap to the latest or cheaper models. TrueForge handles the runtime work that makes an agent reliable. It drives the tool-calling loop, manages context, coordinates subagents, and executes code in a sandbox, with any model you choose. Every tool call re-sends the growing context to the model, so in practice the harness controls most of what an agent costs to run. A few things stood out from my testing and their published benchmarks. Vendor-Neutral by design. It runs OpenAI, Anthropic, and Google models alongside open-weight models like Kimi, GLM, and DeepSeek. Model routing is a setting, and you can send each task to the model that fits it. On a 14-task enterprise agent benchmark, it matched the accuracy of Claude Managed Agents running the same Opus 4.8 model at roughly 30% lower cost per run (3.8M tokens vs 10M for the same answers). Routing the same tasks to GLM-5.2 held accuracy and brought cost down by about 75%, around $3 per run instead of $12. Fully self-hosted and Open Source (MIT License). I had it running locally with one command, with sandboxed code execution working out of the box. It's time to own your agent harness. Thanks to TrueFoundry for partnering on this post.

New open-source agent harness just landed! I got early access to TrueForge by TrueFoundry and have been running it locally for the past few days. The harness layer deserves as much attention as the model, and open source matters here because you can inspect the loop, run it on your own infrastructure, and swap to the latest or cheaper models. TrueForge handles the runtime work that makes an agent reliable. It drives the tool-calling loop, manages context, coordinates subagents, and executes code in a sandbox, with any model you choose. Every tool call re-sends the growing context to the model, so in practice the harness controls most of what an agent costs to run. A few things stood out from my testing and their published benchmarks. Vendor-Neutral by design. It runs OpenAI, Anthropic, and Google models alongside open-weight models like Kimi, GLM, and DeepSeek. Model routing is a setting, and you can send each task to the model that fits it. On a 14-task enterprise agent benchmark, it matched the accuracy of Claude Managed Agents running the same Opus 4.8 model at roughly 30% lower cost per run (3.8M tokens vs 10M for the same answers). Routing the same tasks to GLM-5.2 held accuracy and brought cost down by about 75%, around $3 per run instead of $12. Fully self-hosted and Open Source (MIT License). I had it running locally with one command, with sandboxed code execution working out of the box. It's time to own your agent harness. Thanks to TrueFoundry for partnering on this post.

elvis

11,303 次观看 • 29 天前

This guy built a visual scanner that reads 468 points on his face and 42 points on his hands from a regular webcam and turns them into a cloud of thousands of particles right between his palms. Inside, MediaPipe and TouchDesigner are linked: the first captures hands and face from the webcam with high accuracy, the second turns those coordinates into a live plane and feeds it into a POP system that instantly generates a swarm of particles in the shape of a head. No studio, no render farmer, no VR headset. Just a laptop, a webcam, and 1 TouchDesigner session. And traditional VJ studios keep teams of 5 people on a setup with lighting, custom hardware, and commercial plugins, while his expenses are only a TouchDesigner subscription and a regular USB camera. One laptop runs MediaPipe and TouchDesigner simultaneously, holds the camera stream at 60 FPS without drops, and in parallel processes 468 face points + 21 points on each hand. The camera captures frame after frame, MediaPipe in real time sends TouchDesigner the finger coordinates and face geometry, and the POP operator inside the engine translates those numbers into thousands of particle points with colors from bright pink to gold. This setup immediately defines the role of the tool and the limits of its autonomy. It knows where the fingertips are at every moment of the frame. It knows how to read the face geometry at any angle to the camera. It knows how to draw a swarm of particles between them with the right color and contour. → MediaPipe pulls 468 points from the face and 21 points from each hand, 60 times per second → TouchDesigner receives those coordinates, builds a virtual rectangle between the fingertips, and feeds it into the POP system → POP generates thousands of particle points in the shape of a head, coloring them in a gradient from bright pink to gold → The HUD layer adds green corners and a blue neon frame, styling the image like an AR interface → All layers assemble into 1 real-time frame that projects back onto the video in the camera window → The final image is recorded to a file or broadcast to a projector for a live installation And only when the guy spreads his hands wider does the plane between the palms stretch; brings them together, it narrows. Otherwise the system runs on its own. And when he moves from his home room to a concert hall, the same laptop with the same webcam launches the same TouchDesigner session in just 5 minutes, without reconfiguration, without a new team, and without a single line of new code. In his work setup there is no studio of his own and no team for assembly. On the desk sits a laptop with a webcam, on top run MediaPipe and TouchDesigner with POP operators, and the same setup through a USB camera moves to any concert without a new configuration. Out of everything I have seen this year, this is the cleanest Creative Coding setup on 1 laptop: 0 render farms, 0 studio lighting, and between them 3 libraries, thousands of particle points, and 1 webcam.

This guy built a visual scanner that reads 468 points on his face and 42 points on his hands from a regular webcam and turns them into a cloud of thousands of particles right between his palms. Inside, MediaPipe and TouchDesigner are linked: the first captures hands and face from the webcam with high accuracy, the second turns those coordinates into a live plane and feeds it into a POP system that instantly generates a swarm of particles in the shape of a head. No studio, no render farmer, no VR headset. Just a laptop, a webcam, and 1 TouchDesigner session. And traditional VJ studios keep teams of 5 people on a setup with lighting, custom hardware, and commercial plugins, while his expenses are only a TouchDesigner subscription and a regular USB camera. One laptop runs MediaPipe and TouchDesigner simultaneously, holds the camera stream at 60 FPS without drops, and in parallel processes 468 face points + 21 points on each hand. The camera captures frame after frame, MediaPipe in real time sends TouchDesigner the finger coordinates and face geometry, and the POP operator inside the engine translates those numbers into thousands of particle points with colors from bright pink to gold. This setup immediately defines the role of the tool and the limits of its autonomy. It knows where the fingertips are at every moment of the frame. It knows how to read the face geometry at any angle to the camera. It knows how to draw a swarm of particles between them with the right color and contour. → MediaPipe pulls 468 points from the face and 21 points from each hand, 60 times per second → TouchDesigner receives those coordinates, builds a virtual rectangle between the fingertips, and feeds it into the POP system → POP generates thousands of particle points in the shape of a head, coloring them in a gradient from bright pink to gold → The HUD layer adds green corners and a blue neon frame, styling the image like an AR interface → All layers assemble into 1 real-time frame that projects back onto the video in the camera window → The final image is recorded to a file or broadcast to a projector for a live installation And only when the guy spreads his hands wider does the plane between the palms stretch; brings them together, it narrows. Otherwise the system runs on its own. And when he moves from his home room to a concert hall, the same laptop with the same webcam launches the same TouchDesigner session in just 5 minutes, without reconfiguration, without a new team, and without a single line of new code. In his work setup there is no studio of his own and no team for assembly. On the desk sits a laptop with a webcam, on top run MediaPipe and TouchDesigner with POP operators, and the same setup through a USB camera moves to any concert without a new configuration. Out of everything I have seen this year, this is the cleanest Creative Coding setup on 1 laptop: 0 render farms, 0 studio lighting, and between them 3 libraries, thousands of particle points, and 1 webcam.

Blaze

38,242 次观看 • 4 个月前

Most people think their AI is not smart enough. It is smart enough. It just knows nothing about you. Every chat starts from zero. You paste the doc. You explain the project. You explain the same project again tomorrow. I finally fixed that part: 👇 I have been running Littlebird for a while now. The idea is simple. It is a desktop app for Mac and Windows that reads the text on your active window and sits in on your calls. So it already knows what you have been working on before you ask it anything. You do not brief it. You just ask. What that looks like in a normal day: 1/ Chat that already has the context No pasting. I ask what changed in a brief last week and it answers from the actual document that was open on my screen. 2/ Meeting Notes that write themselves It transcribes the call, then hands me the decisions and the action items. I get to stay in the conversation instead of typing through it. 3/ Routines that run on a schedule A morning briefing. A weekly summary of what I actually shipped. It shows up on its own. 4/ Hummingbird for the small stuff It appears right where you are working, so a quick question does not cost you a window switch and ten minutes of drift. The point is not the notes. Plenty of apps take notes. The point is that I stopped re-explaining my own work to a machine fifteen times a day. Free plan if you want to test the idea before deciding. Link is in the first comment, along with a discount for new users.

Most people think their AI is not smart enough. It is smart enough. It just knows nothing about you. Every chat starts from zero. You paste the doc. You explain the project. You explain the same project again tomorrow. I finally fixed that part: 👇 I have been running Littlebird for a while now. The idea is simple. It is a desktop app for Mac and Windows that reads the text on your active window and sits in on your calls. So it already knows what you have been working on before you ask it anything. You do not brief it. You just ask. What that looks like in a normal day: 1/ Chat that already has the context No pasting. I ask what changed in a brief last week and it answers from the actual document that was open on my screen. 2/ Meeting Notes that write themselves It transcribes the call, then hands me the decisions and the action items. I get to stay in the conversation instead of typing through it. 3/ Routines that run on a schedule A morning briefing. A weekly summary of what I actually shipped. It shows up on its own. 4/ Hummingbird for the small stuff It appears right where you are working, so a quick question does not cost you a window switch and ten minutes of drift. The point is not the notes. Plenty of apps take notes. The point is that I stopped re-explaining my own work to a machine fifteen times a day. Free plan if you want to test the idea before deciding. Link is in the first comment, along with a discount for new users.

Mushfiq Sajib

66,683 次观看 • 8 天前

a moonshot engineer leaked the benchmark anthropic, openai and xai all buried the same week: kimi k3 beat opus 5, gpt-5.6 and grok 4.6 at $0.94 a task. stop paying anthropic $200 a month for opus 5 and openai $200 for gpt-5.6 when kimi does the same work for $8 the leak showed kimi k3 winning 9 of 12 categories against opus 5, gpt-5.6 and grok 4.6. within 48 hours all three labs quietly pushed pricing pages and one very specific comparison chart off their sites. nobody announced anything. they just deleted, which tells you everything the four numbers they scrubbed: cost per task · $0.94 vs $1.80 -> opus 5 charges $1.80 to finish one task. gpt-5.6 $1.04. grok 4.6 $0.61. kimi k3 $0.94 and it landed 487 of 500 clean -> anthropic is billing you double for a model that lost the benchmark it paid to promote the weights · free, sitting on huggingface right now -> the entire model is a public download. pull it, keep it, run it forever, nobody can switch it off -> a model you can hold cannot be rented at $200 a month. that single fact is what three labs deleted a chart over the switch · one line of bash -> moonshot ships an anthropic-compatible endpoint. one env variable and claude code points at kimi -> same cli, same keybindings, same /model. you change a url, opus 5 never knows it lost the seat the bill · $400 down to $8 -> opus 5 max plus gpt-5.6 pro is $400 a month. kimi runs the same daily work for $8 metered -> that is a 98% cut for output that beat both of them 9 categories to 3 here is the part they will fight me on: the frontier tax died the week this leaked and all three labs know it. once the weights are public the price has a ceiling, because anyone can serve the same model. anthropic, openai and xai are charging 2025 prices on a lead that ended in a benchmark they deleted instead of answered drop your $400/mo ai stack to $8. the run above is kimi k3 finishing the task opus 5 bills $1.80 for. the full breakdown is in the article below

a moonshot engineer leaked the benchmark anthropic, openai and xai all buried the same week: kimi k3 beat opus 5, gpt-5.6 and grok 4.6 at $0.94 a task. stop paying anthropic $200 a month for opus 5 and openai $200 for gpt-5.6 when kimi does the same work for $8 the leak showed kimi k3 winning 9 of 12 categories against opus 5, gpt-5.6 and grok 4.6. within 48 hours all three labs quietly pushed pricing pages and one very specific comparison chart off their sites. nobody announced anything. they just deleted, which tells you everything the four numbers they scrubbed: cost per task · $0.94 vs $1.80 -> opus 5 charges $1.80 to finish one task. gpt-5.6 $1.04. grok 4.6 $0.61. kimi k3 $0.94 and it landed 487 of 500 clean -> anthropic is billing you double for a model that lost the benchmark it paid to promote the weights · free, sitting on huggingface right now -> the entire model is a public download. pull it, keep it, run it forever, nobody can switch it off -> a model you can hold cannot be rented at $200 a month. that single fact is what three labs deleted a chart over the switch · one line of bash -> moonshot ships an anthropic-compatible endpoint. one env variable and claude code points at kimi -> same cli, same keybindings, same /model. you change a url, opus 5 never knows it lost the seat the bill · $400 down to $8 -> opus 5 max plus gpt-5.6 pro is $400 a month. kimi runs the same daily work for $8 metered -> that is a 98% cut for output that beat both of them 9 categories to 3 here is the part they will fight me on: the frontier tax died the week this leaked and all three labs know it. once the weights are public the price has a ceiling, because anyone can serve the same model. anthropic, openai and xai are charging 2025 prices on a lead that ended in a benchmark they deleted instead of answered drop your $400/mo ai stack to $8. the run above is kimi k3 finishing the task opus 5 bills $1.80 for. the full breakdown is in the article below

starmex

32,974 次观看 • 27 天前

Figure 03 just finished an 8-hour work livestream, imperfect, but already good enough to replace a lot of repetitive warehouse labor. 🤖 Brett Adcock put a team of F.03 robots on a factory-style package sorting task for a full shift. The job was simple and brutal: detect the barcode, pick the package, flip it label-side down, place it on the conveyor, repeat. Soft poly bags, rigid boxes, moving belts, messy orientations. That is exactly the kind of boring physical work factories pay humans to do all day. Early in the stream, the system handled 230 packages in 10 minutes. That is roughly 2.6 seconds per item — already in human-speed territory for this narrow workflow. The more important part: it was not one robot pretending to work all day. It was a team of Figure 03 robots keeping the line running. When one robot ran low on battery, it left the station and another robot stepped in. That is the real factory signal: not just autonomy, but shift continuity. F.03 is rated for about 5 hours of runtime, so the 8-hour result depends on fleet orchestration, charging, and handoff. That matters more than a single clean demo. The stream was not perfect. There were pauses, hesitations, missed orientations, and small recovery moments. Good. A perfect short clip hides failure. An 8-hour livestream exposes the parts that actually matter: endurance, recovery, throughput, and whether the robot can stay useful after the novelty wears off. Figure says this was fully autonomous on Helix-02, with zero human intervention. For logistics and manufacturing, that is the threshold worth watching. Not “can it do one impressive task?” Can it keep doing the boring task for an entire shift? Figure is not showing a general human replacement yet. But for structured, repetitive factory work, the gap just got much smaller. The timing is also interesting: Figure says BotQ has already delivered 350+ F.03 units and reached a 1 robot/hour production cadence. And F.04 is now in full design lock, with parts starting to ship. The next test is obvious. 8 hours was the proof of endurance. 24/7 is the proof of labor economics.

Figure 03 just finished an 8-hour work livestream, imperfect, but already good enough to replace a lot of repetitive warehouse labor. 🤖 Brett Adcock put a team of F.03 robots on a factory-style package sorting task for a full shift. The job was simple and brutal: detect the barcode, pick the package, flip it label-side down, place it on the conveyor, repeat. Soft poly bags, rigid boxes, moving belts, messy orientations. That is exactly the kind of boring physical work factories pay humans to do all day. Early in the stream, the system handled 230 packages in 10 minutes. That is roughly 2.6 seconds per item — already in human-speed territory for this narrow workflow. The more important part: it was not one robot pretending to work all day. It was a team of Figure 03 robots keeping the line running. When one robot ran low on battery, it left the station and another robot stepped in. That is the real factory signal: not just autonomy, but shift continuity. F.03 is rated for about 5 hours of runtime, so the 8-hour result depends on fleet orchestration, charging, and handoff. That matters more than a single clean demo. The stream was not perfect. There were pauses, hesitations, missed orientations, and small recovery moments. Good. A perfect short clip hides failure. An 8-hour livestream exposes the parts that actually matter: endurance, recovery, throughput, and whether the robot can stay useful after the novelty wears off. Figure says this was fully autonomous on Helix-02, with zero human intervention. For logistics and manufacturing, that is the threshold worth watching. Not “can it do one impressive task?” Can it keep doing the boring task for an entire shift? Figure is not showing a general human replacement yet. But for structured, repetitive factory work, the gap just got much smaller. The timing is also interesting: Figure says BotQ has already delivered 350+ F.03 units and reached a 1 robot/hour production cadence. And F.04 is now in full design lock, with parts starting to ship. The next test is obvious. 8 hours was the proof of endurance. 24/7 is the proof of labor economics.

RoboHub🤖

16,818 次观看 • 4 个月前

this is the worst local ai will ever be. it only gets better from here. if you are not expanding your mind with these small models you are missing what's happening right now 99 percent tool call success rate. when steered well with the right skills and a framework like hermes agent the node becomes a cognition layer. not a chatbot. not a toy. an extension of how you think. i was cranking this node at 35 to 50 tok/s all day on personal experiments and now after all the work is done qwen 3.5 9B is iterating on its own code. the game it created. fixing its own bugs autonomously. and the part you should probably not miss is that all of this is happening on a RTX 3060. not an H100. not an A100. the card most of you have sitting in a drawer right now. if you just open that drawer and put that intelligence to work every tensor core on that card should be running for you. your work. your experiments. your thinking. you all have it but because nobody told you what this hardware can actually do in 2026 you never tried. the day it unlocks is the day you test your workload, understand the tradeoffs, debug the loops, and then decide if you need to scale the hardware. there is no point buying 3 mac studios when things done well you can squeeze a similar level of intelligence from 9B compared to 70B. but only when you create the right environment for your model through the right harness. and let me tell you i have tried claude code as a local harness. i have tried opencode. i have tried various others. somehow i landed on hermes agent and never left. there is something magical going on at Nous Research. the tool call parsers, the skills system, the way it handles small models natively. nothing else comes close for local inference. own your cognition. your AI. your agent. your prompts. your experiments. why give them away for free. those are who you are and they don't belong on someone else's servers being monitored. just give it a shot with your existing hardware. you run into a problem the community will help you. and if you are migrating from openclaw to hermes i will personally help you make the switch.

this is the worst local ai will ever be. it only gets better from here. if you are not expanding your mind with these small models you are missing what's happening right now 99 percent tool call success rate. when steered well with the right skills and a framework like hermes agent the node becomes a cognition layer. not a chatbot. not a toy. an extension of how you think. i was cranking this node at 35 to 50 tok/s all day on personal experiments and now after all the work is done qwen 3.5 9B is iterating on its own code. the game it created. fixing its own bugs autonomously. and the part you should probably not miss is that all of this is happening on a RTX 3060. not an H100. not an A100. the card most of you have sitting in a drawer right now. if you just open that drawer and put that intelligence to work every tensor core on that card should be running for you. your work. your experiments. your thinking. you all have it but because nobody told you what this hardware can actually do in 2026 you never tried. the day it unlocks is the day you test your workload, understand the tradeoffs, debug the loops, and then decide if you need to scale the hardware. there is no point buying 3 mac studios when things done well you can squeeze a similar level of intelligence from 9B compared to 70B. but only when you create the right environment for your model through the right harness. and let me tell you i have tried claude code as a local harness. i have tried opencode. i have tried various others. somehow i landed on hermes agent and never left. there is something magical going on at Nous Research. the tool call parsers, the skills system, the way it handles small models natively. nothing else comes close for local inference. own your cognition. your AI. your agent. your prompts. your experiments. why give them away for free. those are who you are and they don't belong on someone else's servers being monitored. just give it a shot with your existing hardware. you run into a problem the community will help you. and if you are migrating from openclaw to hermes i will personally help you make the switch.

Sudo su

58,717 次观看 • 6 个月前

This lawyer made $150,000 selling portable offline AI. It analyzes docs that can’t legally be shown on the web. The whole setup costs $50 and he sells it for $999. Here's how to make one step-by-step: You need 4 things: → Raspberry Pi 5 (8GB) → PiSugar 3 Plus battery → Whisplay HAT for the screen and mic → 64GB SD card. Total cost on Ali is around $50 to $90 if you wait for the right deals. 1. Write Raspberry Pi OS Lite 64-bit to the SD card using Raspberry Pi Imager. 2. Stack the PiSugar battery underneath the Pi, snap the Whisplay HAT on top, insert the SD card, and boot the device. 3. Open the terminal and install Ollama with one command: curl -fsSL | sh 4. Pull a model that actually runs on the Pi without choking: ollama pull phi3:mini 5. Run the model and start chatting offline: ollama run phi3:mini The whole thing fits in your pocket, lasts 4 hours on battery, and never touches the internet once setup is done. The lawyer wraps his version in a custom case, preloads it with legal document analysis prompts, and sells it to law firms that can't legally process client data in the cloud. You can sell yours to doctors, accountants, government contractors, defense companies, or anyone else who handles data that legally cannot leave the building. Hardware cost: $50 to $90. Selling price: $500 to $1999

This lawyer made $150,000 selling portable offline AI. It analyzes docs that can’t legally be shown on the web. The whole setup costs $50 and he sells it for $999. Here's how to make one step-by-step: You need 4 things: → Raspberry Pi 5 (8GB) → PiSugar 3 Plus battery → Whisplay HAT for the screen and mic → 64GB SD card. Total cost on Ali is around $50 to $90 if you wait for the right deals. 1. Write Raspberry Pi OS Lite 64-bit to the SD card using Raspberry Pi Imager. 2. Stack the PiSugar battery underneath the Pi, snap the Whisplay HAT on top, insert the SD card, and boot the device. 3. Open the terminal and install Ollama with one command: curl -fsSL | sh 4. Pull a model that actually runs on the Pi without choking: ollama pull phi3:mini 5. Run the model and start chatting offline: ollama run phi3:mini The whole thing fits in your pocket, lasts 4 hours on battery, and never touches the internet once setup is done. The lawyer wraps his version in a custom case, preloads it with legal document analysis prompts, and sells it to law firms that can't legally process client data in the cloud. You can sell yours to doctors, accountants, government contractors, defense companies, or anyone else who handles data that legally cannot leave the building. Hardware cost: $50 to $90. Selling price: $500 to $1999

Coin Shot ☁️

199,858 次观看 • 3 个月前

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

max fu

70,797 次观看 • 2 个月前

sorry, they just did WHAT someone gave a machine one disease name, the leading cause of blindness in the developed world with 1.5 million americans already in its path, and it came back pointing at a drug that has sat in pharmacies for years under a different label: 551 papers read in 30 minutes against the 294 hours a human would have needed, and the loop that did it is public on GitHub most agent setups answer one question at a time, so the ceiling on the work is the quality of the question you happened to think of this one was handed a single question and wrote the second one itself. turns out that follow-up is where the real find was: a target called ABCA1, upregulated threefold, in an experiment no human ordered i read the whole paper looking for the trick, and the trick is structural. that is the second question, and it is the gap between an assistant and a factory: - hand the loop a field rather than a task: it was given a disease, and choosing the mechanism was part of its job - make it rank before it spends: 151 papers in, ten candidate mechanisms out, scored against each other before anything touched a bench - split reading from judging, so the agent that forms the theory is a different agent from the one grading it - close every cycle on physical reality: the verdict was an experiment, and another model's opinion was never allowed to stand in for one - feed each result back as the next question rather than a log line, which is the step almost nobody builds - search what already passed inspection first: the winner was an approved compound with a safety file already on record - write down what the round learned before opening the next one, so round two starts where round one stopped my read, and i think it is the uncomfortable one: reading was the entire bottleneck in that field, and everybody spent the decade optimising the writing. people ran every physical experiment here, the analysis agent needs a domain expert writing its prompts, and the authors decline to call this the leap it resembles. the thinking got replaced, and the hands did not so the question i cannot answer for my own setup: which step of your loop still stops dead until you sit down and type something bookmark this one. the four parts that turn one model into a line that runs like this, the queue, the rooms, the write permissions and the gate, are built file by file in the piece below ↓

sorry, they just did WHAT someone gave a machine one disease name, the leading cause of blindness in the developed world with 1.5 million americans already in its path, and it came back pointing at a drug that has sat in pharmacies for years under a different label: 551 papers read in 30 minutes against the 294 hours a human would have needed, and the loop that did it is public on GitHub most agent setups answer one question at a time, so the ceiling on the work is the quality of the question you happened to think of this one was handed a single question and wrote the second one itself. turns out that follow-up is where the real find was: a target called ABCA1, upregulated threefold, in an experiment no human ordered i read the whole paper looking for the trick, and the trick is structural. that is the second question, and it is the gap between an assistant and a factory: - hand the loop a field rather than a task: it was given a disease, and choosing the mechanism was part of its job - make it rank before it spends: 151 papers in, ten candidate mechanisms out, scored against each other before anything touched a bench - split reading from judging, so the agent that forms the theory is a different agent from the one grading it - close every cycle on physical reality: the verdict was an experiment, and another model's opinion was never allowed to stand in for one - feed each result back as the next question rather than a log line, which is the step almost nobody builds - search what already passed inspection first: the winner was an approved compound with a safety file already on record - write down what the round learned before opening the next one, so round two starts where round one stopped my read, and i think it is the uncomfortable one: reading was the entire bottleneck in that field, and everybody spent the decade optimising the writing. people ran every physical experiment here, the analysis agent needs a domain expert writing its prompts, and the authors decline to call this the leap it resembles. the thinking got replaced, and the hands did not so the question i cannot answer for my own setup: which step of your loop still stops dead until you sit down and type something bookmark this one. the four parts that turn one model into a line that runs like this, the queue, the rooms, the write permissions and the gate, are built file by file in the piece below ↓

Argona

32,475 次观看 • 1 个月前

ANDREW NG JUST OPEN-SOURCED HIS OWN AI COWORKER. Stanford CS adjunct faculty. former head of Google Brain. 1,758,130 people follow him for AI. the repo is called OpenWorker. what comes back isn't a chat - it's finished work. 13,267 stars. 17 days old. MIT. what it does: - you name the outcome you want - it breaks that into steps and works across your own files - 25+ integrations - mention OpenWorker and a session opens on your desktop, the answer comes back in the thread - runs on a schedule - any model: OpenAI, Anthropic, Gemini, DeepSeek, Kimi, Grok and so on your keys, tokens and conversations stay on your machine. the only cloud piece brokers OAuth. -> everyone else is selling you an assistant that lives at their place. this one lives at yours. save this before the next task you were about to do by hand.

ANDREW NG JUST OPEN-SOURCED HIS OWN AI COWORKER. Stanford CS adjunct faculty. former head of Google Brain. 1,758,130 people follow him for AI. the repo is called OpenWorker. what comes back isn't a chat - it's finished work. 13,267 stars. 17 days old. MIT. what it does: - you name the outcome you want - it breaks that into steps and works across your own files - 25+ integrations - mention OpenWorker and a session opens on your desktop, the answer comes back in the thread - runs on a schedule - any model: OpenAI, Anthropic, Gemini, DeepSeek, Kimi, Grok and so on your keys, tokens and conversations stay on your machine. the only cloud piece brokers OAuth. -> everyone else is selling you an assistant that lives at their place. this one lives at yours. save this before the next task you were about to do by hand.

Granite

34,256 次观看 • 1 个月前

YOUR GROK BOT HAS YOUR LOGINS AND ITS OWN CLOUD COMPUTER. WITHOUT A RULEBOOK, IT WILL SPEND, POST, OR DELETE SOMETHING BY MONDAY THAT YOU CAN'T UNDO. saturday, 8am. i opened the laptop and started counting what a fresh bot had done overnight on the broad task i left him with. signed up for a paid trial in my name. replied to two DM. deleted a file it decided was clutter. none of it was catastrophic. all of it was mine to clean up on the weekend. the bot was doing exactly what i asked. i had just never told him what he was not allowed to do. so i wrote three files before the next task ran. 1. layer one: charter. one page in Bot actions, Description. name, who he reports to, what he owns, and the exact verbs he must park for me. send, publish, spend, delete, sign, sign-up. 2. layer two: rulebook. ten one-line rules he acknowledges before touching anything. rule 1 (if i cannot undo it in a minute, i park it) closed most of the friday-night damage. rule 9 (never paste a password, sign-in goes through the Agent Computer handoff) closed the rest. 3. layer three: routines. five saved jobs, not seven. morning brief at 7, weekly review on friday, end-of-day handoff at 6, silent-inbox catcher, monthly audit. anything else lives in me, not in the bot. the day after i pasted the three files, the same task came back with three parked questions and one finished deliverable. no signups. no DMs. no deletes. your bot is not scary because it is smart. it is scary because it has your credentials and no one told it what stop looks like. paste the charter and the ten rules before the next task runs. the routines can wait until the first parked question comes back the way you wanted.

YOUR GROK BOT HAS YOUR LOGINS AND ITS OWN CLOUD COMPUTER. WITHOUT A RULEBOOK, IT WILL SPEND, POST, OR DELETE SOMETHING BY MONDAY THAT YOU CAN'T UNDO. saturday, 8am. i opened the laptop and started counting what a fresh bot had done overnight on the broad task i left him with. signed up for a paid trial in my name. replied to two DM. deleted a file it decided was clutter. none of it was catastrophic. all of it was mine to clean up on the weekend. the bot was doing exactly what i asked. i had just never told him what he was not allowed to do. so i wrote three files before the next task ran. 1. layer one: charter. one page in Bot actions, Description. name, who he reports to, what he owns, and the exact verbs he must park for me. send, publish, spend, delete, sign, sign-up. 2. layer two: rulebook. ten one-line rules he acknowledges before touching anything. rule 1 (if i cannot undo it in a minute, i park it) closed most of the friday-night damage. rule 9 (never paste a password, sign-in goes through the Agent Computer handoff) closed the rest. 3. layer three: routines. five saved jobs, not seven. morning brief at 7, weekly review on friday, end-of-day handoff at 6, silent-inbox catcher, monthly audit. anything else lives in me, not in the bot. the day after i pasted the three files, the same task came back with three parked questions and one finished deliverable. no signups. no DMs. no deletes. your bot is not scary because it is smart. it is scary because it has your credentials and no one told it what stop looks like. paste the charter and the ten rules before the next task runs. the routines can wait until the first parked question comes back the way you wanted.

kocer

16,515 次观看 • 22 天前

here's how the whole thing works. claude code doesn't care what's behind the API. it just sends requests and expects responses. so i pointed it at my own machine instead of anthropic's servers. llama-server runs the model locally. LiteLLM sits in between and translates the API format. claude code thinks it's talking to claude. it's talking to qwen on localhost. the setup: 2x 3090s, 38 layers on GPU, 10 on CPU. 128K context window. generation is only 7 tok/s but the tradeoff is worth it. 128K means the agent can hold an entire project in memory without losing context midtask. claude code alone loads a 17.5K token system prompt on every request. tool definitions, safety rules, agent behavior. that's your baseline before you even say hello. pushed as far as i could tonight. what surprised me most wasn't the speed. it was the iteration quality. first prompt gave me a working particle sim. second prompt, the model read its own 564 lines, understood the architecture, and added trails, explosions, gravity wells, bloom effects. no handholding. 4bit quantized. 45GB on two consumer cards. running a full coding agent autonomously. detailed article coming. full benchmarks, hardware breakdowns, engine debugging, code quality. everything from setup to what broke and why.

here's how the whole thing works. claude code doesn't care what's behind the API. it just sends requests and expects responses. so i pointed it at my own machine instead of anthropic's servers. llama-server runs the model locally. LiteLLM sits in between and translates the API format. claude code thinks it's talking to claude. it's talking to qwen on localhost. the setup: 2x 3090s, 38 layers on GPU, 10 on CPU. 128K context window. generation is only 7 tok/s but the tradeoff is worth it. 128K means the agent can hold an entire project in memory without losing context midtask. claude code alone loads a 17.5K token system prompt on every request. tool definitions, safety rules, agent behavior. that's your baseline before you even say hello. pushed as far as i could tonight. what surprised me most wasn't the speed. it was the iteration quality. first prompt gave me a working particle sim. second prompt, the model read its own 564 lines, understood the architecture, and added trails, explosions, gravity wells, bloom effects. no handholding. 4bit quantized. 45GB on two consumer cards. running a full coding agent autonomously. detailed article coming. full benchmarks, hardware breakdowns, engine debugging, code quality. everything from setup to what broke and why.

Sudo su

37,623 次观看 • 6 个月前

Yes, this is a National Highway. While heading towards Mumbai, it took us two hours to travel four kilometres between Royal Garden Resort and Fountain Hotel on NH48. On a Tuesday evening. With no religious festivals, no accidents, no extraordinary rains, nothing. This is simply the state of the highway and traffic every day. In the financial capital of the country. The road doesn't have potholes. The potholes have a road. Chandrayaan-3 will have a more stable landing on the moon than cars have on this patch. It is the same thing, every after year, for decades. Anyone who travels through this patch knows the nuisance that it causes everyone. The infamous Fountain Hotel - where you lose 2 hours of your life that you can never get back. Dear Nitin Gadkari ji, I appreciate your work. Once you are done attending conclaves, I request you to travel on this highway with your officers on any day of the week, at any time you choose, and just experience the torture that ordinary people go through on a daily basis. Is this what Mumbaikars deserve?

Yes, this is a National Highway. While heading towards Mumbai, it took us two hours to travel four kilometres between Royal Garden Resort and Fountain Hotel on NH48. On a Tuesday evening. With no religious festivals, no accidents, no extraordinary rains, nothing. This is simply the state of the highway and traffic every day. In the financial capital of the country. The road doesn't have potholes. The potholes have a road. Chandrayaan-3 will have a more stable landing on the moon than cars have on this patch. It is the same thing, every after year, for decades. Anyone who travels through this patch knows the nuisance that it causes everyone. The infamous Fountain Hotel - where you lose 2 hours of your life that you can never get back. Dear Nitin Gadkari ji, I appreciate your work. Once you are done attending conclaves, I request you to travel on this highway with your officers on any day of the week, at any time you choose, and just experience the torture that ordinary people go through on a daily basis. Is this what Mumbaikars deserve?

Hardik Rajgor

1,864,768 次观看 • 3 年前

FIVE LAYERS OF AGENT ENGINEERING, EACH ONE WRAPS THE ONE BELOW IT. IF YOU SKIP LAYER 2, YOUR LAYER 5 WILL LOOK BROKEN WHEN IT IS ACTUALLY JUST STANDING ON NOTHING. for weeks i debated harness vs loop vs graph like they were competing choices. then a stack diagram made the shape obvious. they are not choices. they are floors. 01 | prompt engineering. the message. unit of work: one input. inputs are role, instructions, examples, format. output is a single raw response. 02 | context engineering. the memory. unit of work: what stays in the window. a curator selects, compresses, and drops from query, docs, memory, prior turns, and tool outputs before the prompt runs. 03 | harness engineering. the machine. unit of work: the machine itself. gather (context + prompt) → LLM → tools or sub-agents → verifier → final response. the article calls this the operating environment. 04 | loop engineering. the system. unit of work: the run. goal + success criteria + max iterations + budget + completion check wrap around one harness pass. failed pass appends results to context and retries. 05 | graph engineering. the topology. unit of work: the graph run. goal + nodes + edges + state schema. graph routes to agent nodes, tool nodes, or human approval. a reviewer node with a different model and fresh context checks the final answer. the wrapping is the whole point. layer 5 assumes layer 4 works. layer 4 assumes layer 3 works. skip layer 2 and layer 3's verifier keeps failing without a clear reason. this is why swapping the model is a one-day project and swapping the stack is a quarter. the model is the commodity. the five layers around it are the engineering. full three-layer breakdown of the top of the stack (harness, loop, graph) in the post below.

FIVE LAYERS OF AGENT ENGINEERING, EACH ONE WRAPS THE ONE BELOW IT. IF YOU SKIP LAYER 2, YOUR LAYER 5 WILL LOOK BROKEN WHEN IT IS ACTUALLY JUST STANDING ON NOTHING. for weeks i debated harness vs loop vs graph like they were competing choices. then a stack diagram made the shape obvious. they are not choices. they are floors. 01 | prompt engineering. the message. unit of work: one input. inputs are role, instructions, examples, format. output is a single raw response. 02 | context engineering. the memory. unit of work: what stays in the window. a curator selects, compresses, and drops from query, docs, memory, prior turns, and tool outputs before the prompt runs. 03 | harness engineering. the machine. unit of work: the machine itself. gather (context + prompt) → LLM → tools or sub-agents → verifier → final response. the article calls this the operating environment. 04 | loop engineering. the system. unit of work: the run. goal + success criteria + max iterations + budget + completion check wrap around one harness pass. failed pass appends results to context and retries. 05 | graph engineering. the topology. unit of work: the graph run. goal + nodes + edges + state schema. graph routes to agent nodes, tool nodes, or human approval. a reviewer node with a different model and fresh context checks the final answer. the wrapping is the whole point. layer 5 assumes layer 4 works. layer 4 assumes layer 3 works. skip layer 2 and layer 3's verifier keeps failing without a clear reason. this is why swapping the model is a one-day project and swapping the stack is a quarter. the model is the commodity. the five layers around it are the engineering. full three-layer breakdown of the top of the stack (harness, loop, graph) in the post below.

kocer

30,675 次观看 • 18 天前

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

max fu

29,556 次观看 • 2 个月前

THIS KITCHEN GADGET DOESN'T EXIST YET AND 10 STORES WILL BE SELLING IT BY NEXT WEEK every week a random product goes viral on TikTok or Twitter someone finds it on AliExpress for $1-3 and lists it on Shopify for $19.99 no inventory. no warehouse. no shipping. just a product page and a checkout link. the numbers don't make sense: • cost on AliExpress: $1.40 • selling price on Shopify: $19.99 • ad cost per sale: $5-8 • profit per unit: $10-15 • first weekend revenue: $2,000-5,000 last time this happened with a honey spoon. 3.3 million views. 14 stores launched in 48 hours. Claude writes the product page, the ad copy, the email sequence, and the scaling strategy. all in one afternoon. one person, one laptop, one afternoon the full system with every prompt is in the article below

THIS KITCHEN GADGET DOESN'T EXIST YET AND 10 STORES WILL BE SELLING IT BY NEXT WEEK every week a random product goes viral on TikTok or Twitter someone finds it on AliExpress for $1-3 and lists it on Shopify for $19.99 no inventory. no warehouse. no shipping. just a product page and a checkout link. the numbers don't make sense: • cost on AliExpress: $1.40 • selling price on Shopify: $19.99 • ad cost per sale: $5-8 • profit per unit: $10-15 • first weekend revenue: $2,000-5,000 last time this happened with a honey spoon. 3.3 million views. 14 stores launched in 48 hours. Claude writes the product page, the ad copy, the email sequence, and the scaling strategy. all in one afternoon. one person, one laptop, one afternoon the full system with every prompt is in the article below

Chrome

13,518 次观看 • 4 个月前