Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto...

omarsar0's profile picture

elvis

324,856 subscribers

261,770 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von elvis
elvisvor 2 Jahren

source: github:

Profilbild von Adam Pippert
Adam Pippertvor 2 Jahren

This is HUGE. I always suspected that there would be a way to break up the architecture of a model to alleviate the GPU cost and availability bottleneck. Will play with this on LLaMa2, but highly anticipating Mistral-7B.

Profilbild von Hamid R. Darabi
Hamid R. Darabivor 2 Jahren

It's very interesting! 11.69x improvement over llama.cpp sounds a lot, given that it's already super efficient. Are you sure it's not 11.7%? Even that could be a good result.

Profilbild von Loki (cute/acc)
Loki (cute/acc)vor 2 Jahren

I almost tried this before I realized it's one more model exchange format 🥲 ".powerinfer.gguf" Can't all of you just use onnx and move on in life? 😭

Profilbild von catid (e/acc)
catid (e/acc)vor 2 Jahren

Nice I was working in this direction recently noting the same things. 12x speedup is very respectable compared to 8x being the best so far achieved by any one approach (e.g. pruning/quantization)!

Profilbild von Filippo Pedrazzini
Filippo Pedrazzinivor 2 Jahren

15 GiBs of weights? And this is supposed to run on Consumer Devices?! 🧐

Profilbild von Medium Boss - 70b_Float16.Q8.gguf
Medium Boss - 70b_Float16.Q8.ggufvor 2 Jahren

The week I got a new PC with a beefy CPU. Fucking nice.

Profilbild von Sahar Mor
Sahar Morvor 2 Jahren

Paper tl;dr

Profilbild von s3nh
s3nhvor 2 Jahren

Fastest bookmark 🤫🤫🤫

Profilbild von Ruairi
Ruairivor 2 Jahren

Noob question, is this just for the activation function or is it also cutting down the number of entries for GPU matrix multiplication as well?

Ähnliche Videos

Your $250 GPU can now run a 35B coding model at 50 tokens/sec. Qwen3.8-35B-A3B (3B active) hitting 50 tok/s on a 12GB RTX 3060 with 16GB RAM + 550 tok/s prompt processing. - 262K configured context This is the kind of result that makes upgrade cooling, not for a GPU. & MoE & hybrid CPU/GPU offload is quietly making cloud inference look sad for a lot of coding workflows.
1:06

Your $250 GPU can now run a 35B coding model at 50 tokens/sec. Qwen3.8-35B-A3B (3B active) hitting 50 tok/s on a 12GB RTX 3060 with 16GB RAM + 550 tok/s prompt processing. - 262K configured context This is the kind of result that makes upgrade cooling, not for a GPU. & MoE & hybrid CPU/GPU offload is quietly making cloud inference look sad for a lot of coding workflows.

Md Ismail Šojal 🕷️

136,816 Aufrufe • vor 14 Tagen

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers
0:20

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 Aufrufe • vor 2 Monaten

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.
6:33

NVIDIA'S $30K H100 JUST GOT OUT-INFERENCED BY A $250 RYZEN. 18 TOKENS/SEC ON A 35B MODEL, $0.03 IN ELECTRICITY, $47B GONE FROM NVDA BY FRIDAY download qwen3 35b-a3b -> load in lm studio -> disable gpu entirely -> run on cpu + ram -> hit 18 tokens/sec on a 6-core ryzen -> pay nothing per token. that loop is why moe models just made every consumer cpu a viable ai host and nvidia's inference monopoly is quietly on borrowed time. qwen3 35b-a3b + lm studio + ryzen 5 9600x + 32gb ddr5 + zero gpu - that's the stack. watch and save it, then run a 35b model on your laptop this weekend.

Antid

23,745 Aufrufe • vor 3 Monaten

🚨 Cerebras Inference is now 3x faster: Llama3.1-70B just broke 2,100 tokens/s - 16x faster than the fastest GPU solution - 8x faster than GPUs running Llama *3B* - It's like the perf of a new hardware generation in a single software release Available now at
0:26

🚨 Cerebras Inference is now 3x faster: Llama3.1-70B just broke 2,100 tokens/s - 16x faster than the fastest GPU solution - 8x faster than GPUs running Llama *3B* - It's like the perf of a new hardware generation in a single software release Available now at

Cerebras

236,597 Aufrufe • vor 2 Jahren

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰
0:32

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 Aufrufe • vor 2 Jahren

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.
1:27

Understanding GPUs will change your AI Engineering journey beyond agents and LLM. Rise of LLMs and deep learning introduced dedicated GPU hardware which serves your models for training and inference. You don't need to start with C++, CUDA, and Maths, but: > Components of a GPU and roles > How a token flows through hardware > Compute vs memory, the two limits > What FLOPS means in practice > Training vs inference hardware needs > What is CUDA Kernels > Calculating GPU memory from model size > Quantization to fit bigger models > Multi-GPU and splitting large models Hugging Face has the feature of calculating FLOps and recommending GPUs for your use case. You must explore. In terms of inference engineering, GPU understanding are much helpful your platform performance and handling multiple requests. Read this article for complete breakdown of GPUs for AI Engineers doesn't require any pre-requisite.

Karan🧋

33,453 Aufrufe • vor 1 Monat

A 12GB RTX 3060 is reportedly generating around 50 tokens/s with Qwen3.8-35B-A3B, a model with 35 billion parameters. The same setup is also reporting approximately 550 tokens/s in prompt processing. No 24GB GPU or expensive multi-GPU setup. Just an RTX 3060, 16GB of system RAM and some careful inference optimisation. The hardware and configuration The reported setup uses: GPU: RTX 3060 (12GB VRAM) System RAM: 16GB Model: Qwen3.8-35B-A3B Quantization: Q4_K_M Generation speed: ~50 tokens/s Prompt processing: ~550 tokens/s Configured context: 262K tokens Runtime: llama.cpp Optimisations: Flash Attention and Q8 KV cache The two speed figures measure different stages of inference Generation speed measures how quickly the model produces new tokens. Prompt processing measures how quickly it processes incoming text before generating a response Both are useful, but they shouldn’t be treated as interchangeable How does a 35B model run on 12GB VRAM? The explanation starts with the model’s architecture Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) design Although it contains approximately 35 billion parameters, only around 3 billion are activated for each token Instead of running every part of the model during every generation step, the routing system selects particular experts to perform the computation This reduces the amount of computation required per token compared with a dense model of similar total size However, there’s an important distinction: 3B active parameters does not mean the remaining 32B parameters disappear The full model weights still need to be stored and accessed. That’s where hybrid inference becomes important The CPU and GPU work together The setup uses llama.cpp to distribute the model across GPU and system memory According to the reported configuration, 26 MoE layers are offloaded to the CPU This allows system RAM to accommodate parts of the model that cannot fit entirely within the RTX 3060’s 12GB VRAM The GPU handles the portions assigned to it, while the CPU and system memory take care of the offloaded components Three optimisations help make this arrangement practical: 1. Q4_K_M quantization The model weights are stored in a 4-bit quantized format, significantly reducing their memory footprint compared with higher-precision versions The trade-off is that quantization can affect model quality, so coding performance still needs to be evaluated 2. Flash Attention Flash Attention improves how attention operations are computed by reducing unnecessary memory movement It can also help control memory requirements, particularly when processing longer sequences 3. Q8 KV cache The KV cache stores attention keys and values from previously processed tokens Using Q8 stores this information at 8-bit precision rather than a higher-precision format, reducing memory consumption as the context grows Together, these techniques allow the system to run a model that would otherwise be difficult to accommodate entirely in GPU memory What about the 262K context? The reported configuration supports a context window of up to 262K tokens But a configured context limit doesn’t mean the system can process that much text at the same speed or without memory pressure The KV cache grows as more context is processed. With only 16GB of system RAM and 12GB of VRAM, longer prompts could introduce substantial memory pressure and reduce performance. The actual usable context will depend on the runtime configuration, cache allocation and available memory The real test: coding agents Getting 50 tokens/s from a local model is encouraging, but generation speed alone doesn’t tell us how useful it is for software development A coding agent needs to: Inspect files and understand an existing codebase Identify dependencies and trace how components interact Modify multiple files Execute commands and tests Diagnose errors Recover from failed attempts Verify that the final changes actually work
1:06

A 12GB RTX 3060 is reportedly generating around 50 tokens/s with Qwen3.8-35B-A3B, a model with 35 billion parameters. The same setup is also reporting approximately 550 tokens/s in prompt processing. No 24GB GPU or expensive multi-GPU setup. Just an RTX 3060, 16GB of system RAM and some careful inference optimisation. The hardware and configuration The reported setup uses: GPU: RTX 3060 (12GB VRAM) System RAM: 16GB Model: Qwen3.8-35B-A3B Quantization: Q4_K_M Generation speed: ~50 tokens/s Prompt processing: ~550 tokens/s Configured context: 262K tokens Runtime: llama.cpp Optimisations: Flash Attention and Q8 KV cache The two speed figures measure different stages of inference Generation speed measures how quickly the model produces new tokens. Prompt processing measures how quickly it processes incoming text before generating a response Both are useful, but they shouldn’t be treated as interchangeable How does a 35B model run on 12GB VRAM? The explanation starts with the model’s architecture Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) design Although it contains approximately 35 billion parameters, only around 3 billion are activated for each token Instead of running every part of the model during every generation step, the routing system selects particular experts to perform the computation This reduces the amount of computation required per token compared with a dense model of similar total size However, there’s an important distinction: 3B active parameters does not mean the remaining 32B parameters disappear The full model weights still need to be stored and accessed. That’s where hybrid inference becomes important The CPU and GPU work together The setup uses llama.cpp to distribute the model across GPU and system memory According to the reported configuration, 26 MoE layers are offloaded to the CPU This allows system RAM to accommodate parts of the model that cannot fit entirely within the RTX 3060’s 12GB VRAM The GPU handles the portions assigned to it, while the CPU and system memory take care of the offloaded components Three optimisations help make this arrangement practical: 1. Q4_K_M quantization The model weights are stored in a 4-bit quantized format, significantly reducing their memory footprint compared with higher-precision versions The trade-off is that quantization can affect model quality, so coding performance still needs to be evaluated 2. Flash Attention Flash Attention improves how attention operations are computed by reducing unnecessary memory movement It can also help control memory requirements, particularly when processing longer sequences 3. Q8 KV cache The KV cache stores attention keys and values from previously processed tokens Using Q8 stores this information at 8-bit precision rather than a higher-precision format, reducing memory consumption as the context grows Together, these techniques allow the system to run a model that would otherwise be difficult to accommodate entirely in GPU memory What about the 262K context? The reported configuration supports a context window of up to 262K tokens But a configured context limit doesn’t mean the system can process that much text at the same speed or without memory pressure The KV cache grows as more context is processed. With only 16GB of system RAM and 12GB of VRAM, longer prompts could introduce substantial memory pressure and reduce performance. The actual usable context will depend on the runtime configuration, cache allocation and available memory The real test: coding agents Getting 50 tokens/s from a local model is encouraging, but generation speed alone doesn’t tell us how useful it is for software development A coding agent needs to: Inspect files and understand an existing codebase Identify dependencies and trace how components interact Modify multiple files Execute commands and tests Diagnose errors Recover from failed attempts Verify that the final changes actually work

FHILY👑

52,308 Aufrufe • vor 12 Tagen

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.
1:08

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,282 Aufrufe • vor 5 Monaten

Build a fully local AI coding agent on Ubuntu using Canonical Workshop and the Qwen3-Coder Inference Snap. In this step-by-step demo, you'll set up an isolated development environment, connect your coding agent to a locally hosted LLM, and run coding tasks without an internet connection. Follow along to recreate the setup on your own machine, using CPU or GPU inference: #Ubuntu #Linux #AI #Agents #Workshop #OpenSource
1:29

Build a fully local AI coding agent on Ubuntu using Canonical Workshop and the Qwen3-Coder Inference Snap. In this step-by-step demo, you'll set up an isolated development environment, connect your coding agent to a locally hosted LLM, and run coding tasks without an internet connection. Follow along to recreate the setup on your own machine, using CPU or GPU inference: #Ubuntu #Linux #AI #Agents #Workshop #OpenSource

Ubuntu

44,594 Aufrufe • vor 14 Tagen

ERRC: Entropy-Reinvested Residual Correction for Tensor-Parallel LLM Communication PyTorch Foundation Ambassador Abdulsalam Bande will present a poster on ERRC (Entropy-Reinvested Residual Correction for Tensor-Parallel LLM Communication) at PyTorch Conference North America 2026. ERRC focuses on optimizing GPU data exchange during LLM inference. By compressing inter-GPU communication and using the saved bandwidth to mitigate quantization errors, ERRC makes multi-GPU inference significantly more efficient. Register and join us to San Jose, October 20-21: #PyTorchCon Abdulsalam Bande
0:30

ERRC: Entropy-Reinvested Residual Correction for Tensor-Parallel LLM Communication PyTorch Foundation Ambassador Abdulsalam Bande will present a poster on ERRC (Entropy-Reinvested Residual Correction for Tensor-Parallel LLM Communication) at PyTorch Conference North America 2026. ERRC focuses on optimizing GPU data exchange during LLM inference. By compressing inter-GPU communication and using the saved bandwidth to mitigate quantization errors, ERRC makes multi-GPU inference significantly more efficient. Register and join us to San Jose, October 20-21: #PyTorchCon Abdulsalam Bande

PyTorch

17,043 Aufrufe • vor 1 Monat

Stress testing Metropolis 1998! - Population: 53,500 - CPU FPS: 80 - GPU FPS: 10-80 - RAM: 3,100 MB So: At the moment, the game is GPU bound. I know why and there's a lot of room for improvement. It will involve developing or customizing an existing GPU framework. There's also a lot of room to improve CPU/RAM, but right now it's in an okay place. #gamedev
0:27

Stress testing Metropolis 1998! - Population: 53,500 - CPU FPS: 80 - GPU FPS: 10-80 - RAM: 3,100 MB So: At the moment, the game is GPU bound. I know why and there's a lot of room for improvement. It will involve developing or customizing an existing GPU framework. There's also a lot of room to improve CPU/RAM, but right now it's in an okay place. #gamedev

Yesbox - Metropolis 1998

84,873 Aufrufe • vor 1 Jahr

Traditional GPU rentals often fluctuate due to supply bottlenecks and geopolitical factors. Aethir takes a different approach by providing bare-metal, on-demand GPU access, ensuring cost-efficient pricing and availability for AI training, inference, & agentic workflows.
0:48

Traditional GPU rentals often fluctuate due to supply bottlenecks and geopolitical factors. Aethir takes a different approach by providing bare-metal, on-demand GPU access, ensuring cost-efficient pricing and availability for AI training, inference, & agentic workflows.

Aethir

13,336 Aufrufe • vor 9 Monaten

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

346,057 Aufrufe • vor 2 Monaten

Running a 27B grade model on a 6 GB RTX 4050 laptop !! I kept Bonsai 2’s transformer layers on GPU, moved only the output head to CPU, and sped up its PTQ1 dot product with AVX2. Same answer but with increased throughput of 3.7 → 11.8 tok/s in this screen recorded demo. Built on Prism’s llama.cpp
1:11

Running a 27B grade model on a 6 GB RTX 4050 laptop !! I kept Bonsai 2’s transformer layers on GPU, moved only the output head to CPU, and sped up its PTQ1 dot product with AVX2. Same answer but with increased throughput of 3.7 → 11.8 tok/s in this screen recorded demo. Built on Prism’s llama.cpp

cheesecake

18,380 Aufrufe • vor 1 Monat

Jesse Pollak on the ecosystem forming around tokenized inference "There's this really cool tokenized inference market that Venice has innovated, where they've taken inference and turned it into a token called DIEM where you can buy it and you get $1 of inference per day" "That innovation of tokenizing inference has unlocked a ton of innovation around it, where people are using that tokenized inference to do things" "There's other projects that let people sell their tokenized inference at a discount so other people can get cheaper inference and the whole system works more efficiently" "I feel like that's the first time we're really seeing the intersection of AI plus markets plus crypto and it's thanks to Venice and it's happening on Base"
1:13

Jesse Pollak on the ecosystem forming around tokenized inference "There's this really cool tokenized inference market that Venice has innovated, where they've taken inference and turned it into a token called DIEM where you can buy it and you get $1 of inference per day" "That innovation of tokenizing inference has unlocked a ton of innovation around it, where people are using that tokenized inference to do things" "There's other projects that let people sell their tokenized inference at a discount so other people can get cheaper inference and the whole system works more efficiently" "I feel like that's the first time we're really seeing the intersection of AI plus markets plus crypto and it's thanks to Venice and it's happening on Base"

Market Bubble

26,240 Aufrufe • vor 3 Monaten

We built Curie, colibrì’s own model: 17B parameters, 33 tokens/s on a single CPU core. No GPU. Trained from scratch on a standard laptop. Written in C. Weights on SSD. An engine and a model designed together for the hardware you already own. Early alpha.This is just the beginning
0:33

We built Curie, colibrì’s own model: 17B parameters, 33 tokens/s on a single CPU core. No GPU. Trained from scratch on a standard laptop. Written in C. Weights on SSD. An engine and a model designed together for the hardware you already own. Early alpha.This is just the beginning

JustVugg

51,567 Aufrufe • vor 2 Monaten

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.
0:26

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

Avi Chawla

127,683 Aufrufe • vor 17 Tagen

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.
0:50

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 Aufrufe • vor 2 Monaten

Scale your AI inference, graphics, and analytics workloads with the GPU power to get you there. Amazon EC2 G7 instances are now generally available—accelerated by @NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs, first on AWS.
0:50

Scale your AI inference, graphics, and analytics workloads with the GPU power to get you there. Amazon EC2 G7 instances are now generally available—accelerated by @NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs, first on AWS.

Amazon Web Services

1,960,506 Aufrufe • vor 4 Monaten

Ray tracing sound on the GPU (Work in progress). Since the whole game is running on the GPU, I found it convenient to do the sound management and the ray tracing on the GPU as well. Each frame the CPU receives a sound queue from the GPU to process, and applies effects like reverb
0:40

Ray tracing sound on the GPU (Work in progress). Since the whole game is running on the GPU, I found it convenient to do the sound management and the ray tracing on the GPU as well. Each frame the CPU receives a sound queue from the GPU to process, and applies effects like reverb

xima

19,839 Aufrufe • vor 2 Jahren