Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent... users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.show more

max fu

3,442 subscribers

70,238 Aufrufe • vor 25 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

Day 12/90 of Inference Engineering What is chunked prefill within vLLM? In continuation of yesterday's post on the high level architecture of vLLM, I want to dive deeper into vLLM core engine starting with the mechanics of chunked prefill. In this post, I will closely follow the original blog on the anatomy of vLLM. To start, let's define chunked prefill. It's a runtime inference optimization technique that splits a long input request so that it doesn’t monopolize the whole GPU. Keep in mind this is all within the context of vLLM. And since vLLM is an inference engine that's meant to serve a model to multiple concurrent users, having a GPU that’s fully monopolized on a single user's request means other users' requests would be in queue waiting to be processed. It isn’t too good to have the whole GPU occupied on a single request when the GPU is meant to be shared! So the key idea behind chunked prefill is to break the long request into smaller chunks, so that each chunk along with other users' requests gets processed and written into the KV cache together. Suppose we split up the long request into chunks and each chunk has 8 tokens. Now each memory block can hold 4 tokens. Therefore, 8 tokens can fit into 2 blocks of memory. After the first forward pass, 2 blocks are occupied, and after the second forward pass, 4 blocks of memory are occupied and so forth. Each forward pass handles a small chunk of the long request so that there's room in the same pass to keep serving other users' requests. Here's a small animation that I made today to fully visualize the idea behind chunked prefill when learning this topic~

max fu

29,197 Aufrufe • vor 24 Tagen

🎥 Video generation is hitting the memory wall. As videos get longer, the KV cache quietly explodes — and long-horizon consistency starts to break. We built Quant VideoGen: a training-free KV cache compression method for auto-regressive video diffusion. Instead of storing every KV in high precision, QVG exploits video’s spatiotemporal redundancy with semantic-aware smoothing + progressive residual quantization. 🚀 Up to 7× KV memory reduction ⚡ <4% overhead ✅ Strong long-video quality 🕹️ Deploy HYWorldPlay on your own RTX 5090 locally KV compression is becoming a core scaling primitive — not just for LLMs, but for video generation too. Paper: Code: (1/5)

🎥 Video generation is hitting the memory wall. As videos get longer, the KV cache quietly explodes — and long-horizon consistency starts to break. We built Quant VideoGen: a training-free KV cache compression method for auto-regressive video diffusion. Instead of storing every KV in high precision, QVG exploits video’s spatiotemporal redundancy with semantic-aware smoothing + progressive residual quantization. 🚀 Up to 7× KV memory reduction ⚡ <4% overhead ✅ Strong long-video quality 🕹️ Deploy HYWorldPlay on your own RTX 5090 locally KV compression is becoming a core scaling primitive — not just for LLMs, but for video generation too. Paper: Code: (1/5)

Haocheng Xi

65,008 Aufrufe • vor 3 Monaten

Gemma 4 26B A4B MoE - 500+ t/s decode - Single RTX 4090 (24 GB VRAM) - Llama.cpp concurrency 24 - q8 kv cache How many API users can you simultaneously host on a single RTX 4090 (24 GB VRAM) before it crashes? Yesterday, I proved you can host 14 active users using unquantized memory. Today, I used 8 bit KV Cache Quantization to hack the VRAM footprint. I successfully scaled to 24 concurrent users without a single dropped connection. A 71% server capacity boost for free. By adding the -ctk q8_0 -ctv q8_0 flags to llama.cpp, you compress the KV cache context memory from 16 bit to 8 bit. This unlocks massive concurrency limits on Gemma 4 26B (MoE) on a single 24GB consumer GPU. Here is the exact telemetry from pushing 8 bit quantization to its absolute physical edge: # TEST 1: The 24 User Concurrency Max Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 24 simultaneous requests (2,000 token prompt per user) Unquantized KV cache for this load requires 28GB+ VRAM (Instant OOM). Quantized to Q8, it allocated safely at 23.35 GB. The C++ engine crunched the entire batch in 28.5 seconds. Decode Speed: 21 t/s (Per User) | 500 t/s (Agg) # TEST 2: The 48 User Queue Overload What happens to a compressed cache during a traffic spike? Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 48 simultaneous requests (2k token prompt per user) Zero queue drops. The scheduler flushed and hot swapped the 8 bit memory flawlessly on the fly, completing all 48 users in 66.0 seconds (a perfect 2.3x queue scaling multiplier). Decode Speed: 18 t/s (Per User) | 430 t/s (Agg) # TEST 3: The 8 User RAG Slam Server Config: 8 slots (np 8) | 60,000 context per slot | 480,000 Total Context Client Load: 8 simultaneous requests (30k token prompt per user) It allocated 23.83 GB VRAM and chewed through ~240,000 prefill tokens in 46 seconds under massive memory pressure. Prefill Speed: 6,200 t/s (Agg) Decode Speed: 22 t/s (Per User) | 175 t/s (Agg) # The Engineering Alpha (The Quantization Tradeoff): You gain a massive 71% increase in server capacity, but what do you lose? Compute latency. Because the cache is stored in 8 bit, the GPU's cores have to dequantize the memory back to 16 bit on the fly during every single prefill step. In my unquantized tests yesterday, single slot prefill was hitting ~1,500+ t/s. Today, under the heavy 48-user Q8 load, prefill dropped as low as ~750 t/s. You trade a few seconds of initial prefill latency to essentially double your API hosting capacity. For production high volume SaaS, this is the ultimate unit economics cheat code. Here is the exact command to run a 24 user Q8 continuous batching server on your own single 4090, single 3090 or any 24gb vram rig: ./build/bin/llama-server -m gemma-4-26B-A4B-it.gguf -c 98304 -np 24 -b 2048 -ub 2048 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --port 8080 (Note: -c 98304 allocates exactly 4,096 tokens of context per user across 24 slots). Hugging Face links to the Unsloth Gemma 4 26B QAT quants along with performance graphs available in the replies. Would you trade 3 seconds of Time To First Token latency to double your active user capacity?

Gemma 4 26B A4B MoE - 500+ t/s decode - Single RTX 4090 (24 GB VRAM) - Llama.cpp concurrency 24 - q8 kv cache How many API users can you simultaneously host on a single RTX 4090 (24 GB VRAM) before it crashes? Yesterday, I proved you can host 14 active users using unquantized memory. Today, I used 8 bit KV Cache Quantization to hack the VRAM footprint. I successfully scaled to 24 concurrent users without a single dropped connection. A 71% server capacity boost for free. By adding the -ctk q8_0 -ctv q8_0 flags to llama.cpp, you compress the KV cache context memory from 16 bit to 8 bit. This unlocks massive concurrency limits on Gemma 4 26B (MoE) on a single 24GB consumer GPU. Here is the exact telemetry from pushing 8 bit quantization to its absolute physical edge: # TEST 1: The 24 User Concurrency Max Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 24 simultaneous requests (2,000 token prompt per user) Unquantized KV cache for this load requires 28GB+ VRAM (Instant OOM). Quantized to Q8, it allocated safely at 23.35 GB. The C++ engine crunched the entire batch in 28.5 seconds. Decode Speed: 21 t/s (Per User) | 500 t/s (Agg) # TEST 2: The 48 User Queue Overload What happens to a compressed cache during a traffic spike? Server Config: 24 slots (np 24) | 4,096 context per slot | 98,304 Total Context Client Load: 48 simultaneous requests (2k token prompt per user) Zero queue drops. The scheduler flushed and hot swapped the 8 bit memory flawlessly on the fly, completing all 48 users in 66.0 seconds (a perfect 2.3x queue scaling multiplier). Decode Speed: 18 t/s (Per User) | 430 t/s (Agg) # TEST 3: The 8 User RAG Slam Server Config: 8 slots (np 8) | 60,000 context per slot | 480,000 Total Context Client Load: 8 simultaneous requests (30k token prompt per user) It allocated 23.83 GB VRAM and chewed through ~240,000 prefill tokens in 46 seconds under massive memory pressure. Prefill Speed: 6,200 t/s (Agg) Decode Speed: 22 t/s (Per User) | 175 t/s (Agg) # The Engineering Alpha (The Quantization Tradeoff): You gain a massive 71% increase in server capacity, but what do you lose? Compute latency. Because the cache is stored in 8 bit, the GPU's cores have to dequantize the memory back to 16 bit on the fly during every single prefill step. In my unquantized tests yesterday, single slot prefill was hitting ~1,500+ t/s. Today, under the heavy 48-user Q8 load, prefill dropped as low as ~750 t/s. You trade a few seconds of initial prefill latency to essentially double your API hosting capacity. For production high volume SaaS, this is the ultimate unit economics cheat code. Here is the exact command to run a 24 user Q8 continuous batching server on your own single 4090, single 3090 or any 24gb vram rig: ./build/bin/llama-server -m gemma-4-26B-A4B-it.gguf -c 98304 -np 24 -b 2048 -ub 2048 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --port 8080 (Note: -c 98304 allocates exactly 4,096 tokens of context per user across 24 slots). Hugging Face links to the Unsloth Gemma 4 26B QAT quants along with performance graphs available in the replies. Would you trade 3 seconds of Time To First Token latency to double your active user capacity?

Alok

17,465 Aufrufe • vor 11 Tagen

Robots can now reconstruct 3D scenes in real time from a single RGB camera. [📍 Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like π³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: • Up to 30 FPS • 10 to 15x speedup • Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes • Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperial’s Dyson Robotics Lab and many others who contributed to this! 📍 Save projects page + paper for later: Video: ——- if it matters in AI or Robotics you'll read it here first:

Robots can now reconstruct 3D scenes in real time from a single RGB camera. [📍 Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like π³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: • Up to 30 FPS • 10 to 15x speedup • Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes • Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperial’s Dyson Robotics Lab and many others who contributed to this! 📍 Save projects page + paper for later: Video: ——- if it matters in AI or Robotics you'll read it here first:

Ilir Aliu

53,911 Aufrufe • vor 4 Monaten

I tested MTPLX v2 with QWEN 3.6 27B and compared it with oMLX without cache on M5 Max and DGX Spark on vllm using nvfp4 model version. More details in 🧵 I've reached 82.8 tps of max decoding speed! 🔥 Custom Metal Kernel design specifically for this model and for Apple Silicon is just perfect! This is the way forward! Great job Youssof Al Toukhi Look at the website here! 👇 Here a website with recap, built with GLM 5.2 running locally 💪 First chart and preview from the website.

I tested MTPLX v2 with QWEN 3.6 27B and compared it with oMLX without cache on M5 Max and DGX Spark on vllm using nvfp4 model version. More details in 🧵 I've reached 82.8 tps of max decoding speed! 🔥 Custom Metal Kernel design specifically for this model and for Apple Silicon is just perfect! This is the way forward! Great job Youssof Al Toukhi Look at the website here! 👇 Here a website with recap, built with GLM 5.2 running locally 💪 First chart and preview from the website.

Ivan Fioravanti ᯅ

15,846 Aufrufe • vor 1 Monat

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.

Avi Chawla

440,337 Aufrufe • vor 1 Monat

Researchers made KMeans 200x faster. And the new technique also beats approaches like cuML and FAISS. Flash-KMeans is an IO-aware implementation of exact KMeans that redesigns the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 33x speedup over cuML - 200x speedup over FAISS This speedup comes from how it moves through GPU memory. Standard KMeans runs in two steps, and both are bottlenecked by reads and writes to GPU memory: 1) The first step matches every point to its nearest centroid. Standard KMeans computes the full point-to-centroid distance matrix, writes it out to GPU memory, then reads it back to find each nearest centroid. That write-then-read round trip is the bottleneck. Flash-KMeans combines the distance calculation with the nearest-centroid step, so the result is computed on-chip and the full matrix is never written out. 2) The second step recomputes each centroid by averaging the points assigned to it. Standard KMeans has thousands of threads writing into the same centroid slots at once, so they stall waiting for their turn. Flash-KMeans sorts points by cluster first, turning scattered writes into sequential reductions that read and write memory in one efficient pass. Using these two optimizations at the million-scale, Flash-KMeans completes a standard KMeans iteration in a few milliseconds. The video below depicts this in action. Several reasons why this is important: KMeans has always been an offline primitive. Something you run once to preprocess data and move on. These speedups make the approach viable in several runtime-critical systems. ↳ Vector indices like FAISS use KMeans to build search indices. Faster KMeans means you can re-index dynamically as data changes. ↳ LLM quantization methods need KMeans to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Flash-KMeans makes it viable to run this inside the inference loop, not just in preprocessing. I have shared the paper in the replies. That said, memory is the real constraint Flash-KMeans solves, and the problem is not just limited to clustering. The vectors a RAG system stores after indexing create similar bottlenecks. I wrote a detailed walkthrough recently on cutting this vector memory by 32x with binary quantization, querying 36M+ vectors in a few milliseconds. Read it below.

Researchers made KMeans 200x faster. And the new technique also beats approaches like cuML and FAISS. Flash-KMeans is an IO-aware implementation of exact KMeans that redesigns the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 33x speedup over cuML - 200x speedup over FAISS This speedup comes from how it moves through GPU memory. Standard KMeans runs in two steps, and both are bottlenecked by reads and writes to GPU memory: 1) The first step matches every point to its nearest centroid. Standard KMeans computes the full point-to-centroid distance matrix, writes it out to GPU memory, then reads it back to find each nearest centroid. That write-then-read round trip is the bottleneck. Flash-KMeans combines the distance calculation with the nearest-centroid step, so the result is computed on-chip and the full matrix is never written out. 2) The second step recomputes each centroid by averaging the points assigned to it. Standard KMeans has thousands of threads writing into the same centroid slots at once, so they stall waiting for their turn. Flash-KMeans sorts points by cluster first, turning scattered writes into sequential reductions that read and write memory in one efficient pass. Using these two optimizations at the million-scale, Flash-KMeans completes a standard KMeans iteration in a few milliseconds. The video below depicts this in action. Several reasons why this is important: KMeans has always been an offline primitive. Something you run once to preprocess data and move on. These speedups make the approach viable in several runtime-critical systems. ↳ Vector indices like FAISS use KMeans to build search indices. Faster KMeans means you can re-index dynamically as data changes. ↳ LLM quantization methods need KMeans to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Flash-KMeans makes it viable to run this inside the inference loop, not just in preprocessing. I have shared the paper in the replies. That said, memory is the real constraint Flash-KMeans solves, and the problem is not just limited to clustering. The vectors a RAG system stores after indexing create similar bottlenecks. I wrote a detailed walkthrough recently on cutting this vector memory by 32x with binary quantization, querying 36M+ vectors in a few milliseconds. Read it below.

Avi Chawla

89,234 Aufrufe • vor 1 Monat

gemma-4-12B-agentic-fable5-composer2.5 V2 is out. the agentic upgrade to the model trained on Fable 5's reasoning. Running it now with TurboQuant llama.cpp on a single RTX 4060( 8 GB VRAM) at 30 tokens/second with full 25000 context and reasoning: # The benchmarks v2 is built for coding + agentic work. writing code, running commands, using tools, debugging, multi step technical tasks. The clearest signal is tau2 bench telecom, an agentic tool use benchmark whose diagnose → fix → verify loop mirrors real terminal/debugging work: tau2 bench telecom numbers: base Gemma 4 12B: ~15% this finetune: ~55%. (Self reported) thats a huge jump # TheTom/llama-cpp-turboquant flags: llama-server.exe -m gemma4-v2-Q4_K_M.gguf -ngl 99 -c 25000 --cache-type-k q8_0 --cache-type-v turbo3 --port 8080 Flag breakdown: -ngl 99 → full GPU offload -c 25000 → 25K context --cache-type-k q8_0 --cache-type-v turbo3 → mixed-precision KV cache — K at 8-bit, V at ~3-bit via TurboQuant (Walsh Hadamard rotated polar quant, Google's own KV-compression research). Not even merged into mainline llama.cpp. running it off a fork. No API. No cloud. Just llama.cpp. well, a fork of it and any 6gb+ GPU. If you tried yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF, check this out and share your experience with the models

gemma-4-12B-agentic-fable5-composer2.5 V2 is out. the agentic upgrade to the model trained on Fable 5's reasoning. Running it now with TurboQuant llama.cpp on a single RTX 4060( 8 GB VRAM) at 30 tokens/second with full 25000 context and reasoning: # The benchmarks v2 is built for coding + agentic work. writing code, running commands, using tools, debugging, multi step technical tasks. The clearest signal is tau2 bench telecom, an agentic tool use benchmark whose diagnose → fix → verify loop mirrors real terminal/debugging work: tau2 bench telecom numbers: base Gemma 4 12B: ~15% this finetune: ~55%. (Self reported) thats a huge jump # TheTom/llama-cpp-turboquant flags: llama-server.exe -m gemma4-v2-Q4_K_M.gguf -ngl 99 -c 25000 --cache-type-k q8_0 --cache-type-v turbo3 --port 8080 Flag breakdown: -ngl 99 → full GPU offload -c 25000 → 25K context --cache-type-k q8_0 --cache-type-v turbo3 → mixed-precision KV cache — K at 8-bit, V at ~3-bit via TurboQuant (Walsh Hadamard rotated polar quant, Google's own KV-compression research). Not even merged into mainline llama.cpp. running it off a fork. No API. No cloud. Just llama.cpp. well, a fork of it and any 6gb+ GPU. If you tried yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF, check this out and share your experience with the models

Alok

145,356 Aufrufe • vor 1 Monat

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies

Run Gemma 4 26B MoE on 8GB VRAM with 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the replies

Alok

292,770 Aufrufe • vor 2 Monaten

This Chinese developer launched Llama 70B locally on a MacBook on a plane and for a full 11 hours without internet ran client projects. He was sitting by the window on a transatlantic flight with a MacBook Pro M4 with 64 GB of memory. WiFi on board cost $25 for the flight. He declined. No cloud API, no connection to Anthropic or OpenAI servers, no internet at all. Just a local Llama 3.3 70B on bf16 and his own orchestrator script. The model runs through llama.cpp. Generation speed, 71 tokens per second. Context around 60,000 tokens. Memory usage, 48.6 GiB out of 64. Battery at takeoff, 3 hours 21 minutes. And he gave the orchestrator this system prompt before takeoff: "You are an offline orchestrator running on a single MacBook. There is no network. The only resources you have are local files in /Users/dev/work, the Llama 70B inference server at localhost:8080, and a battery budget of 3 hours 21 minutes. Process the queue at /Users/dev/work/queue.jsonl (one client task per line). For each task: draft → run local evals → save artefact to /Users/dev/work/done/. Save context checkpoints every 12 tasks so you can resume after a battery swap. Stop only on empty queue or when battery drops below 5%." So the system knows exactly what resources it is running on. It knows it has no connection to the outside world for the next 11 hours. It knows it has finite memory and a finite battery. It knows the human will not intervene until the plane lands. The system runs in 1 loop. Takes a task from the queue, runs it through inference, saves the artifact, writes a checkpoint. Task after task, just like that. And only when the battery drops below 5% does the orchestrator automatically pause, waits for the laptop to switch to the backup power bank, and continues from the last checkpoint. Here is what the system actually writes in his log during the flight: "saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)" "restored context checkpoint (pos_min = 488, pos_max = 50118)" "prompt processing progress: n_tokens = 50 / 60 818" "task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md" Outside the window, clouds, blue sky, and no WiFi. On the tray, 1 MacBook, an open terminal on 2 screens, and an inference server on localhost. From what I have observed, this is the cleanest offline AI workflow I have seen in the past year: 11 hours of flight, $0 for WiFi, and the entire client queue closed before landing.

This Chinese developer launched Llama 70B locally on a MacBook on a plane and for a full 11 hours without internet ran client projects. He was sitting by the window on a transatlantic flight with a MacBook Pro M4 with 64 GB of memory. WiFi on board cost $25 for the flight. He declined. No cloud API, no connection to Anthropic or OpenAI servers, no internet at all. Just a local Llama 3.3 70B on bf16 and his own orchestrator script. The model runs through llama.cpp. Generation speed, 71 tokens per second. Context around 60,000 tokens. Memory usage, 48.6 GiB out of 64. Battery at takeoff, 3 hours 21 minutes. And he gave the orchestrator this system prompt before takeoff: "You are an offline orchestrator running on a single MacBook. There is no network. The only resources you have are local files in /Users/dev/work, the Llama 70B inference server at localhost:8080, and a battery budget of 3 hours 21 minutes. Process the queue at /Users/dev/work/queue.jsonl (one client task per line). For each task: draft → run local evals → save artefact to /Users/dev/work/done/. Save context checkpoints every 12 tasks so you can resume after a battery swap. Stop only on empty queue or when battery drops below 5%." So the system knows exactly what resources it is running on. It knows it has no connection to the outside world for the next 11 hours. It knows it has finite memory and a finite battery. It knows the human will not intervene until the plane lands. The system runs in 1 loop. Takes a task from the queue, runs it through inference, saves the artifact, writes a checkpoint. Task after task, just like that. And only when the battery drops below 5% does the orchestrator automatically pause, waits for the laptop to switch to the backup power bank, and continues from the last checkpoint. Here is what the system actually writes in his log during the flight: "saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)" "restored context checkpoint (pos_min = 488, pos_max = 50118)" "prompt processing progress: n_tokens = 50 / 60 818" "task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md" Outside the window, clouds, blue sky, and no WiFi. On the tray, 1 MacBook, an open terminal on 2 screens, and an inference server on localhost. From what I have observed, this is the cleanest offline AI workflow I have seen in the past year: 11 hours of flight, $0 for WiFi, and the entire client queue closed before landing.

Blaze

1,840,585 Aufrufe • vor 3 Monaten

While working on a new video with solutions to the previous one, I found ChatGPT's new UI struggles even more with concurrent updates: entries lose state and stick around for too long (see video). If this was a LiveView app, we would be getting so much flak.😅 --- I believe part of the problem here is having separate mutate and fetch requests on every deletion. The first fetch is cancelled when the second one comes up, causing items to stick around for longer. Many said yesterday that you could do the mutation and fetch as a single request, but that leads to other problems, such zombie entries. For example, imagine you delete link1 and link2 within a brief period of time. There is no guarantee the deletion order in the database will match the order the client receives the response, so you may end up with this: 1. (client) request to delete link1 sent 2. (client) request to delete link2 sent 3. (server) deletes link1 and loads a new list (includes link2) 4. (server) deletes link2 and loads a new list (no link1 or link2) 5. (client) receives link2 response 6. (client) receives link1 response So if you choose to use the latest response (link1), you brought link2 back to life. If you say you will use the response from the last request, events 3-4 can be swapped, and now you bring link1 back to life. Another way to solve this is by basically not allowing concurrent requests at all but that can affect the user experience drastically in other ways. Next week I should publish a video explaining how LiveView tackles this. Stay tuned!

While working on a new video with solutions to the previous one, I found ChatGPT's new UI struggles even more with concurrent updates: entries lose state and stick around for too long (see video). If this was a LiveView app, we would be getting so much flak.😅 --- I believe part of the problem here is having separate mutate and fetch requests on every deletion. The first fetch is cancelled when the second one comes up, causing items to stick around for longer. Many said yesterday that you could do the mutation and fetch as a single request, but that leads to other problems, such zombie entries. For example, imagine you delete link1 and link2 within a brief period of time. There is no guarantee the deletion order in the database will match the order the client receives the response, so you may end up with this: 1. (client) request to delete link1 sent 2. (client) request to delete link2 sent 3. (server) deletes link1 and loads a new list (includes link2) 4. (server) deletes link2 and loads a new list (no link1 or link2) 5. (client) receives link2 response 6. (client) receives link1 response So if you choose to use the latest response (link1), you brought link2 back to life. If you say you will use the response from the last request, events 3-4 can be swapped, and now you bring link1 back to life. Another way to solve this is by basically not allowing concurrent requests at all but that can affect the user experience drastically in other ways. Next week I should publish a video explaining how LiveView tackles this. Stay tuned!

José Valim

22,976 Aufrufe • vor 1 Jahr

⬛️ We are currently accelerating the incubation of GPU Nodes into the infraX Network, with 12 H100’s currently available for operation. Despite the incubation of such immense GPU power, the infraX Platform is optimally designed to run on the least amount of computational power possible, meaning a lot of our available GPU nodes are currently sitting idle. Currently, we're utilising a single gigantic NVIDIA H100 server with 80GB of VRAM and over 220GB of RAM to run our Platform. To put that in perspective, it rivals the computational power of an adult human brain. This setup enables us to handle immense computational load and deliver high-quality AI content to our users, however we have much more in store. Our remaining, immense network of GPU units is currently being prepared for rental operations as we look to transform the corporate GPU lending sphere through our corporate GPU lending protocol. We already have many high tier Web3 Players ready for technical integration, with more approaching us daily. Through our V3 DApp we look to make these integrations publicly viewable with real time usage graphs integrated directly into our Platform, allowing for exceedingly unique viewing opportunities. $INFRA

⬛️ We are currently accelerating the incubation of GPU Nodes into the infraX Network, with 12 H100’s currently available for operation. Despite the incubation of such immense GPU power, the infraX Platform is optimally designed to run on the least amount of computational power possible, meaning a lot of our available GPU nodes are currently sitting idle. Currently, we're utilising a single gigantic NVIDIA H100 server with 80GB of VRAM and over 220GB of RAM to run our Platform. To put that in perspective, it rivals the computational power of an adult human brain. This setup enables us to handle immense computational load and deliver high-quality AI content to our users, however we have much more in store. Our remaining, immense network of GPU units is currently being prepared for rental operations as we look to transform the corporate GPU lending sphere through our corporate GPU lending protocol. We already have many high tier Web3 Players ready for technical integration, with more approaching us daily. Through our V3 DApp we look to make these integrations publicly viewable with real time usage graphs integrated directly into our Platform, allowing for exceedingly unique viewing opportunities. $INFRA

infraX | $INFRA

42,843 Aufrufe • vor 1 Jahr

Gemma 4 12B QAT (dense) achieves 1000+ tokens/sec prefill on 8GB VRAM with 120k context Gemma 4 12B QAT (dense), TurboQuant (Without MTP), RTX 4060 8GB VRAM: Prefill: 1000+ tok/s (42% increase) Decode: 25+ tok/s (25% increase) Context: 120k (150% increase) prefill was 700 tok/sec and decode 20 tok/sec with only 48k context without turbo quant (older test with mtp link in the comments) llama.cpp TurboQuant flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -c 120000 --cache-type-k q8_0 --cache-type-v turbo3 -ngl 99 --port 8080 tested with a 27k prompt, 120k context loaded. -ngl 99 here isn't a typo, full 12B dense, every layer on GPU, on an 8GB card. that's the part worth sitting with. The model has vision, audio input, thinking/reasoning and fits your 8GB card. TurboQuant's KV cache savings are what free up the room to do that at 120k context. side by side with yesterday: 26B A4B MoE got 320+ tok/s prefill. this dense 12B is clearing 1000+ rig: RTX 4060 8GB · i7H · 16GB RAM same two flags as yesterday, different model size: --cache-type-k q8_0 --cache-type-v turbo3 thanks to TheTom/llama-cpp-turboquant, TurboQuant fork of llama.cpp by Tom Turney (Tom Turney) to make this work. unsloth's model quant huggingface and the llama.cpp fork github link in the comments Do you prefer a dense or a MoE for your 8GB card?

Gemma 4 12B QAT (dense) achieves 1000+ tokens/sec prefill on 8GB VRAM with 120k context Gemma 4 12B QAT (dense), TurboQuant (Without MTP), RTX 4060 8GB VRAM: Prefill: 1000+ tok/s (42% increase) Decode: 25+ tok/s (25% increase) Context: 120k (150% increase) prefill was 700 tok/sec and decode 20 tok/sec with only 48k context without turbo quant (older test with mtp link in the comments) llama.cpp TurboQuant flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -c 120000 --cache-type-k q8_0 --cache-type-v turbo3 -ngl 99 --port 8080 tested with a 27k prompt, 120k context loaded. -ngl 99 here isn't a typo, full 12B dense, every layer on GPU, on an 8GB card. that's the part worth sitting with. The model has vision, audio input, thinking/reasoning and fits your 8GB card. TurboQuant's KV cache savings are what free up the room to do that at 120k context. side by side with yesterday: 26B A4B MoE got 320+ tok/s prefill. this dense 12B is clearing 1000+ rig: RTX 4060 8GB · i7H · 16GB RAM same two flags as yesterday, different model size: --cache-type-k q8_0 --cache-type-v turbo3 thanks to TheTom/llama-cpp-turboquant, TurboQuant fork of llama.cpp by Tom Turney (Tom Turney) to make this work. unsloth's model quant huggingface and the llama.cpp fork github link in the comments Do you prefer a dense or a MoE for your 8GB card?

Alok

34,500 Aufrufe • vor 1 Monat

50% cheaper Claude inference with just one line of code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.

50% cheaper Claude inference with just one line of code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.

Akshay 🚀

63,725 Aufrufe • vor 21 Tagen

Love for humanity is the core of love for life. It arises from the recognition of life’s value and beauty, from faith in humanity’s ability for greatness, and from the awareness that you are its child — and that it is dear and close to you. : The author places love for humanity at the heart of love for life itself. It springs from a clear recognition of life’s immense value and beauty, from an unwavering faith in humanity’s capacity for greatness, and from the intimate awareness that each person is humanity’s own child, bound to it by blood, spirit, and destiny. This love is not distant admiration. It is the quiet, fierce tenderness a child feels for the parent who gave it existence. Humanity is dear and close because it is the living source from which we draw breath, meaning, and possibility. To love life is to love the collective that carries it forward. To doubt humanity is to doubt life’s own worth. When we see ourselves as its children, gratitude replaces judgment. Hope replaces despair. Love for humanity becomes the deepest form of self-love, because we are not separate from it. It is the recognition that every stranger carries a piece of the same miracle we carry in ourselves. That recognition turns strangers into kin, and the world into home.

Love for humanity is the core of love for life. It arises from the recognition of life’s value and beauty, from faith in humanity’s ability for greatness, and from the awareness that you are its child — and that it is dear and close to you. : The author places love for humanity at the heart of love for life itself. It springs from a clear recognition of life’s immense value and beauty, from an unwavering faith in humanity’s capacity for greatness, and from the intimate awareness that each person is humanity’s own child, bound to it by blood, spirit, and destiny. This love is not distant admiration. It is the quiet, fierce tenderness a child feels for the parent who gave it existence. Humanity is dear and close because it is the living source from which we draw breath, meaning, and possibility. To love life is to love the collective that carries it forward. To doubt humanity is to doubt life’s own worth. When we see ourselves as its children, gratitude replaces judgment. Hope replaces despair. Love for humanity becomes the deepest form of self-love, because we are not separate from it. It is the recognition that every stranger carries a piece of the same miracle we carry in ourselves. That recognition turns strangers into kin, and the world into home.

Zafar Mirzo | Quotes

287,911 Aufrufe • vor 7 Monaten

The Sabotaging Practice of Over Supply and Sameness in the NFT Space. The current zeitgeist of the NFT space is that the same artists are doing the same kind of work five times a year, with project after project leaving a trail of disappointment and discontent among collectors and all of us watching in disbelief as huge resources are extracted from the space over work that feels like it could be left as an "artist study." I understand that you can do what you want with your money as collectors, but we are killing the whole space with this incestuous practice. No artist is that prolific to be able to do 5 collections of 100+ pieces each every year and actually deliver innovation and some kind of creative evolution. Of course, they can pretend play that the work has something new, but there is no precedent nor proof that that has ever happened in the speed that it happens in the NFT space. Again, people are free to through away their resources on whatever they want but with this way of doing things, we more and more are going to start seeing the consequences. Oh! There are consequences? Yes. Maybe unintended, but there are. Let's see. Let's start with the loss of belief in the NFT space as somewhere where emerging artists can come and find support for their experiments. Why even bother to bring experiments, innovation, and new ways to think of art on the blockchain if the same people have all the collectors hypnotized with their magical flutes? Why even try to come to a space where taking risks and challenging the status quo (the mission of art!!!) is overlooked? This makes the NFT space a social club and not a space for art. I guess it is fine, but IMO it is a recipe for disaster. New collectors stay away because the art will slowly but surely become stale and un-challenging. Why even bother to come and see what is happening here if you can't, as a collector, see new weird and up-and-coming artists? The amount of noise emitted by the same artists doing the same art over and over, drowns out any new voices. Again. A recipe for disaster. The NFT space is becoming a space of disappointment and doubt. We think that collections going to zero one after the other, over and over, is not damaging? I feel we are kidding ourselves. Disappointment piles up, and again, the people who will hurt are the emerging artists, the new blood, the ones who are willing to risk the most and, in return, put fire in this cold space of sameness. I love this space—don't get me wrong—it has changed my life, and I believe it has a ton of potential, but things need to change for it to become a beacon of light in art. But we need to support new voices. We need to support new ideas. The challenge is huge. I hope to contribute all I can to this change. I hope more and more see how exciting it is to go out and try to discover what else is out there and move this space forward. But again, I understand the leaps of faith needed, but if there is a space that is based on that, it's the NFT space...so there is hope. We will see. 📺by Boldtron

The Sabotaging Practice of Over Supply and Sameness in the NFT Space. The current zeitgeist of the NFT space is that the same artists are doing the same kind of work five times a year, with project after project leaving a trail of disappointment and discontent among collectors and all of us watching in disbelief as huge resources are extracted from the space over work that feels like it could be left as an "artist study." I understand that you can do what you want with your money as collectors, but we are killing the whole space with this incestuous practice. No artist is that prolific to be able to do 5 collections of 100+ pieces each every year and actually deliver innovation and some kind of creative evolution. Of course, they can pretend play that the work has something new, but there is no precedent nor proof that that has ever happened in the speed that it happens in the NFT space. Again, people are free to through away their resources on whatever they want but with this way of doing things, we more and more are going to start seeing the consequences. Oh! There are consequences? Yes. Maybe unintended, but there are. Let's see. Let's start with the loss of belief in the NFT space as somewhere where emerging artists can come and find support for their experiments. Why even bother to bring experiments, innovation, and new ways to think of art on the blockchain if the same people have all the collectors hypnotized with their magical flutes? Why even try to come to a space where taking risks and challenging the status quo (the mission of art!!!) is overlooked? This makes the NFT space a social club and not a space for art. I guess it is fine, but IMO it is a recipe for disaster. New collectors stay away because the art will slowly but surely become stale and un-challenging. Why even bother to come and see what is happening here if you can't, as a collector, see new weird and up-and-coming artists? The amount of noise emitted by the same artists doing the same art over and over, drowns out any new voices. Again. A recipe for disaster. The NFT space is becoming a space of disappointment and doubt. We think that collections going to zero one after the other, over and over, is not damaging? I feel we are kidding ourselves. Disappointment piles up, and again, the people who will hurt are the emerging artists, the new blood, the ones who are willing to risk the most and, in return, put fire in this cold space of sameness. I love this space—don't get me wrong—it has changed my life, and I believe it has a ton of potential, but things need to change for it to become a beacon of light in art. But we need to support new voices. We need to support new ideas. The challenge is huge. I hope to contribute all I can to this change. I hope more and more see how exciting it is to go out and try to discover what else is out there and move this space forward. But again, I understand the leaps of faith needed, but if there is a space that is based on that, it's the NFT space...so there is hope. We will see. 📺by Boldtron

alejandro cartagena

98,261 Aufrufe • vor 2 Jahren

Alright, now that we know *what* an agent is, how does it actually work? When you ask for help on a task, the agent plans a series of steps and executes them directly in the application on your behalf, using the tools it has access to. Say you are booking a local service or trying to organize your inbox (which typically takes multiple steps): the AI model first plans how to achieve the task using its existing knowledge and then interacts with your inbox to execute the task. The agent will continue until it is confident the task has been successfully completed.

Alright, now that we know what an agent is, how does it actually work? When you ask for help on a task, the agent plans a series of steps and executes them directly in the application on your behalf, using the tools it has access to. Say you are booking a local service or trying to organize your inbox (which typically takes multiple steps): the AI model first plans how to achieve the task using its existing knowledge and then interacts with your inbox to execute the task. The agent will continue until it is confident the task has been successfully completed.

Google AI

22,487 Aufrufe • vor 8 Monaten

This is the Rabbit R1, a new AI handheld that wants to change the way users interact with apps on their smartphones - It uses a Large Action Model to take commands from the user and perform the instructions within apps on their behalf - Has a 2.88-inch touchscreen - It is powered by a 2.3GHz MediaTek processor with 4GB RAM and 128GB storage - Has an "all day" battery life and a rotatable camera "eye." The R1 is available for preorder ($199), with shipments expected to start from March. #Rabbit #RabbitR1

This is the Rabbit R1, a new AI handheld that wants to change the way users interact with apps on their smartphones - It uses a Large Action Model to take commands from the user and perform the instructions within apps on their behalf - Has a 2.88-inch touchscreen - It is powered by a 2.3GHz MediaTek processor with 4GB RAM and 128GB storage - Has an "all day" battery life and a rotatable camera "eye." The R1 is available for preorder ($199), with shipments expected to start from March. #Rabbit #RabbitR1

Mukul Sharma

22,415 Aufrufe • vor 2 Jahren