Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

You asked for it, so here it is. Visualizing CPU cache speeds relative to RAM. Cache optimization is important too!

BenjDicken's profile picture

Ben Dicken

46,243 subscribers

1,474,595 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Some of the biggest news today is Cloudflare Worker's Cache. So I built this small catalog to experience it. It allows you to purge the cache for certain pages, or the entire catalog. It also shows you the exact caching headers used. See it in action or try it yourself:
0:16

Some of the biggest news today is Cloudflare Worker's Cache. So I built this small catalog to experience it. It allows you to purge the cache for certain pages, or the entire catalog. It also shows you the exact caching headers used. See it in action or try it yourself:

Jilles Soeters

17,920 Aufrufe • vor 2 Monaten

Submitted Easy Vertex Anim for #UE5.8 too. You can create pivot painter 2 effects and bake chaos cache to vat, 100% NO AI, check it out!
0:16

Submitted Easy Vertex Anim for #UE5.8 too. You can create pivot painter 2 effects and bake chaos cache to vat, 100% NO AI, check it out!

Inu Games

10,373 Aufrufe • vor 2 Monaten

I don't think anyone realizes yet how powerful Convex + Next.js 'use cache' is going to be You are able to get completely dynamic sites at the speed of static, with no flashes of incorrect data Also you will never have to think about what cache keys to set, it just works
0:16

I don't think anyone realizes yet how powerful Convex + Next.js 'use cache' is going to be You are able to get completely dynamic sites at the speed of static, with no flashes of incorrect data Also you will never have to think about what cache keys to set, it just works

Rhys

27,547 Aufrufe • vor 1 Jahr

WanGP's latest MiniMax H3 update is huge: - Way faster Generation - 2X faster Ref2VA - Lower-RAM & new accelerators - Long videos with smoother visual/audio transitions. - Huge speed gains with Sol-Attention, First Block Cache, etc. And Pinokio launcher is optimized for it 👇
0:14

WanGP's latest MiniMax H3 update is huge: - Way faster Generation - 2X faster Ref2VA - Lower-RAM & new accelerators - Long videos with smoother visual/audio transitions. - Huge speed gains with Sol-Attention, First Block Cache, etc. And Pinokio launcher is optimized for it 👇

cocktail peanut

58,539 Aufrufe • vor 1 Monat

A lot of people asked me to post FOOTage of Iuno's burst It features her, you guessed it, feet So here it is Enjoy!
0:16

Sensitive content

A lot of people asked me to post FOOTage of Iuno's burst It features her, you guessed it, feet So here it is Enjoy!

Geralt of Goonia

261,127 Aufrufe • vor 4 Monaten

Want the latest, flashiest skins? Lucky Cache is here for you every week! Every Friday to Sunday, Step into the Lucky Cache and challenge destiny! (level 80 needed) 🎁 One free draw every week A single draw could change everything and rewards can refresh 3 times a week — are you ready? Luck doesn't wait!
0:16

Want the latest, flashiest skins? Lucky Cache is here for you every week! Every Friday to Sunday, Step into the Lucky Cache and challenge destiny! (level 80 needed) 🎁 One free draw every week A single draw could change everything and rewards can refresh 3 times a week — are you ready? Luck doesn't wait!

Call of Odin's Chosen

121,198 Aufrufe • vor 8 Monaten

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.
0:14

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

max fu

70,797 Aufrufe • vor 1 Monat

🚨 Today I published my own Stream Deck plugin on @Elgato Marketplace . I call it SimpleStats. It's a very lightweight plugin for viewing CPU usage, Ping, Net Throughput, GPU temps, even what is using up all your RAM!!!!!!! The most important part for me: Everything is included.
0:19

🚨 Today I published my own Stream Deck plugin on @Elgato Marketplace . I call it SimpleStats. It's a very lightweight plugin for viewing CPU usage, Ping, Net Throughput, GPU temps, even what is using up all your RAM!!!!!!! The most important part for me: Everything is included.

Brett

114,702 Aufrufe • vor 4 Monaten

being able to pop into a level at an exact location is so important and so common, you NEED a button for it.
0:17

being able to pop into a level at an exact location is so important and so common, you NEED a button for it.

thomas "noio" vandenberg

20,938 Aufrufe • vor 5 Monaten

YOU ASKED FOR A LOVE SONG, SO HERE IS MY FIRST ATTEMPT. IF YOU DONT LIKE IT, LIE. ORIGINAL SONG COMING OUT TOMORROW! 🎉🥳 🪩
0:16

YOU ASKED FOR A LOVE SONG, SO HERE IS MY FIRST ATTEMPT. IF YOU DONT LIKE IT, LIE. ORIGINAL SONG COMING OUT TOMORROW! 🎉🥳 🪩

Michi Mochievee 💍

48,868 Aufrufe • vor 2 Monaten

The people asked… so here it is. 😂 Some treacherous conditions out here in Wichita.
0:18

The people asked… so here it is. 😂 Some treacherous conditions out here in Wichita.

Korn Ferry Tour

1,106,305 Aufrufe • vor 3 Jahren

shroud shared his actual thoughts on new version of Cache: "Yeah, I don't love it. I wasn't expecting to not love it, but it definitely needs some work for sure, I don't hate it either."
0:16

shroud shared his actual thoughts on new version of Cache: "Yeah, I don't love it. I wasn't expecting to not love it, but it definitely needs some work for sure, I don't hate it either."

Counter-Strike News

285,605 Aufrufe • vor 1 Jahr

Thank you so much for the 9,700+ likes! Here it is, green screen, so you can use it however you want! 😁
0:16

Thank you so much for the 9,700+ likes! Here it is, green screen, so you can use it however you want! 😁

Patdarux Dream

30,127 Aufrufe • vor 11 Monaten

My favorite editing model, FLUX.2 [klein] 9B, just got 2x faster: Meet FLUX.2 [klein] 9B-KV 😍💨 > Using KV-Cache Optimization to reduce computation & speed up inference by up to 2.5 times for multi-reference editing love how well it edits "around" the bullets
0:11

My favorite editing model, FLUX.2 [klein] 9B, just got 2x faster: Meet FLUX.2 [klein] 9B-KV 😍💨 > Using KV-Cache Optimization to reduce computation & speed up inference by up to 2.5 times for multi-reference editing love how well it edits "around" the bullets

Linoy Tsaban

30,134 Aufrufe • vor 5 Monaten

I asked #iubb freshman Prince-Alexander Moody why it was important to him to interact with all of you here.
0:18

I asked #iubb freshman Prince-Alexander Moody why it was important to him to interact with all of you here.

Mike Schumann @ The Daily Hoosier

19,084 Aufrufe • vor 2 Monaten

is it too thick for you?
0:14

Sensitive content

is it too thick for you?

graciebon

116,628 Aufrufe • vor 1 Monat

You guys asked for a tutorial, so here it is. Animated Apple-style websites FULLY with AI, perfect for web and UI work. Let's break it down step-by-step below. Prompts included.
0:13

You guys asked for a tutorial, so here it is. Animated Apple-style websites FULLY with AI, perfect for web and UI work. Let's break it down step-by-step below. Prompts included.

Viktor Oddy

82,355 Aufrufe • vor 7 Monaten

You asked for the Item Selection Wheel to make the item selection (and usage) less tedious, so here it is. This feature will roll out with the next update. 🧙‍♂️
0:14

You asked for the Item Selection Wheel to make the item selection (and usage) less tedious, so here it is. This feature will roll out with the next update. 🧙‍♂️

Wizordum 🧙‍♂️ | NOW on Consoles 🎮

13,584 Aufrufe • vor 1 Jahr

So… is it enough for you to come?
0:12

So… is it enough for you to come?

LazyLeaf

17,546 Aufrufe • vor 8 Monaten