Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

GPU by hand ✍️ I drew 42 frames to show how a GPU speeds up an array operation of 8 elements in parallel over 4 threads in 2 clock cycles. Below is an overview: CPU • It has one core. • Its global memory has 120 locations (0-119). •... To use the GPU, it needs to copy data from the global memory to the GPU. • After GPU is done, it will copy the results back. GPU • It has four cores to run four threads (0-3). • It has a register file of 28 locations (0-27) • This register file has four banks (0-3). • All threads share the same register file. • But they must read/write using the four banks. • Each bank allows 2 reads (Read 0, Read 1) and 1 write in a single clock cycle.show more

Tom Yeh

51,088 subscribers

101,788 views • 1 year ago •via X (Twitter)

Science & Technology Education Arts

Anya Rossi• Live Now

Private livecam show

9 Comments

hengcherkeng1 year ago

this kind of illustration and movie is very important for learning cuda programming and optimization. And I think chatgpt, etc are very useful for creating such movie. Information processing and presentation is one of the practical use of current AI tool.

Mobile Scanner1 year ago

Scan any documents, convert images into text, PDF files, etc. 👍

Vladimir Tchuiev1 year ago

That's really cool 😊 I'll want to learn this in detail, saved, 🙏

KuromiAI_CS1 year ago

nice

NGR1 year ago

But we have multi core CPUs too? How is that different?

Tom Yeh1 year ago

CPU has 4-8 cores, each is like a Swiss knife with hundreds of tools. GPU can have thousands of cores, each has just a handful of tools.

Mike 🇺🇦🇮🇪1 year ago

AVX512?

Anurag Pant1 year ago

👍👍👍

Prathamesh Gokulkar1 year ago

Very insightful

Related Videos

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

The creator of High Bandwidth Memory (HBM) put a number on the AI build that should stop every infra investor cold. A cluster of a million GPUs runs at roughly 10-20% utilization (Save this). Kim Jung-ho spent thirty years building what feeds the GPU, and his claim is that the GPU is barely working. Here is what is actually happening. Every time a model generates output, the data has to be read out of memory, computed, and written back. The read and the write swallow almost the entire cycle. While that data moves, the GPU does nothing. It sits there, fully powered, fully paid for, waiting. By Kim's estimate the memory is doing only about 30 percent of the work it needs to do. The processor idles the rest. So a million installed GPUs run at 10 to 20 percent. You are not compute constrained. You are memory constrained, and the expensive part is standing around. Adding more GPUs does not fix this. It gives you more processors starving for the same data. Here is the part that decides the next decade. Memory can grow. When a cell cannot shrink any further, you stack it into a high-rise, layer on layer. A GPU cannot be stacked. It runs too hot and needs a cooler bolted to its back, so the one move that rescues memory is closed to the processor. The thing that can keep stacking compounds. The thing that cannot plateaus. The marginal dollar in an AI build now buys more by fixing the memory path than by bolting on another idle GPU. Which is why the companies that control memory bandwidth and supply are not suppliers to the AI trade. They are the AI trade.

Fireside Alpha

38,370 views • 17 days ago

New blackboard lecture w Reiner Pope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

New blackboard lecture w Reiner Pope How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. 0:00:00 – Building a multiply-accumulate from logic gates 0:16:20 – Muxes and the cost of data movement 0:25:59 – How systolic arrays work 0:39:00 – Clock cycles and pipeline registers 0:51:40 – FPGAs vs ASICs 1:03:14 – Cache vs scratchpad 1:07:16 – Why CPU cores are much bigger than GPU cores 1:11:49 – Brains vs chips 1:15:22 – A GPU is just a bunch of tiny TPUs Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

Dwarkesh Patel

933,337 views • 1 month ago

The CPU-intensive nature of #Samson makes it one of the best use cases for Frame Generation. The GPU has plenty of headroom, allowing FG to operate more efficiently, this results in a substantial boost to smoothness and frame rate with a small increase in latency.

The CPU-intensive nature of #Samson makes it one of the best use cases for Frame Generation. The GPU has plenty of headroom, allowing FG to operate more efficiently, this results in a substantial boost to smoothness and frame rate with a small increase in latency.

BenchmarKing

25,538 views • 3 months ago

We are excited to share a deeper look into the GPU Sharing Platform that will power AI applications on the InfraX V4 DApp. In this demo we showcase a GPU running and completing an image generation task. Our backend successfully leverages this GPU (and CPU) in real time to complete the job, earning the user money, and powering the infraX platform. Here’s how it works 👉 when you connect to the InfraX DApp, our platform automatically detects the GPU on your machine and immediately checks it eligibility and where it can be used. From there, each user dictates exactly how and when they want to contribute their GPU power by setting the days their PC is most available (throughout the night, for example). Additional, each user will be able to pause their GPU's connection to infraX whenever they need to use it locally, resuming with a just a single click. Once activated, your GPU seamlessly plugs into the InfraX network and begins fulfilling compute jobs, earning YOU money. Go on, take the $INFRA pill..

We are excited to share a deeper look into the GPU Sharing Platform that will power AI applications on the InfraX V4 DApp. In this demo we showcase a GPU running and completing an image generation task. Our backend successfully leverages this GPU (and CPU) in real time to complete the job, earning the user money, and powering the infraX platform. Here’s how it works 👉 when you connect to the InfraX DApp, our platform automatically detects the GPU on your machine and immediately checks it eligibility and where it can be used. From there, each user dictates exactly how and when they want to contribute their GPU power by setting the days their PC is most available (throughout the night, for example). Additional, each user will be able to pause their GPU's connection to infraX whenever they need to use it locally, resuming with a just a single click. Once activated, your GPU seamlessly plugs into the InfraX network and begins fulfilling compute jobs, earning YOU money. Go on, take the $INFRA pill..

infraX | $INFRA

30,615 views • 10 months ago

React tip: "use client" misconceptions (1/5) 🚫 You need to mark EVERY individual file with "use client" to ensure it is a Client Component. ✅ You only need to mark each client entry point ONCE. Modules imported into a "use client" file are already part of the client bundle. (0:05) Adding "use client" to `3.js` marks the file as a client entry point, including it in the client bundle. (0:11) Importing `d()` into `3.js` (a client entry point) also includes it in the client bundle. (0:14) Adding "use client" to `4.js` has no additional bundling effect on `d()` because it is already part of the client bundle. (0:23) All modules (and their dependencies) imported to `3.js` are included in the client bundle. (0:26) Removing "use client" from `3.js` moves it and its dependencies into the server bundle.

React tip: "use client" misconceptions (1/5) 🚫 You need to mark EVERY individual file with "use client" to ensure it is a Client Component. ✅ You only need to mark each client entry point ONCE. Modules imported into a "use client" file are already part of the client bundle. (0:05) Adding "use client" to `3.js` marks the file as a client entry point, including it in the client bundle. (0:11) Importing `d()` into `3.js` (a client entry point) also includes it in the client bundle. (0:14) Adding "use client" to `4.js` has no additional bundling effect on `d()` because it is already part of the client bundle. (0:23) All modules (and their dependencies) imported to `3.js` are included in the client bundle. (0:26) Removing "use client" from `3.js` moves it and its dependencies into the server bundle.

Delba

157,365 views • 2 years ago

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 views • 2 years ago

THE MS-A2 IS THE MINISFORUM BOX HERMES AGENT WAS NEVER MEANT TO RUN ON, AND THAT'S BY DESIGN The MS-S1 MAX earns its spot running Hermes Agent because of one thing. 128GB of unified memory, up to 96GB of it handed straight to the GPU. That's the only reason a 120B model fits and runs locally for $0 a month The MS-A2 solves a different problem. Ryzen 9 9955HX, 16 cores, 32 threads, up to 96GB of regular DDR5-5600, no unified pool Three M.2 PCIe 4.0 slots, one U.2, two 22110. Dual 10Gbps SFP+ LAN plus 2.5G. WiFi 6E. Bluetooth 5.3. A slide-out motherboard for fast upgrades. A real PCIe x16 slot that actually takes a low-profile GPU That last part is where the two machines split for good. The MS-S1 MAX's PCIe slot won't take a GPU at all, every bit of GPU power has to come from the unified chip itself. The MS-A2 trades that unified memory trick for raw expandability instead One box runs a local AI agent. The other runs a home lab that needs storage, networking, and room to grow. Minisforum built both on purpose, not as the same product wearing two names

NO1ennn

25,119 views • 26 days ago

you can't tell me AI isn't a game changer for programming it's connecting over webrtc the screen is made of 16 by 16 regions each region is 256 by 256 ive come up with a custom encoding I spam UDP packets for each reigion for each client each region is designed to fit in a single packet the next state of the simulation is computed in my GPU using cuda the cuda is linked to golang the program manages a double buffer this is so that the network broadcastloop doesn't contend with the cuda update for the object when you write it sends a udp packet for every mouse event i manage a stream to update it on my gpu the client takes each packet and has custom webgl to update the regions and render everything in a single pass. texture 3d

you can't tell me AI isn't a game changer for programming it's connecting over webrtc the screen is made of 16 by 16 regions each region is 256 by 256 ive come up with a custom encoding I spam UDP packets for each reigion for each client each region is designed to fit in a single packet the next state of the simulation is computed in my GPU using cuda the cuda is linked to golang the program manages a double buffer this is so that the network broadcastloop doesn't contend with the cuda update for the object when you write it sends a udp packet for every mouse event i manage a stream to update it on my gpu the client takes each packet and has custom webgl to update the regions and render everything in a single pass. texture 3d

kache

97,819 views • 1 year ago

Discover how an indie team of just four uses Unreal Engine to produce Bellyfant🐘 The popular children’s show on YouTube has garnered over 50 million views and through the use of UE, the team has rediscovered the fun in animating. Read all about it here:

Discover how an indie team of just four uses Unreal Engine to produce Bellyfant🐘 The popular children’s show on YouTube has garnered over 50 million views and through the use of UE, the team has rediscovered the fun in animating. Read all about it here:

Unreal Engine

11,066 views • 8 months ago

Dylan Patel on the importance of memory and storage Two key quotes: "An $NVDA GPU is faster than an $AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads." “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly" Full Quote: “We have over $80 million of compute: GPUs from $NVDA and $AMD, TPUs from Google, and Trainium from Amazon. We constantly run this benchmark using the newest inference engines, drivers, PyTorch versions, and other software. It runs every day through automated CI across the latest Chinese models from GLM, Zhipu, Moonshot, Kimi, Alibaba, and others. Initially, when we were benchmarking the differences between these chips, inference engines, and parallelism schemes, we used fixed context lengths. But with Agent X, we have now analyzed more than $5 million worth of Claude Code traces. This is real production traffic that users have donated to us, combined with internally generated data, so we now understand what an actual agent workload looks like. When we implement those workloads and run the benchmarks, it turns out that the chip you are using is very important, but how you handle memory offload can be even more important. An Nvidia GPU is faster than an AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads. Similarly, you can use a less powerful GPU with a much better storage solution and outperform the best GPU when it lacks those solutions. Simply buying the newest GPU does not necessarily give you the best inference economics. You need to layer in other innovations, including storage and memory.” Interviewer: “Who is the top player on your chart? Can you tell us?” Dylan Patel: “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly.”

Dylan Patel on the importance of memory and storage Two key quotes: "An $NVDA GPU is faster than an $AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads." “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly" Full Quote: “We have over $80 million of compute: GPUs from $NVDA and $AMD, TPUs from Google, and Trainium from Amazon. We constantly run this benchmark using the newest inference engines, drivers, PyTorch versions, and other software. It runs every day through automated CI across the latest Chinese models from GLM, Zhipu, Moonshot, Kimi, Alibaba, and others. Initially, when we were benchmarking the differences between these chips, inference engines, and parallelism schemes, we used fixed context lengths. But with Agent X, we have now analyzed more than $5 million worth of Claude Code traces. This is real production traffic that users have donated to us, combined with internally generated data, so we now understand what an actual agent workload looks like. When we implement those workloads and run the benchmarks, it turns out that the chip you are using is very important, but how you handle memory offload can be even more important. An Nvidia GPU is faster than an AMD GPU in most cases, but because AMD GPUs have more memory, they can outperform Nvidia in certain workloads. Similarly, you can use a less powerful GPU with a much better storage solution and outperform the best GPU when it lacks those solutions. Simply buying the newest GPU does not necessarily give you the best inference economics. You need to layer in other innovations, including storage and memory.” Interviewer: “Who is the top player on your chart? Can you tell us?” Dylan Patel: “It is a difficult, multivariable problem. Generally, you need the best GPU, such as a GB300, but you also need the best storage solutions. I will not spoil who comes out on top, but storage solutions matter a lot, memory solutions matter a lot, and frontend networking also matters significantly.”

Daniel Romero

38,220 views • 6 days ago

Nvidia announces the new RTX Spark, a new platform powered by the NX1 CPU, and shows off Spark laptops running 007 First Light and Forza 6. The CPU has 20 ARM based cores and a Blackwell RTX GPU with 6144 CUDA Cores. This is the same core count as a 5070, but with 128GB of unified LPDDR5X RAM memory sitting in the same package as the CPU and GPU. The entire Nvidia software stack is available, particularly CUDA, vital for AI. Nvidia's new laptops will likely be ideal for running local LLMs be cause the unified memory means you can load models up to 120-180B parameters (quantized). These laptops are expected to ship later this year and could become strong competitors to high-end MacBooks and even Mac Studios for local AI workloads, thanks to CUDA support and unified memory. Price is unannounced.

Nvidia announces the new RTX Spark, a new platform powered by the NX1 CPU, and shows off Spark laptops running 007 First Light and Forza 6. The CPU has 20 ARM based cores and a Blackwell RTX GPU with 6144 CUDA Cores. This is the same core count as a 5070, but with 128GB of unified LPDDR5X RAM memory sitting in the same package as the CPU and GPU. The entire Nvidia software stack is available, particularly CUDA, vital for AI. Nvidia's new laptops will likely be ideal for running local LLMs be cause the unified memory means you can load models up to 120-180B parameters (quantized). These laptops are expected to ship later this year and could become strong competitors to high-end MacBooks and even Mac Studios for local AI workloads, thanks to CUDA support and unified memory. Price is unannounced.

Grummz

30,573 views • 1 month ago

ARE 1 ON 0 WORKOUTS THE BEST WAY TO TRAIN? I moved away from 1 on 0 workouts a long time ago. They are great if the player has a specific technical need. It allows you to slow down and dive deep with the player. Other than that… multi player workouts are exponentially more productive and beneficial for the players. This allows them to still work on technical skill, but it allows you to put them in an environment where they have to read multiple players on the court- exactly how the game will be. Add in the fact that it forces players to compete and it’s a no brainer!

ARE 1 ON 0 WORKOUTS THE BEST WAY TO TRAIN? I moved away from 1 on 0 workouts a long time ago. They are great if the player has a specific technical need. It allows you to slow down and dive deep with the player. Other than that… multi player workouts are exponentially more productive and beneficial for the players. This allows them to still work on technical skill, but it allows you to put them in an environment where they have to read multiple players on the court- exactly how the game will be. Add in the fact that it forces players to compete and it’s a no brainer!

Steve Dagostino

11,423 views • 1 year ago

I got Llama 3 running in my browser using only my GPU with my Wi-Fi switched OFF completely client-side WebGPU is a new feature in browsers where JS can use the GPU of the device and apparently you can run LLMs on it too, and it's fast! My end goal for 🧠 is a locally run client side 100% private conversation where nothing gets sent to, processed in or stored in the cloud This will be helpful because I prefer making it web based instead of coding for 4 different native platforms (iOS, Android, MacOS, Windows) Two challenges still: 1) the user has to download the LLM model first and the smallest model is still ~3GB, but they only have to download it once 2) not all devices have fast GPUs yet for LLMs, but it's starting to become a default in most devices e.g.s smartphones

I got Llama 3 running in my browser using only my GPU with my Wi-Fi switched OFF completely client-side WebGPU is a new feature in browsers where JS can use the GPU of the device and apparently you can run LLMs on it too, and it's fast! My end goal for 🧠 is a locally run client side 100% private conversation where nothing gets sent to, processed in or stored in the cloud This will be helpful because I prefer making it web based instead of coding for 4 different native platforms (iOS, Android, MacOS, Windows) Two challenges still: 1) the user has to download the LLM model first and the smallest model is still ~3GB, but they only have to download it once 2) not all devices have fast GPUs yet for LLMs, but it's starting to become a default in most devices e.g.s smartphones

@levelsio

261,352 views • 2 years ago

i been running Qwen3.5-35B-A3B UD-Q4_K_XL through Claude Code since llama.cpp merged the Anthropic endpoint. configured it in minutes. everything was great. projects grew from single scripts to multifile systems with 8 modules and 3,000+ lines. then the chains started breaking. 3 to 5 minutes of pure autonomy and suddenly it stops. tool call fails. reprompt. it recovers. 2 minutes later it stops again. the model is fine. the harness is the bottleneck. saw a comment suggesting OpenCode. installed it. pointed it at the same localhost endpoint running the same model on the same GPU. the game is different. instead of stopping on a bad tool call it just keeps going. on wrong read it adjusts. if file not found it retries. the flow is unbroken. i watched it plan a refactor across 8 files, read every module, and start building without a single pause. in Claude Code that same task would have stopped 4 times. the tradeoff is sometimes it loops. same tool call repeated because the model loses track of what it already read. but here is the thing. i choose loops over pauses. a loop you can interrupt and redirect. a broken chain stops the flow and you have to reprompt to get it moving again. someone is solving this at the core level and i have a feeling it is the open source community. the fact that i can run this level of autonomous coding intelligence on a single consumer GPU with 24gb VRAM at 112 tokens per second. respect to the chinese labs. respect to the open source builders making this possible.

i been running Qwen3.5-35B-A3B UD-Q4_K_XL through Claude Code since llama.cpp merged the Anthropic endpoint. configured it in minutes. everything was great. projects grew from single scripts to multifile systems with 8 modules and 3,000+ lines. then the chains started breaking. 3 to 5 minutes of pure autonomy and suddenly it stops. tool call fails. reprompt. it recovers. 2 minutes later it stops again. the model is fine. the harness is the bottleneck. saw a comment suggesting OpenCode. installed it. pointed it at the same localhost endpoint running the same model on the same GPU. the game is different. instead of stopping on a bad tool call it just keeps going. on wrong read it adjusts. if file not found it retries. the flow is unbroken. i watched it plan a refactor across 8 files, read every module, and start building without a single pause. in Claude Code that same task would have stopped 4 times. the tradeoff is sometimes it loops. same tool call repeated because the model loses track of what it already read. but here is the thing. i choose loops over pauses. a loop you can interrupt and redirect. a broken chain stops the flow and you have to reprompt to get it moving again. someone is solving this at the core level and i have a feeling it is the open source community. the fact that i can run this level of autonomous coding intelligence on a single consumer GPU with 24gb VRAM at 112 tokens per second. respect to the chinese labs. respect to the open source builders making this possible.

Sudo su

67,032 views • 4 months ago

JUST IN AND UNUSUAL CHINA is stripping U.S. companies of a key technological advantage… China has officially entered the global gaming‑GPU race with the launch of its first fully home‑grown graphics processor, the “LX 7G100” — a milestone that reflects its rapid rise in advanced chip and semiconductor technologies. Developed by Lisuan Tech, the new GPU is China’s first direct challenger to industry giants NVIDIA and AMD, and it has already received Microsoft’s WHQL certification, confirming its readiness for global deployment. Despite releasing only 1,000 units in the first batch, pre‑orders exceeded 22,000 requests within 24 hours — a strong signal of growing confidence in China’s emerging tech ecosystem.

JUST IN AND UNUSUAL CHINA is stripping U.S. companies of a key technological advantage… China has officially entered the global gaming‑GPU race with the launch of its first fully home‑grown graphics processor, the “LX 7G100” — a milestone that reflects its rapid rise in advanced chip and semiconductor technologies. Developed by Lisuan Tech, the new GPU is China’s first direct challenger to industry giants NVIDIA and AMD, and it has already received Microsoft’s WHQL certification, confirming its readiness for global deployment. Despite releasing only 1,000 units in the first batch, pre‑orders exceeded 22,000 requests within 24 hours — a strong signal of growing confidence in China’s emerging tech ecosystem.

China pulse 🇨🇳

33,274 views • 1 month ago

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:

Awni Hannun

42,761 views • 11 months ago

$NBIS is becoming one of the biggest winners in the new AI economy because it can deliver compute efficiently at scale. The number that mattered most to me was Nebius seeing four or more customers competing for every GPU it brings online (huge for AI cloud market). Management also said they raised prices again and are still selling out across both old and new GPU generations which is one of the cleanest pricing power signals you can get. This is a $100B company in the making.

$NBIS is becoming one of the biggest winners in the new AI economy because it can deliver compute efficiently at scale. The number that mattered most to me was Nebius seeing four or more customers competing for every GPU it brings online (huge for AI cloud market). Management also said they raised prices again and are still selling out across both old and new GPU generations which is one of the cleanest pricing power signals you can get. This is a $100B company in the making.

Shay Boloor

538,242 views • 2 months ago

Blendr's Beta Platform is Now Live 🖥 Our GPU Renting and Lending platform is officially live and you can use it now at Watch the tutorial video attached to this post to discover how to use the platform and read on to understand exactly how it works! 🔄 On-Demand Instance Creation Instead of keeping GPUs running 24/7 on the cloud, which can be very costly, AWS instances are created as and when they are requested to prevent unnecessary costs and inefficiency. 🔒 Secure Access A new set of SSH keys are generated for each instance for the user to download and login with. This ensures that only the current renter has access to their instance. 🔧 Instance Management Blendr’s Instance Manager service automatically terminates the instance after the rental period ends which ensures efficient use of resources. 📊 Transaction Tracking Users can view their transaction history through the dedicated Transaction Tab for ease of use and management. 💰 Lend Your Own GPU Power If you have spare GPU Capacity, you can join Blendr’s decentralization movement and leverage it to make some extra money! All you need to do is head to the Blendr CLI at to download our CLI tool, connect your GPU, and start contributing to the network. Dive in and utilise the power of decentralized GPU computing today!

Blendr's Beta Platform is Now Live 🖥 Our GPU Renting and Lending platform is officially live and you can use it now at Watch the tutorial video attached to this post to discover how to use the platform and read on to understand exactly how it works! 🔄 On-Demand Instance Creation Instead of keeping GPUs running 24/7 on the cloud, which can be very costly, AWS instances are created as and when they are requested to prevent unnecessary costs and inefficiency. 🔒 Secure Access A new set of SSH keys are generated for each instance for the user to download and login with. This ensures that only the current renter has access to their instance. 🔧 Instance Management Blendr’s Instance Manager service automatically terminates the instance after the rental period ends which ensures efficient use of resources. 📊 Transaction Tracking Users can view their transaction history through the dedicated Transaction Tab for ease of use and management. 💰 Lend Your Own GPU Power If you have spare GPU Capacity, you can join Blendr’s decentralization movement and leverage it to make some extra money! All you need to do is head to the Blendr CLI at to download our CLI tool, connect your GPU, and start contributing to the network. Dive in and utilise the power of decentralized GPU computing today!

Blendr Network

130,435 views • 2 years ago

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,148 views • 1 month ago

Gavin Baker (Gavin Baker) says the disaggregation of inference can extend GPU useful lives from 3-4 years to 10-15. That may single-handedly save private credit and reduce the financing rates for GPUs, which will drive demand and help finance the build-out. "The disaggregation of prefill and inference is going to be amazing for the useful lives of GPU and may single-handedly save private credit. Private credit is in pain from these SaaS loans. But there's a lot of private credit in GPUs too. They were underwriting that to 3-4. The disaggregation of inference means that these GPUs are going to have 10 or 15-year lives. The AI skeptics are like, "Oh, these companies are all cooking their books. The useful life of a GPU is only a year or two. The useful life of a CPU is only four years because the rapid technological change." No. What rapid technological change has done with the disaggregation of prefill and inference is you can put a Cerebras system or Groq LPUs effectively in front of a Hopper or even an Ampere, use that Hopper and Ampere for prefill, and extend the useful life of that GPU until it melts. This is going to be really good for the whole private credit industry. It's gonna help finance the AI build-out. Because if you can start to finance GPUs at 5% or 6% instead of – I think CoreWeave's lowest financing was low sevens – that actually mathematically changes the cost to finance this build-out."

Gavin Baker (Gavin Baker) says the disaggregation of inference can extend GPU useful lives from 3-4 years to 10-15. That may single-handedly save private credit and reduce the financing rates for GPUs, which will drive demand and help finance the build-out. "The disaggregation of prefill and inference is going to be amazing for the useful lives of GPU and may single-handedly save private credit. Private credit is in pain from these SaaS loans. But there's a lot of private credit in GPUs too. They were underwriting that to 3-4. The disaggregation of inference means that these GPUs are going to have 10 or 15-year lives. The AI skeptics are like, "Oh, these companies are all cooking their books. The useful life of a GPU is only a year or two. The useful life of a CPU is only four years because the rapid technological change." No. What rapid technological change has done with the disaggregation of prefill and inference is you can put a Cerebras system or Groq LPUs effectively in front of a Hopper or even an Ampere, use that Hopper and Ampere for prefill, and extend the useful life of that GPU until it melts. This is going to be really good for the whole private credit industry. It's gonna help finance the AI build-out. Because if you can start to finance GPUs at 5% or 6% instead of – I think CoreWeave's lowest financing was low sevens – that actually mathematically changes the cost to finance this build-out."

Invest Like the Best

206,845 views • 1 month ago