Загрузка видео...

Не удалось загрузить видео

На главную

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

charles_irl's profile picture

Charles 🎉 Frye

18,786 subscribers

17,452 просмотров • 6 месяцев назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Microsoft killed the GPU mafia 🤯 They finally open-sourced their 1-bit LLM inference framework called bitnet.cpp. It lets you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.
0:13

Microsoft killed the GPU mafia 🤯 They finally open-sourced their 1-bit LLM inference framework called bitnet.cpp. It lets you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.

Oliver

1,632,616 просмотров • 9 месяцев назад

Now available: NVIDIA Nemotron™ 3.5 Super VL on DigitalOcean Serverless Inference, Dedicated Inference and Managed Agents. Build agents that understand documents, images, and video—and use that context to reason and call tools. Start building:
0:16

Now available: NVIDIA Nemotron™ 3.5 Super VL on DigitalOcean Serverless Inference, Dedicated Inference and Managed Agents. Build agents that understand documents, images, and video—and use that context to reason and call tools. Start building:

DigitalOcean

354,802 просмотров • 1 месяц назад

Real-time Moondream inference using our new inference engine
0:18

Real-time Moondream inference using our new inference engine

vik

144,677 просмотров • 7 месяцев назад

First AI Credit Marketplace + inference trading on @Arc. Coming soon. 👀 Sell unused AI credits. Buy inference for less. Trade OpenAI ↔ Claude and more. AI inference is becoming a market.
0:13

First AI Credit Marketplace + inference trading on @Arc. Coming soon. 👀 Sell unused AI credits. Buy inference for less. Trade OpenAI ↔ Claude and more. AI inference is becoming a market.

AI Flow

36,810 просмотров • 1 месяц назад

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers
0:20

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 просмотров • 2 месяцев назад

the inference engine
0:11

the inference engine

Kat ⊷ the Poet Engineer

94,802 просмотров • 3 месяцев назад

Microsoft just changed the game 🤯 They've open-sourced bitnet.cpp, a 1-bit LLM inference framework. It let's you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.
0:13

Microsoft just changed the game 🤯 They've open-sourced bitnet.cpp, a 1-bit LLM inference framework. It let's you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.

Oliver

116,751 просмотров • 1 месяц назад

Now available: Qwen 3.8-2.4T-A95B from Alibaba Cloud on DigitalOcean Serverless Inference via NVIDIA HGX™ B300 GPUs. 🤖 1M context, built for long-horizon coding. 🔗 One API, usage-based pricing, no infra to manage.
0:17

Now available: Qwen 3.8-2.4T-A95B from Alibaba Cloud on DigitalOcean Serverless Inference via NVIDIA HGX™ B300 GPUs. 🤖 1M context, built for long-horizon coding. 🔗 One API, usage-based pricing, no infra to manage.

DigitalOcean

451,546 просмотров • 3 месяцев назад

Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. Congratulations Georgi Gerganov ! This is a triumph.
0:13

Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. Congratulations Georgi Gerganov ! This is a triumph.

Nat Friedman

1,764,422 просмотров • 3 лет назад

First steps for a specialized DeepSeek v4 Flash inference engine focused on inference quality / stability at different quantizations, with networked API that is batching capable. This is the 2 bit quants model running on my M3 Max 128GB.
0:12

First steps for a specialized DeepSeek v4 Flash inference engine focused on inference quality / stability at different quantizations, with networked API that is batching capable. This is the 2 bit quants model running on my M3 Max 128GB.

antirez

14,220 просмотров • 6 месяцев назад

My AI broke the world record on Tempest yesterday! But I still hold the human record :-) [on Extreme difficulty settings] Here's a little demo reel of the Tempest AI doing inference and training at the same time up on the hardest Tempest levels. This is all running on our Dell Technologies 7875 Workstation, with the 9995WX CPU handling 2000 fps of Tempest while the dual Blackwell RTX6000 GPUs do inference and training.
0:19

My AI broke the world record on Tempest yesterday! But I still hold the human record :-) [on Extreme difficulty settings] Here's a little demo reel of the Tempest AI doing inference and training at the same time up on the hardest Tempest levels. This is all running on our Dell Technologies 7875 Workstation, with the 9995WX CPU handling 2000 fps of Tempest while the dual Blackwell RTX6000 GPUs do inference and training.

Dave W Plummer

37,436 просмотров • 8 месяцев назад

It’s live. AxonDAO’s GPU fleet is installed and will soon be rentable on the open market via vast.ai. Premium GPU capacity - 8x NVIDIA B200 and 8x RTX Pro 6000 - built for large-scale AI training, inference, rendering, and research workloads. 🧵
0:14

It’s live. AxonDAO’s GPU fleet is installed and will soon be rentable on the open market via vast.ai. Premium GPU capacity - 8x NVIDIA B200 and 8x RTX Pro 6000 - built for large-scale AI training, inference, rendering, and research workloads. 🧵

AxonDAO

17,725 просмотров • 8 месяцев назад

Today, Nesa is excited to announce it is partnering with io.net to bring their decentralized fleet of GPUs to Nesa. Nesa and are natural complements to one another. Nesa’s private, distributed AI inference executed on ocean of decentralized compute means greater accessibility for the network. Together, we are one step closer to decentralizing AI for all.
0:16

Today, Nesa is excited to announce it is partnering with io.net to bring their decentralized fleet of GPUs to Nesa. Nesa and are natural complements to one another. Nesa’s private, distributed AI inference executed on ocean of decentralized compute means greater accessibility for the network. Together, we are one step closer to decentralizing AI for all.

Nesa

258,062 просмотров • 2 лет назад

Depth Anything 3 now runs as pure C++/ggml (ggml) . No Python, no PyTorch, no CUDA toolkit at inference, just one self-contained GGUF. It's faster than PyTorch on CPU! and ties speed on GPU. The CPU win came from the last place..I'd have looked. Quantized GGUF on Hugging Face🤗 Shout out to Georgi Gerganov for ggml (we are building a ggml-world!❤️) and to ByteDance Open Source and Depth Anything 3 authors Bingyi Kang Jun Hao Liew Donny Y. Chen !
0:11

Depth Anything 3 now runs as pure C++/ggml (ggml) . No Python, no PyTorch, no CUDA toolkit at inference, just one self-contained GGUF. It's faster than PyTorch on CPU! and ties speed on GPU. The CPU win came from the last place..I'd have looked. Quantized GGUF on Hugging Face🤗 Shout out to Georgi Gerganov for ggml (we are building a ggml-world!❤️) and to ByteDance Open Source and Depth Anything 3 authors Bingyi Kang Jun Hao Liew Donny Y. Chen !

Ettore Di Giacinto

34,915 просмотров • 4 месяцев назад

Just the example itself is so fun to play with! 🔆 I can't wait to pull the code and swap the rendering pipeline to achieve even more cool effects. This is the dream of TypeGPU, AI inference, complex compositing, all on the GPU in a seamless, type-safe pipeline.
0:15

Just the example itself is so fun to play with! 🔆 I can't wait to pull the code and swap the rendering pipeline to achieve even more cool effects. This is the dream of TypeGPU, AI inference, complex compositing, all on the GPU in a seamless, type-safe pipeline.

Iwo Plaza | TypeGPU

122,314 просмотров • 2 месяцев назад

gameday snapshots
0:15

gameday snapshots

NEW YORK KNICKS

32,413 просмотров • 13 дней назад

Excited to share our NeurIPS 2024 Oral, Convolutional Differentiable Logic Gate Networks, leading to a range of inference efficiency records, including inference in only 4 nanoseconds 🏎️. We reduce model sizes by factors of 29x-61x over the SOTA. Paper:
0:12

Excited to share our NeurIPS 2024 Oral, Convolutional Differentiable Logic Gate Networks, leading to a range of inference efficiency records, including inference in only 4 nanoseconds 🏎️. We reduce model sizes by factors of 29x-61x over the SOTA. Paper:

Felix Petersen

157,840 просмотров • 2 лет назад