正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Inkling, Thinking Machines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio. Serving and RL support are already live: you can run and shape it on an open stack, starting now. Day 0 support on SGLang... SGLang and Miles RadixArk👇 - Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache - Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization - DFlash speculative decoding from Modal for low-latency serving Launching now, blog and cookbook in the comments ⬇️show more

LMSYS Org

17,933 subscribers

146,770 次观看 • 3 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 条评论

LMSYS Org 的头像

LMSYS Org3 个月前

Run Inkling with SGLang:

LMSYS Org 的头像

LMSYS Org3 个月前

Blog with full technical details:

LMSYS Org 的头像

LMSYS Org3 个月前

Shape Inkling with Miles:

Confusing Perspectives 的头像

Confusing Perspectives3 个月前

@thinkymachines This is the kind of launch that actually matters: not just weights, but a usable training + serving path on day one. 1M context, native multimodal reasoning, and open-stack RL support makes Inkling

Loong🐉 的头像

Loong🐉3 个月前

@thinkymachines day-0 SGLang + RL on a 975B/41B MoE with 1M context is the real flex — weights alone don't move the stack; runnable tooling from launch day does.

相关视频

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles RadixArk! How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production! Thanks to Kimi.ai, NVIDIA, AMD, KVCache.AI, Modal, and Baseten for building this with us, and to Google Cloud, Nebius Token Factory, fal, DigitalOcean, Runpod, DeepInfra and GMI Cloud for serving K3 on SGLang. Blog, cookbook, benchmarks in the comments. P.S. This demo video? Kimi K3 made it itself. Play the game 👇

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles RadixArk! How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production! Thanks to Kimi.ai, NVIDIA, AMD, KVCache.AI, Modal, and Baseten for building this with us, and to Google Cloud, Nebius Token Factory, fal, DigitalOcean, Runpod, DeepInfra and GMI Cloud for serving K3 on SGLang. Blog, cookbook, benchmarks in the comments. P.S. This demo video? Kimi K3 made it itself. Play the game 👇

LMSYS Org

206,484 次观看 • 3 个月前

🎉 Reflection-Beam-1.0 from Reflection has Day-0 support in vLLM. 501B total parameters, 23B active per token, and a 1M-token context window. Beam combines hybrid attention with Harmony-format reasoning and tool use. Beam also has Day-0 RL support in SkyRL (NovaSky): LoRA, BF16 full fine-tuning and MXFP8 RL, with vLLM rollouts. We worked with Reflection and Inferact on the serving path, from fused kernels to DFlash. Here is what is under the hood. 🧵

🎉 Reflection-Beam-1.0 from Reflection has Day-0 support in vLLM. 501B total parameters, 23B active per token, and a 1M-token context window. Beam combines hybrid attention with Harmony-format reasoning and tool use. Beam also has Day-0 RL support in SkyRL (NovaSky): LoRA, BF16 full fine-tuning and MXFP8 RL, with vLLM rollouts. We worked with Reflection and Inferact on the serving path, from fused kernels to DFlash. Here is what is under the hood. 🧵

vLLM

13,140 次观看 • 24 天前

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

🎉 MiniMax M3.1 Flash from MiniMax (official) has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

vLLM

11,790 次观看 • 18 天前

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

40,306 次观看 • 5 个月前

⚡ Speed of flash. Just 2 days after launch, DFlash is already running in SGLang (SGLang). With serving-engine support, we can now unlock speedup with higher concurrency, and we’ve quickly worked on a new demo based on it. We'll be cooking up more and better draft models over the next few weeks.🔥 Stay tuned!

⚡ Speed of flash. Just 2 days after launch, DFlash is already running in SGLang (SGLang). With serving-engine support, we can now unlock speedup with higher concurrency, and we’ve quickly worked on a new demo based on it. We'll be cooking up more and better draft models over the next few weeks.🔥 Stay tuned!

Zhijian Liu

21,233 次观看 • 10 个月前

📢 Qwen's Qwen3.8-2.4T-A95B weights just dropped, and SGLang already has day-0 support with DSpark. HF: Cookbook: 🖖 What kind of model can help optimize an inference engine? Qwen3.8-2.4T-A95B can take on complex engineering tasks, plan and iterate on its own, and keep going until the job is done. We even asked it to optimize its own serving on SGLang, and it ran unattended for 4.5 hours with verified performance improvements. Thanks to the Qwen team Qwen, NVIDIA AI, and AI at AMD for the close collaboration.

📢 Qwen's Qwen3.8-2.4T-A95B weights just dropped, and SGLang already has day-0 support with DSpark. HF: Cookbook: 🖖 What kind of model can help optimize an inference engine? Qwen3.8-2.4T-A95B can take on complex engineering tasks, plan and iterate on its own, and keep going until the job is done. We even asked it to optimize its own serving on SGLang, and it ran unattended for 4.5 hours with verified performance improvements. Thanks to the Qwen team Qwen, NVIDIA AI, and AI at AMD for the close collaboration.

SGLang

27,317 次观看 • 2 个月前

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys LMSYS Org and RadixArk RadixArk, and taught by Richard Chen Richard Chen, a Member of Technical Staff at RadixArk. Running LLMs in production is expensive, and much of that cost comes from redundant computation. This short course teaches you to eliminate that waste using SGLang, an open-source inference framework that caches computation already done and reuses it across future requests. When ten users share the same system prompt, SGLang processes it once, not ten times. The speedups compound quickly, especially when there's a lot of shared context across requests. Skills you'll gain: - Implement a KV cache from scratch to eliminate redundant computation within a single request - Scale caching across users and requests with RadixAttention, so shared context is only processed once - Accelerate image generation with diffusion models using SGLang's caching and multi-GPU parallelism Join and learn to make LLM inference faster and more cost-efficient at scale!

Andrew Ng

103,664 次观看 • 7 个月前

MiniMax (official) H3 is live in SGLang Diffusion, with day-0 serving support 🎬 This open model matches Seedance 2.0 at 1/3 the cost, or $0 if you run it locally on 2x 5090 or 1 RTX 6000. With SGLang Diffusion, you can build visual concepts, motion design, e-commerce creatives, precise video edits, animation and stylized visuals, all locally on your own machine. No API bill, no waitlist. H3 takes text, image, video and audio in a single context, and generates 5-15s clips at native 2K, 24fps with native stereo audio. SGLang Diffusion also runs on NVIDIA AI Blackwell and Hopper, and AMD MI355X / MI300X. Let's create something with MiniMax H3 👇

MiniMax (official) H3 is live in SGLang Diffusion, with day-0 serving support 🎬 This open model matches Seedance 2.0 at 1/3 the cost, or $0 if you run it locally on 2x 5090 or 1 RTX 6000. With SGLang Diffusion, you can build visual concepts, motion design, e-commerce creatives, precise video edits, animation and stylized visuals, all locally on your own machine. No API bill, no waitlist. H3 takes text, image, video and audio in a single context, and generates 5-15s clips at native 2K, 24fps with native stereo audio. SGLang Diffusion also runs on NVIDIA AI Blackwell and Hopper, and AMD MI355X / MI300X. Let's create something with MiniMax H3 👇

LMSYS Org

101,623 次观看 • 3 个月前

Today we’re launching INTELLECT-2: The first decentralized 32B-parameter RL training run open to join for anyone with compute — fully permissionless. Scaling towards frontier reasoning across coding, math and science.

Today we’re launching INTELLECT-2: The first decentralized 32B-parameter RL training run open to join for anyone with compute — fully permissionless. Scaling towards frontier reasoning across coding, math and science.

Prime Intellect

354,097 次观看 • 1 年前

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉 DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales. We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec. At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads. With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300. DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks DeepSeek for open-sourcing! Blog with full technical details and commands to run below 👇

LMSYS Org

170,752 次观看 • 4 个月前

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses. pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second. That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use. FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that's the stack.

h100envy

32,605 次观看 • 4 个月前

🎉 Congrats to Fish Audio on launching Fish Audio S2, a frontier TTS model with fine-grained prosody & emotion control via natural-language inline tags. SGLang Day-0 support is now live! 🏆 Best WER on Seed-TTS Eval; 81.88% win rate on EmergentTTS-Eval 🎙️ Voice cloning with 86.4% prefix-cache hit rate via RadixAttention ⚡️ RTF 0.34, 63.3 tok/s on single H200 (single batch) 🌍 Trained on 10M+ hours of audio across ~100 languages, GRPO-aligned 🔧 Dual-AR (Slow + Fast AR) is LLM-isomorphic: continuous batching, paged KV cache & CUDA graphs inherited natively 🗣️ Native multi-speaker: turn-taking, interruptions & cross-speaker emotion in a single pass 👉Cookbook: 👉Blog: 🎬 Curious how to run with SGLang? Check out this voice cloning demo from Guitar Cat + LLM 「Will @ SGLang Summit in SF」 with Fishaudio-S2-Pro:

🎉 Congrats to Fish Audio on launching Fish Audio S2, a frontier TTS model with fine-grained prosody & emotion control via natural-language inline tags. SGLang Day-0 support is now live! 🏆 Best WER on Seed-TTS Eval; 81.88% win rate on EmergentTTS-Eval 🎙️ Voice cloning with 86.4% prefix-cache hit rate via RadixAttention ⚡️ RTF 0.34, 63.3 tok/s on single H200 (single batch) 🌍 Trained on 10M+ hours of audio across ~100 languages, GRPO-aligned 🔧 Dual-AR (Slow + Fast AR) is LLM-isomorphic: continuous batching, paged KV cache & CUDA graphs inherited natively 🗣️ Native multi-speaker: turn-taking, interruptions & cross-speaker emotion in a single pass 👉Cookbook: 👉Blog: 🎬 Curious how to run with SGLang? Check out this voice cloning demo from Guitar Cat + LLM 「Will @ SGLang Summit in SF」 with Fishaudio-S2-Pro:

LMSYS Org

41,930 次观看 • 8 个月前

DBRX by Databricks ...it's REALLY good!! The New MoE 132b parameter model is open-source and costs $10 m to train. Thank you, Databricks, for your contribution to OS. Check out the full explanation and testing: 🎥👇

DBRX by Databricks ...it's REALLY good!! The New MoE 132b parameter model is open-source and costs $10 m to train. Thank you, Databricks, for your contribution to OS. Check out the full explanation and testing: 🎥👇

Matthew Berman

28,341 次观看 • 2 年前

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: Built with Reese Chong. Check out the repos and blog if you want to learn more. Shoutout to Modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc sunny madra Gavin

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: Built with Reese Chong. Check out the repos and blog if you want to learn more. Shoutout to Modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc sunny madra Gavin

Aadi Kulshrestha

829,709 次观看 • 6 个月前

🎉 Meet Krea 2 from Krea, an aesthetic open-source image model ranked #1 text-to-image from an independent lab on Artificial Analysis. Day-0 support is now live in SGLang! Krea 2 ships as two models built to work together: 1️⃣ RAW: undistilled base checkpoint: diverse & malleable, made for fine-tuning, post-training & LoRA 2️⃣ Turbo: 8-step distilled checkpoint: fast, high-quality text-to-image Train LoRAs on RAW, run them on Turbo: base for training, Turbo for fast inference on your own hardware Run it now with SGLang!

🎉 Meet Krea 2 from Krea, an aesthetic open-source image model ranked #1 text-to-image from an independent lab on Artificial Analysis. Day-0 support is now live in SGLang! Krea 2 ships as two models built to work together: 1️⃣ RAW: undistilled base checkpoint: diverse & malleable, made for fine-tuning, post-training & LoRA 2️⃣ Turbo: 8-step distilled checkpoint: fast, high-quality text-to-image Train LoRAs on RAW, run them on Turbo: base for training, Turbo for fast inference on your own hardware Run it now with SGLang!

LMSYS Org

11,216 次观看 • 4 个月前

Introducing Magenta RealTime 2 (MRT2): the live music model you can play as an instrument. MRT2 offers MIDI and prompt controls, and runs natively on a MacBook with <200ms latency. Open weights. Open source inference engine. Suite of apps and plugins. Hear what it can do and try it out for yourself below 🧵

Introducing Magenta RealTime 2 (MRT2): the live music model you can play as an instrument. MRT2 offers MIDI and prompt controls, and runs natively on a MacBook with <200ms latency. Open weights. Open source inference engine. Suite of apps and plugins. Hear what it can do and try it out for yourself below 🧵

Google Magenta Project

114,728 次观看 • 5 个月前

Muse Spark Open is built to plan, act, and use computer across agentic tasks for life and work. You can now build a powerful personal AI agent with an open stack, and SGLang to make it: - Ultra-responsive with NVFP4 MegaMoE and sequence parallelism - Faster for agentic and coding tasks with our DFlash2 draft model - Customizable for specialized use cases via fine-tuning with Miles We’re incredibly proud to have made this possible with AI at Meta, NVIDIA AI, AI at AMD, Intel, and RadixArk!

Muse Spark Open is built to plan, act, and use computer across agentic tasks for life and work. You can now build a powerful personal AI agent with an open stack, and SGLang to make it: - Ultra-responsive with NVFP4 MegaMoE and sequence parallelism - Faster for agentic and coding tasks with our DFlash2 draft model - Customizable for specialized use cases via fine-tuning with Miles We’re incredibly proud to have made this possible with AI at Meta, NVIDIA AI, AI at AMD, Intel, and RadixArk!

SGLang

286,889 次观看 • 25 天前

New on DigitalOcean's Inference Engine: GLM-5.3-Flash from Z.ai. Agentic coding that rivals top-tier models, now natively multimodal for browser and computer use. 1M token context, MoE architecture, one endpoint swap to try it. Senior Developer Advocate Ryan O'Connor breaks it down, and Z.ai for Startups is there to help AI startups build faster with GLM models. 👇

New on DigitalOcean's Inference Engine: GLM-5.3-Flash from Z.ai. Agentic coding that rivals top-tier models, now natively multimodal for browser and computer use. 1M token context, MoE architecture, one endpoint swap to try it. Senior Developer Advocate Ryan O'Connor breaks it down, and Z.ai for Startups is there to help AI startups build faster with GLM models. 👇

DigitalOcean

140,795 次观看 • 2 个月前

Introducing "Building with Llama 4." This short course is created with Meta AI at Meta, and taught by Amit Sangani, Director of Partner Engineering for Meta’s AI team. Meta’s new Llama 4 has added three new models and introduced the Mixture-of-Experts (MoE) architecture to its family of open-weight models, making them more efficient to serve. In this course, you’ll work with two of the three new models introduced in Llama 4. First is Maverick, a 400B parameter model, with 128 experts and 17B active parameters. Second is Scout, a 109B parameter model with 16 experts and 17B active parameters. Maverick and Scout support long context windows of up to a million tokens and 10M tokens, respectively. The latter is enough to support directly inputting even fairly large GitHub repos for analysis! In hands-on lessons, you’ll build apps using Llama 4’s new multimodal capabilities including reasoning across multiple images and image grounding, in which you can identify elements in images. You’ll also use the official Llama API, work with Llama 4’s long-context abilities, and learn about Llama’s newest open-source tools: its prompt optimization tool that automatically improves system prompts and synthetic data kit that generates high-quality datasets for fine-tuning. If you need an open model, Llama is a great option, and the Llama 4 family is an important part of any GenAI developer's toolkit. Through this course, you’ll learn to call Llama 4 via API, use its optimization tools, and build features that span text, images, and large context. Please sign up here:

Introducing "Building with Llama 4." This short course is created with Meta AI at Meta, and taught by Amit Sangani, Director of Partner Engineering for Meta’s AI team. Meta’s new Llama 4 has added three new models and introduced the Mixture-of-Experts (MoE) architecture to its family of open-weight models, making them more efficient to serve. In this course, you’ll work with two of the three new models introduced in Llama 4. First is Maverick, a 400B parameter model, with 128 experts and 17B active parameters. Second is Scout, a 109B parameter model with 16 experts and 17B active parameters. Maverick and Scout support long context windows of up to a million tokens and 10M tokens, respectively. The latter is enough to support directly inputting even fairly large GitHub repos for analysis! In hands-on lessons, you’ll build apps using Llama 4’s new multimodal capabilities including reasoning across multiple images and image grounding, in which you can identify elements in images. You’ll also use the official Llama API, work with Llama 4’s long-context abilities, and learn about Llama’s newest open-source tools: its prompt optimization tool that automatically improves system prompts and synthetic data kit that generates high-quality datasets for fine-tuning. If you need an open model, Llama is a great option, and the Llama 4 family is an important part of any GenAI developer's toolkit. Through this course, you’ll learn to call Llama 4 via API, use its optimization tools, and build features that span text, images, and large context. Please sign up here:

Andrew Ng

68,034 次观看 • 1 年前

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

in the lectures below, i hold your hand through low-level LLM systems engineering. it includes everything up to TODAY! 1) pytorch tensors 2) large matmul on cpu vs gpu 3) JAX (and why xAI uses it instead of pytorch) 4) raw cuda kernels and global threading indexing 5) triton design philosophy and softmax example 6) HIP kernels 7) mapping out the ENTIRE ecosystem + differences between CUDA and ROCm/HIP (BLAS, FFT, DNN) 8) cutlass and cute-dsl 9) pretraining, finetuning, rl, unsloth, axolotl, megatron-lm, deepspeed, nanogpt, nanochat 10) training vs inference, inference serving problems, throughput vs latency vs concurrency scaling, vllm, sglang, tensorrt-llm, tensorrt, llama.cpp, exllamav2, exllamav3, benchmark comparisons 11) projects/companies using llms to generate SOTA cuda/triton kernels 12) luminal inference 13) mojo/modular/max

Elliot Arledge

58,522 次观看 • 1 年前