Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Kimi K3 full fine-tuning is live on AC2. Our memory optimizations reduced GPUs required per training replica by ~40%. At nearly 3T parameters, Kimi forced us to rethink how we manage memory, communication, rollouts, and checkpoints. The result is a much more efficient path to training frontier-scale open models.

appliedcompute's profile picture

Applied Compute

7,802 subscribers

234,323 Aufrufe • vor 19 Tagen •via X (Twitter)

Wissenschaft & TechnologieBildungNachrichten & Politik
LIVE

Anya Rossi• Live Now

Private livecam show

5 Kommentare

Profilbild von Applied Compute
Applied Computevor 19 Tagen

At Kimi K3 scale, memory directly determines how many GPUs you need for training. For example, streaming gradients for Adam updates cuts the host memory peak by ~33%. Another bottleneck was SiTU-GLU activations. Naive autograd saves redundant tensors and consumes >100GB of HBM at ~100k tokens/GPU. A custom operator that streams through a fixed-size workspace saves ~90GB of HBM per GPU.

Profilbild von Applied Compute
Applied Computevor 19 Tagen

On the inference side, MXFP4 rollouts ran on 2 B300 nodes instead of 4, while staying within the same KL range we observed with bf16 inference. Less memory and fewer inference nodes directly lower the cost of frontier-scale RL.

Profilbild von Applied Compute
Applied Computevor 19 Tagen

Supporting Kimi K3 required changes across training memory, rollout precision, weight transfer, checkpointing, and communication. Those improvements also carry over to other models on AC2. Read the full report.

Profilbild von Anant Dwivedi (c/acc)
Anant Dwivedi (c/acc)vor 18 Tagen

Bro what

Profilbild von Neeraj Kumar
Neeraj Kumarvor 19 Tagen

Forty percent fewer GPUs per replica is the real story. Memory layout is the frontier now.

Ähnliche Videos

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6
0:30

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6

vLLM

104,039 Aufrufe • vor 1 Monat

Kimi K3 is out. 2.8T parameters. 1.4 TB of weights, frontier-level performance. Serving it on your own infra is the hard part - you need multi-node inference spread across every cluster you can grab capacity from. And then autoscaling, cache-aware routing, PD disaggregation - an entire serving stack around the model. SkyPilot Endpoints ships a complete stack on day 0 for Kimi.ai's Kimi K3: • Runs on GPUs you own - hyperscalers, neoclouds or on-prem. • Production-ready out of the box - autoscaling, KV-aware routing, PD disaggregation, observability • Fault-tolerant by design - endpoint stays up even when GPUs fail Frontier open models, on GPUs you own. 🔗
0:20

Kimi K3 is out. 2.8T parameters. 1.4 TB of weights, frontier-level performance. Serving it on your own infra is the hard part - you need multi-node inference spread across every cluster you can grab capacity from. And then autoscaling, cache-aware routing, PD disaggregation - an entire serving stack around the model. SkyPilot Endpoints ships a complete stack on day 0 for Kimi.ai's Kimi K3: • Runs on GPUs you own - hyperscalers, neoclouds or on-prem. • Production-ready out of the box - autoscaling, KV-aware routing, PD disaggregation, observability • Fault-tolerant by design - endpoint stays up even when GPUs fail Frontier open models, on GPUs you own. 🔗

SkyPilot

14,794 Aufrufe • vor 1 Monat

Claude Fable 5 vs Kimi K3 — same 3D build, 3.3x cheaper. someone prototyped a 3D stadium seat-view picker with Fable 5 — preview exactly what you'll see from your seat before buying a ticket. I rebuilt the same thing with Kimi K3 in two hours, at 3.3x lower cost. no more static seat maps. pick a seat, see the actual view in 3D, then buy. works for cinemas, concerts, cricket grounds, Wimbledon — any venue. K3 dropped this week: 2.8 trillion parameters, open weights July 27. 3D is the future, and it just got a lot cheaper to build. Live Fable 5: Live Kimi K3: Code: full Kimi K3 breakdown below.
0:47

Claude Fable 5 vs Kimi K3 — same 3D build, 3.3x cheaper. someone prototyped a 3D stadium seat-view picker with Fable 5 — preview exactly what you'll see from your seat before buying a ticket. I rebuilt the same thing with Kimi K3 in two hours, at 3.3x lower cost. no more static seat maps. pick a seat, see the actual view in 3D, then buy. works for cinemas, concerts, cricket grounds, Wimbledon — any venue. K3 dropped this week: 2.8 trillion parameters, open weights July 27. 3D is the future, and it just got a lot cheaper to build. Live Fable 5: Live Kimi K3: Code: full Kimi K3 breakdown below.

Kirill

207,314 Aufrufe • vor 1 Monat

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, VP of AI at AMD. Available now at Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here:
2:27

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, VP of AI at AMD. Available now at Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here:

Andrew Ng

132,304 Aufrufe • vor 10 Monaten

Sam on Kimi, distillation, and open source: “I have always assumed that there are going to be great cheap models in the world, and we better be the greatest and the cheapest. You get a better deal today, at least at a particular latency, using OpenAI's models than Kimi. We distill our own models. That’s how we make smaller, cheaper models. That’s a very good thing to do. There will clearly be an important place for open source models in the world and people that will want their own weights. We have so much usage of our models that we do not need to be a gigantically high-margin business to be able to afford model training. I would rather people not distill from us, for sure. But this is not in my top ten list of worries.”
2:12

Sam on Kimi, distillation, and open source: “I have always assumed that there are going to be great cheap models in the world, and we better be the greatest and the cheapest. You get a better deal today, at least at a particular latency, using OpenAI's models than Kimi. We distill our own models. That’s how we make smaller, cheaper models. That’s a very good thing to do. There will clearly be an important place for open source models in the world and people that will want their own weights. We have so much usage of our models that we do not need to be a gigantically high-margin business to be able to afford model training. I would rather people not distill from us, for sure. But this is not in my top ten list of worries.”

Patrick OShaughnessy

269,663 Aufrufe • vor 1 Monat

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰
0:32

GPU tradeoff series: A100 is not much more powerful than 4090 🫠 GPU Perf and Price: - 4090: 330 fp16 TFLOPs, $1,749 - A100 (80GB): 312 fp16 TFLOPs, $20,000 > A100 is 11.4X more pricy Training speed for GPT-2(124M) with llm.c: - 4090: 153K tokens/s - A100 (80GB): 195K tokens/s > A100 is only 1.3X faster (both trained using a single card, A100 llm.c training is shown in the video, 4090 video is in the quoted tweet) Conclusion: 4090 has a much better cost vs performance ratio Why: As in the H100 vs. 4090 comparison, the biggest difference between A100 and 4090 is their GPU memory size/bandwidth and cross-GPU communication bandwidth, which does not matter too much if your model can fit into a single 4090. Specs: 4090: - GPU memory size: 24GB - memory bandwidth: 1 TB/s - communication bandwidth: 64 GB/s A100: - GPU memory size: 80GB - memory bandwidth: 2 TB/s - communication bandwidth: 900 GB/s Nvidia killed off NVLink (a high-speed communication link that connects GPUs) on 4090. (Jensen Huang smiling face) If multiple 4090s could be interconnected via NVLink, their performance would be closer to datacenter-grade A100 GPUs, even for training larger models. Additionally, 4090 isn't allowed in datacenters, that's how Nvidia makes 💰💰💰

Yuchen Jin

234,951 Aufrufe • vor 2 Jahren

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.
1:05

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

233,484 Aufrufe • vor 13 Tagen

OpenAI's chairman just responded to the Kimi K3 panic 👇 The timeline calls K3 a frontier killer. Bret Taylor's answer on CNBC: every CEO he talks to asks the same 2 questions. 1️⃣ Am I getting value from my token spend? 2️⃣ What's my moat in the age of AI? His counter on cheap Chinese models: frontier models finish the same task in far fewer tokens. Cheap per token means little if you burn 5x more of them The new battleground is tokens per task, and the price war just forced OpenAI to argue efficiency instead of capability
8:11

OpenAI's chairman just responded to the Kimi K3 panic 👇 The timeline calls K3 a frontier killer. Bret Taylor's answer on CNBC: every CEO he talks to asks the same 2 questions. 1️⃣ Am I getting value from my token spend? 2️⃣ What's my moat in the age of AI? His counter on cheap Chinese models: frontier models finish the same task in far fewer tokens. Cheap per token means little if you burn 5x more of them The new battleground is tokens per task, and the price war just forced OpenAI to argue efficiency instead of capability

Ruben

76,682 Aufrufe • vor 1 Monat

Prime Intellect engineers explained how they train reasoning models over the open internet in 30 minutes - better than $3000 distributed training courses. split policy and rollouts across nodes -> run agents in parallel envs -> verify with LLM judges -> gradient-sync over the internet -> train Llama, Qwen, Gemma at cluster scale on rented GPUs. That loop is why open reasoning models are catching closed labs without owning a data center. Prime-RL + verifiers + distributed rollouts + LLM judges + multi-cloud GPUs - that's the stack. Watch and save it, then launch your first distributed RL run this week.
30:09

Prime Intellect engineers explained how they train reasoning models over the open internet in 30 minutes - better than $3000 distributed training courses. split policy and rollouts across nodes -> run agents in parallel envs -> verify with LLM judges -> gradient-sync over the internet -> train Llama, Qwen, Gemma at cluster scale on rented GPUs. That loop is why open reasoning models are catching closed labs without owning a data center. Prime-RL + verifiers + distributed rollouts + LLM judges + multi-cloud GPUs - that's the stack. Watch and save it, then launch your first distributed RL run this week.

h100envy

40,022 Aufrufe • vor 2 Monaten

Bubble Boi reveals the Kimi breakthrough that made him sell his entire portfolio overnight "Kimi is using a technique called linear attention that cuts down memory growth entirely to a fixed size, and they didn't just get okay results, they got world-class results. This same technique was used by the Qwen models but nobody thought it would scale. It's looking more and more likely that it scales pretty well, and if that's the case the memory per user is always fixed." "So if every frontier model goes to this, the demand for DRAM drops significantly. They already bought the memory, they just serve more and more users. It's like you had a car and someone found a way to make it 10 times faster, you're not going to buy a new car." "But you also have to readjust the assumption that hyperscalers are going to crazily spend for the next two years. What's to say there's not another innovation that shrinks this more and now you go from serving a thousand users to a hundred thousand users on one server rack? How do you justify spending 500k on a new rack at that point?"
3:10

Bubble Boi reveals the Kimi breakthrough that made him sell his entire portfolio overnight "Kimi is using a technique called linear attention that cuts down memory growth entirely to a fixed size, and they didn't just get okay results, they got world-class results. This same technique was used by the Qwen models but nobody thought it would scale. It's looking more and more likely that it scales pretty well, and if that's the case the memory per user is always fixed." "So if every frontier model goes to this, the demand for DRAM drops significantly. They already bought the memory, they just serve more and more users. It's like you had a car and someone found a way to make it 10 times faster, you're not going to buy a new car." "But you also have to readjust the assumption that hyperscalers are going to crazily spend for the next two years. What's to say there's not another innovation that shrinks this more and now you go from serving a thousand users to a hundred thousand users on one server rack? How do you justify spending 500k on a new rack at that point?"

Kai

390,730 Aufrufe • vor 1 Monat

Osmantic founder Ahmad says Anthropic isn't worth $2 trillion if open models can catch the frontier with post-training alone: "GLM 5.3 is catching up. Kimi K3 is catching up to Fable 5. We're seeing models that are not even new base trained with just extra post-training reaching the frontier." "Are they scared? Of course they are scared. Their entire moat was that they are the only one able to do research and boost these models in a certain direction." "They no longer can IPO at a $1 or $2 trillion. Anthropic is not worth $2 trillion. I'm sorry. That's a joke. If somebody tells me Anthropic is worth one third of NVIDIA, absolutely not." Osmantic
1:05

Osmantic founder Ahmad says Anthropic isn't worth $2 trillion if open models can catch the frontier with post-training alone: "GLM 5.3 is catching up. Kimi K3 is catching up to Fable 5. We're seeing models that are not even new base trained with just extra post-training reaching the frontier." "Are they scared? Of course they are scared. Their entire moat was that they are the only one able to do research and boost these models in a certain direction." "They no longer can IPO at a $1 or $2 trillion. Anthropic is not worth $2 trillion. I'm sorry. That's a joke. If somebody tells me Anthropic is worth one third of NVIDIA, absolutely not." Osmantic

MTS

16,977 Aufrufe • vor 6 Tagen

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO Sharon Zhou, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations: - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here:
3:18

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO Sharon Zhou, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations: - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here:

Andrew Ng

66,508 Aufrufe • vor 2 Jahren

$WDC CEO says storage demand and requirement compound and grow whether AI runs on US frontier models or cheaper open-source ones. "Whether it's the traditional frontier models that you've seen in the U.S. that are premium-based, or some of the new open-source, more economical models, the underlying requirement is still that they're going to require a lot of data to support the training of these models." "These models are going to generate a lot more data that requires storage for us." "So we view it as very positive, because even as compute maybe drives greater efficiency in terms of compute and memory resources, the requirement for storage is just going to compound and grow." "So we view it very positively for demand going forward."
0:35

$WDC CEO says storage demand and requirement compound and grow whether AI runs on US frontier models or cheaper open-source ones. "Whether it's the traditional frontier models that you've seen in the U.S. that are premium-based, or some of the new open-source, more economical models, the underlying requirement is still that they're going to require a lot of data to support the training of these models." "These models are going to generate a lot more data that requires storage for us." "So we view it as very positive, because even as compute maybe drives greater efficiency in terms of compute and memory resources, the requirement for storage is just going to compound and grow." "So we view it very positively for demand going forward."

Fireside Alpha

14,114 Aufrufe • vor 1 Monat

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."
3:19

Etched is deploying two new technologies in chip design: low-voltage inference and cluster-scale memory. CEO Gavin Uberti says they'll make their chips much more power-efficient and way, way faster than today's leading GPUs. He breaks it down: "We looked at a lot of early research directions, and we realized the key things that models need are way more compute and way faster memory." "If you think about inference, there are two key parts: prefill and decode. For prefill, it's a compute-bound problem. You need to have more FLOPS, more operations per second on each of your chips." "On our GPU, the bottleneck's actually thermals. You can't really run a GPU at more than around 50% of what it could theoretically do, or it'll melt." "So we're using a new technology today called low-voltage inference to try to solve this problem. You bring the voltage of the chip down dramatically, which allows us to have way, way better efficiency in terms of how much power is drawn per unit of math, and thus fit way way more flops onto the chip..." "For decode, it's all about bandwidth. Not just bandwidth on a chip, but bandwidth across your cluster. That's why we have this technology we call cluster-scale memory. It reduces the amount of time it takes to communicate from one chip to another dramatically." "As a result we can go use all of our HBM, HBM bandwidth, SRAM, SRAM bandwidth, and our scale-up domain as a single coherent pool. And that means if you're a user, you can go get much faster tokens per second, while still keeping your costs low."

TBPN

20,404 Aufrufe • vor 2 Monaten

Kimi K3 is too large to run efficiently on a single accelerator so the workload must be distributed across many chips that exchange data at extremely high speeds. $CRDO and $ALAB benefit from the high speed connectivity inside AI clusters while $MRVL and $AVGO provide the networking, optical and custom silicon infrastructure required to scale them. Thats why inference constraints are bullish for the entire surrounding system including memory, networking, optics, switches, power, cooling and orchestration software.
0:42

Kimi K3 is too large to run efficiently on a single accelerator so the workload must be distributed across many chips that exchange data at extremely high speeds. $CRDO and $ALAB benefit from the high speed connectivity inside AI clusters while $MRVL and $AVGO provide the networking, optical and custom silicon infrastructure required to scale them. Thats why inference constraints are bullish for the entire surrounding system including memory, networking, optics, switches, power, cooling and orchestration software.

Shay Boloor

84,505 Aufrufe • vor 1 Monat

Moonshot AI limiting Kimi K3 subscriptions because of compute constraints is a massive signal because the 2.8 trillion parameter model is already trained and the bottleneck is now emerging as usage scales. Serving that demand requires a ton more accelerators, enormous memory capacity and faster networking to keep data moving efficiently across the cluster. So as AI spending shifts toward serving millions or billions of daily requests then memory and networking become critical bottlenecks giving $MU, $SKHY, $AVGO, $CRDO, $ALAB and $MRVL direct exposure to inference at scale.
0:42

Moonshot AI limiting Kimi K3 subscriptions because of compute constraints is a massive signal because the 2.8 trillion parameter model is already trained and the bottleneck is now emerging as usage scales. Serving that demand requires a ton more accelerators, enormous memory capacity and faster networking to keep data moving efficiently across the cluster. So as AI spending shifts toward serving millions or billions of daily requests then memory and networking become critical bottlenecks giving $MU, $SKHY, $AVGO, $CRDO, $ALAB and $MRVL direct exposure to inference at scale.

Shay Boloor

123,791 Aufrufe • vor 1 Monat

Unsloth AI and NVIDIA are Revolutionizing Local LLM Fine-Tuning: From RTX Desktops to DGX Spark Fine-tune popular AI models faster with Unsloth on NVIDIA RTX AI PCs such as GeForce RTX desktops and laptops to RTX PRO workstations and the new DGX Spark to build personalized assistants for coding, creative work, and complex agentic workflows. The landscape of modern AI is shifting. We are moving away from a total reliance on massive, generalized cloud models and entering the era of local, agentic AI. Whether it is tuning a chatbot to handle hyper-specific product support or building a personal assistant that manages intricate schedules, the potential for generative AI on local hardware is boundless. However, developers face a persistent bottleneck: How do you get a Small Language Model (SLM) to punch above its weight class and respond with high accuracy for specialized tasks? The answer is Fine-Tuning, and the tool of choice is Unsloth. Unsloth provides an easy and high-speed method to customize models. Optimized for efficient, low-memory training on NVIDIA GPUs, Unsloth scales effortlessly from GeForce RTX desktops and laptop all the way to the DGX Spark, the world’s smallest AI supercomputer...... Full analysis: NVIDIA NVIDIA AI NVIDIA AIDev NVIDIAnewsroom Unsloth AI Unsloth
0:40

Unsloth AI and NVIDIA are Revolutionizing Local LLM Fine-Tuning: From RTX Desktops to DGX Spark Fine-tune popular AI models faster with Unsloth on NVIDIA RTX AI PCs such as GeForce RTX desktops and laptops to RTX PRO workstations and the new DGX Spark to build personalized assistants for coding, creative work, and complex agentic workflows. The landscape of modern AI is shifting. We are moving away from a total reliance on massive, generalized cloud models and entering the era of local, agentic AI. Whether it is tuning a chatbot to handle hyper-specific product support or building a personal assistant that manages intricate schedules, the potential for generative AI on local hardware is boundless. However, developers face a persistent bottleneck: How do you get a Small Language Model (SLM) to punch above its weight class and respond with high accuracy for specialized tasks? The answer is Fine-Tuning, and the tool of choice is Unsloth. Unsloth provides an easy and high-speed method to customize models. Optimized for efficient, low-memory training on NVIDIA GPUs, Unsloth scales effortlessly from GeForce RTX desktops and laptop all the way to the DGX Spark, the world’s smallest AI supercomputer...... Full analysis: NVIDIA NVIDIA AI NVIDIA AIDev NVIDIAnewsroom Unsloth AI Unsloth

Marktechpost AI Dev News ⚡

31,551 Aufrufe • vor 9 Monaten

Kimi K3 is worth every one of its 2.8T parameters. It's the best open-weight model I've tried. It's frontier-level quality, and it can give any model a run for its money. I'm running it on Nebius Token Factory. It comes with an OpenAI-compatible API endpoint and 120 tokens/s (although I've seen much faster responses). If you want to start using it, here is the link:
5:42

Kimi K3 is worth every one of its 2.8T parameters. It's the best open-weight model I've tried. It's frontier-level quality, and it can give any model a run for its money. I'm running it on Nebius Token Factory. It comes with an OpenAI-compatible API endpoint and 120 tokens/s (although I've seen much faster responses). If you want to start using it, here is the link:

Santiago

22,594 Aufrufe • vor 27 Tagen

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.
2:15

We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.

Physical Intelligence

442,828 Aufrufe • vor 6 Monaten