Kimi K3 full fine-tuning is live on AC2. Our memory optimizations reduced GPUs required per training replica by ~40%. At nearly 3T parameters, Kimi forced us to rethink how we manage memory, communication, rollouts, and checkpoints. The result is a much more efficient path to training frontier-scale open models.

At Kimi K3 scale, memory directly determines how many GPUs you need for training. For example, streaming gradients for Adam updates cuts the host memory peak by ~33%. Another bottleneck was SiTU-GLU activations. Naive autograd saves redundant tensors and consumes >100GB of HBM at ~100k tokens/GPU. A custom operator that streams through a fixed-size workspace saves ~90GB of HBM per GPU.

On the inference side, MXFP4 rollouts ran on 2 B300 nodes instead of 4, while staying within the same KL range we observed with bf16 inference. Less memory and fewer inference nodes directly lower the cost of frontier-scale RL.

Supporting Kimi K3 required changes across training memory, rollout precision, weight transfer, checkpointing, and communication. Those improvements also carry over to other models on AC2. Read the full report.

Bro what

Forty percent fewer GPUs per replica is the real story. Memory layout is the frontier now.