Llama 3 was *still* learning when Meta stopped training it. They only stopped because they decided they needed the GPUs to start testing for Llama 4. AI scaling laws are insane.

“The models just want to learn.” - Ilya

Box, dodge, and squat your way through PowerBeatsVR - Now 40% OFF on Meta Quest for a limited time 🔥

Even crazier is that they only trained with 16k H100s despite having ordered 600k. So only 2.6% of the compute from that order was used to create a GPT-4 level model. Wild times ahead.

I’m only halfway through this so far, but Zuck even says he thinks energy will be a bottleneck before compute. Wild!

Almost all LLMs are undertrained relative to size

@HlibIvanov source of the clip

Very Very promising for FSD and Teslas Fleet advantage

llama 3 still *is* learning there’s a checkpoint on the biggest 400B model today, but they’re still training it

AGI 💖 Mark 🤣

“There are these meta reasoning questions…”

So maybe scale actually is all you need…?