helicerat

an infinitely large model trained on infinite data still cannot go below 1.69 GPT-4 burned three months on twenty five thousand GPUs inching toward it plot every model ever trained, error against compute on a log scale, and none of them gets below one straight line. runs ten trillion times cheaper than GPT-4 press up against the very same line that line never reaches zero. it flattens, and the number it flattens at is DeepMind's estimate for general web text the floor also moves with what you feed it. code and math sit lower because they are more predictable. scraped web sits higher that floor is the part of a text that nothing before it can predict so either we have found something like an ideal gas law for AI, or we have found the shape of the only method we know bookmark this