OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

Source (thanks to @curiousgangsta):

that's an odd comparison... one is a marginal cost and the other is a fixed cost. That's like saying a physical store is 500000x more expensive than an item sold in it... yeah but so what, cost of goods is likely a larger expense.

Essentially, test time compute distributes the cost of compute, rather than concentrating it in open AI’s hand

As the technology progresses , AI models will become smaller , more efficient and better in performance. All this in parallel to cost going down

but if you serve 1Bn request a day.... what happens post day 100?

Fascinating insight from Noam Brown! The scaling of AI performance vs. inference cost shows how much potential there is to optimize efficiency in future models.

wow, thanks for sharing, súper interesting talk!

Hopefully inference chains of though gets more and more parallel though.

Inference Chips: @GroqInc 👍 @CerebrasSystems 👍 @SambaNovaAI👍