How Large Language Models Store Facts

@3blue1brown This is what the internet is for, watching two 3blue1brown llm videos simultaneously

@3blue1brown thx where iz the rest of the video, pls.

@3blue1brown What a coincidence, i was just discussing this very topic last night

@3blue1brown Honestly if you need these animation to understand these concepts. You don’t need to understand them

@3blue1brown It describes a config of weights, usually tied to a layer/embedding. The "facts" in an ANN are distributed across the network, with nonlinearity between layers being crucial for capturing relationships like "big fish eats little fish," which can't be stored in a single embedding.

@3blue1brown Store data* (not facts)

@3blue1brown nailed it

@3blue1brown nearly perpendicular