Marc Andreessen says someone ran a small Llama model on a Windows 98 Dell PC, suggesting we could have been "talking to our computers in English" for nearly 30 years. Timing is everything.

Marc Andreessen speaking at a16z speedrun:

OpenAI, in a key step toward artificial general intelligence, is developing a product to replicate the work of an experienced programmer.

Interesting thought. But wouldn’t it also have required GPU’s, that were decades away, to train the models?

No transformer architecture and pre-training would have taken forever! I believe he's saying there was no technical prerequisite for what's happening now. Just needed the right minds to come up with the right thing at the right time for this alternate timeline.

Doesn’t this ignore the compute needed for training? He says stupid shit that sounds smart every time he talks

If you can keep up with what he's saying, that is..

Andreessen is referencing an experiment by @alexocheema and @exolabs that achieved 35.9 tokens/sec on Windows 98 in December. (I recall seeing this!)

Haha :-) - ok I grok what @pmarca's saying there, and I'm as chuffed as the next guy with the recent AI progress, 🥰having also started with ML and NN-s in 1990-s. (and not expecting much of them.) And I wish it could have happened then. But I'm afraid I fail to see how. ☺️ The "Bitter Lesson" may as well be called "The Iron Low Of (Data,Compute)". 😬 Turns out we needed all of humanity to come online, and write their brains out and in the open Internet, to get enough Data. As for compute, my own mbp M2 96GB RAM rn running gpt-4 level QwQ-32B mlx model at 6 tps is an absurd luxury by the standards of year 2000 I think? I remember-then a Sparc Station with an special card with couple of T400 transputers, with custom implementing floating point libc functions in fixed point, and equipped with then unheard amount of RAM - wait for it! - of 96 MB(!!), was needed to run an ASR stack decoder. That RAM is 1000x times less, then the 96GB of my almost bog standard few years old M2 macbook next to me. So yeah. Wish what Marc's saying was true. 😆 But I don't see how it could have been. Still - far be it from me complaining, with all we witnessed in the past few years! 🤩 What a time to be alive. 🤯

@pmarca Thanks for sharing. It's a bittersweet story.. The new generation certainly stands on the shoulders of giants!

Hmmm but didn’t we require powerful GPUs to build these models?

Right but training that model would have taken 5x the world's compute power in 1998 :-p