Scale AI's Alexandr Wang says "the data wall and the wall on progress that we've hit right now is certainly real" and synthetic data isn't living up to its promise, with more human-generated data needed

Source (thanks to @curiousgangsta):

Contrarian take: we have an abundance of data but it is behind corporate firewalls. That's why the foundational public models (ie. OAI) feel like they've hit a wall. They haven't cracked the nut of the enterprise. And, quite frankly, no enterprise should trust them.

Synthetic data is a huge, huge dead-end. We should be gathering more data from actual reality - cameras, microphones - there's an endless trove of data that isn't on the internet.

conflict of interest tho, his company facilitates human data labeling and synthetic data is his competition.

Of course synthetic data won’t get us to AGI. You can’t just materialize knowledge from a model and hope it makes a smarter model better. It’s like expecting a toddler can teach a PhD student on organic chemistry.

Like they have trained on all of youtube.... that's a ridiculously massive data mine that Google I'm sure are honing their tools on.

translation: i’m gpu poor

"Human annotated data company leader proclaims that synthetic data is not useful and we need more human annotated data." Lol. Lmao.

Sounds like skill issues lol

My man out there giving interviews, creating more training tokens