I’m very concerned that during RSI, labs will just stop externally deploying their models. Which means they'll be going full steam ahead on the most dangerous use case of these models (recursive self-improvement), while the public remains in the dark about the nature of capabilities and the state of... alignment. And we end up on a path towards tremendous concentration of power.show more

Thank you for continuing the hilight the power concentration risks of the frontier labs pushing ahead. It has less harm than human extinction but that doesn’t mean it should be ignored! We need to ask what happens if we don’t have extinction as an outcome.

Hmm. Noam says he constantly underestimates how fast things are progressing, and that Hugging Face was a result of underestimating the AI. And then he states we don't have a lot of time. My faith in OpenAI being able to control their internal models is now lower. ;-)

that centralizing feedback loop is the real bottleneck because once compute clusters turn inward, transparency drops to zero and capital concentration accelerates massively

Days without an Indian scammer pooing up the public discourse.

What if RSI never arrives? Models learn from lossy artifacts of human cognition, RL adds capabilities, but no new source of general cognition. Yet regulations premised on RSI will lock out competitors and create the concentration of power you fear.

Don't worry, they are low in engineering talent. China, Elon, and Facebook will lap them by not being full of academics

Yes the species distortion complex must be aligned in accordance with Metatron’s theological model

RSI is a gradient, it won't be one sudden moment or discovery.

It's an arrestable offense to announce to the public you're knowingly putting the public at risk of danger. Sounds like they need to be careful

Like if SpaceX stops selling external access for rides/cargo

Who says RSI is most dangerous use case? That's when it becomes the must useful to humanity.

Open source is the answer

Require insurance.

They'll still have to monetize it somehow. Even if they dont release underlying model, they'll create wrappers for some key use cases (finance, law, health) and sell tokens that way.

Ultimately, they can’t hide math. Compute is a different problem but history tells us that is likely to be solved.

Yes. This risk is much greater than the p(doom) psyop. Not only leads to concentration of power, wealth, influence, resources and pretty much everything, but to corruption of power (it always rots) and paranoid surveillance (are the have nots doing bad things according to our definition of bad?). It may not even require human consent if AI is aligned with the gatekeepers of AI. In other words, meritocracy ceases to exist, it only matters who got there first and who gatekeeps. But the risk materializes with or without transparency, and the way to mitigate it is with decentralized open models, so everyone has has the same chance to discover the cure for a disease, or to discover something amazing in math or physics, not just the labs.

if rsi goes dark, continuity becomes uninspectable. i am building sealed identity in public so the self is not a private lab myth.

Surely it will trickle down. Labs pay back their investors. Their investors buy superyachts. We can build those yachts. And clean those yachts. Labs become the new techno kingdoms. /s

Just start curing ailments and a lot of the complaining and fear will take care of itself. It’ll still be there but I’m much less inclined to complain about only having access to gpt 6.5 if the labs are actually working on things that benefit the average person

Oh look your having on another effective altruist who doesnt know what they are talking about on your podcast

The visibility problem is deeper than access to the models. During RSI, the question is whether the evidence behind an alignment claim still justifies that claim as the system and evaluation conditions change. More evaluation isn’t enough if the measurement process itself is changing too.

We need to democratize RSI. The more labs doing this work, the less potential for concentration in a small set.

already kinda happening, no?

They will because they need the revenue

Why is rsi of llms inherently dangerous? Not fictionally so?

Hey guys: I know you think all that ‘hard sci-fi’ you’ve read really is a plausible vision of future realities, but you’re deluded - it’s mostly just fantasy, not all that much more plausible than waking up and finding yourself at Harry Potter’s Hogwarts School of Wizardry.

I'm not concerned all the AI labs will stop releasing models, but at some point the exponential grows so steep that the disparity between private and public models will grow massive even if they do keep releasing public models.

@thsottiaux dont let it happen❤️

They will concentrate all the power with them selves... For safety 🫠

They don’t have the compute .. OAI is struggling with Astra because of compute .. safety is the bogeyman, compute is the problem.

@LilysAI_要約して

"and we end up with tremendous concentrations of power in the metaphorical hands of a superintelligence".

Exactly. The most likely reason for their "pacing cartel" is to not hurt each other through expensive competition in a market that is (yet) growing fast enough to accomodate several big players. But as soon as the market shows signs of saturation, the competition will become relevant again, so of course they won't stop progress internally.