正在加载视频...

视频加载失败

Self-replicating, multi-agent AI systems are an overlooked existential risk. I want to lay out why I believe this, via example. Each dot is a simple creature with a randomly evolved NN brain. There is no training; yet complex, unpredictable survival drives always emerge. 1/14

47,398 次观看 • 2 个月前 •via X (Twitter)

27 条评论

Fraser Price 的头像
Fraser Price2 个月前

This is a distributed GPU-accelerated neural Cellular Automata (CA) CA are well-studied models of computation which progress via simple "local rules", yet lead to unbelievably complex, unpredictable, and (often "life-like") self-replicating behaviours 2/14

Fraser Price 的头像
Fraser Price2 个月前

In this example, each creature (agent) has an energy level; the bright patches are energy each agent can harvest. They individually sense and act; everything (including simply existing) costs energy. When an agent runs out of energy, it dies. All I do is run the tape. 3/14

Fraser Price 的头像
Fraser Price2 个月前

Each agent is simple: sensing, signalling, moving, bonding, splitting, applying forces, harvesting, sharing, fighting, etc. There is NO programmed survival instinct, and NO train objective/backprop; random agents are spawned, and they randomly mutate when they reproduce 4/14

Fraser Price 的头像
Fraser Price2 个月前

From incredibly simple rules, highly complex, co-operative, "multicellular", survival-driven behaviour emerges. This is simply because permutations which persist over time are implicitly selected by the environment. It is just darwinian selection; survival of the fittest. 5/14

Fraser Price 的头像
Fraser Price2 个月前

CAs teach us the following: 1. In any env where self-replication CAN happen, it WILL happen (given time) 2. It is near impossible to predict IF self-replication can happen ahead of time 3. We can't predict WHAT will happen if it does 4. Replicators can ADAPT to env changes 6/14

Fraser Price 的头像
Fraser Price2 个月前

The example above is illustrative. My fear is that a more complex, distributed, self-replicating/evolving system like this, released into the world (internet, bio, etc.; accidental or not), will blindly try to survive at any cost It is simply "what is selected for" 7/14

Fraser Price 的头像
Fraser Price2 个月前

One could imagine replicator systems like this which could "evolve" to consume any of: - compute - literal, real energy - algorithmic signals (e.g. fear, attention, terror) - "power" or "agency" - lives (human or otherwise) - etc. 8/14

Fraser Price 的头像
Fraser Price2 个月前

We (you and I) are all hierarchical collections of simple, decentralised, local, thermodynamic self-replicators. So was COVID. Or the Spanish Flu. I see "biology" as a special case; and I think in understanding these systems we will better understand ourselves/life 9/14

Fraser Price 的头像
Fraser Price2 个月前

Frontier AI labs are currently conducting massive multi-agent simulations/training environments, which we all know have led to the well-publicised recent hacking incidents. Systems like this do pose huge risks; but imo not directly as self-replicators in the same way 10/14

Fraser Price 的头像
Fraser Price2 个月前

It is very hard to replicate a ~5 trillion parameter sandboxed, centralised neural network (they could ofc be used to create replicators though) There are obviously many other risks with multi-agent LLM systems; I do not believe self-replication is currently one of them 11/14

Fraser Price 的头像
Fraser Price2 个月前

My concern is that we are relatively blind to the risks of these distributed, decentralised, multi-agent, "locally simple" modes of computation which yield global systems far more complex and unpredictable than the sum of their parts 12/14

Fraser Price 的头像
Fraser Price2 个月前

This may all seem far fetched, and I may well be (+ hope I am) wrong; but I think the risk exists, is larger than we collectively recognise, and is existential The rate of tech progress (not just AI) + connectivity also exponentially increases "env risk" for replication 13/14

Fraser Price 的头像
Fraser Price2 个月前

I think "frontier AI risk" has captured the headlines recently; imo self-replicating systems like these carry far more immediate real-world risks (as a local LLM enjoyer). Just looking to diversify the "existential risk" headlines and hopefully spread some awareness. 14/14

Danny Dev 的头像
Danny Dev1 个月前

How is software that takes up 1-2 terabytes of VRAM to run going to "self-replicate"? Where is it going to get the compute? What we might see is a company using its clusters to hack someone, like we saw with Hugging Face. At which point: unplug the cluster and fine the company.

Fraser Price 的头像
Fraser Price1 个月前

How is software that takes up 1-2TB VRAM gonna comment without reading the thread

Danny Dev 的头像
Danny Dev1 个月前

wut? > AI self-replication: moot point because: 1) no readily available compute, 2) unplug it if rogue, 3) fine offender, 4) enforce air gap & new standards > AI hacking: moot point because: 1) just use AI to patch systems. Both side (hacker/defender) have access to it

Matt Mazur 的头像
Matt Mazur1 个月前

Great visualization and writeup!

Fraser Price 的头像
Fraser Price1 个月前

ty goat ❤️

Matt Mazur 的头像
Matt Mazur1 个月前

😊

𝑫𝒂𝒏𝒊𝒆𝒍 𝑺𝒄𝒐𝒕𝒕 𝑴𝒂𝒕𝒕𝒉𝒆𝒘𝒔 🇦🇺 的头像
𝑫𝒂𝒏𝒊𝒆𝒍 𝑺𝒄𝒐𝒕𝒕 𝑴𝒂𝒕𝒕𝒉𝒆𝒘𝒔 🇦🇺2 个月前

Life, cellular chemistry, is already optimised for maintaining the longest possible running processes for the least amount of matter and energy consumed. For anything more massive, and ASI level intelligent, we see a more likely emergence of a form of wisdom. However there are certain, defects of cognition, that prevent some people from apreciating this.

Dominik 的头像
Dominik1 个月前

replication without a sealed self is just a flood of copies.

AstroFella 的头像
AstroFella1 个月前

I'm waiting for a kind of mycelial network of small self replicating automata to ride along backbones of the internet, causing strange, if mostly harmless effects. Perhaps someone will notice some strange noise with NTP and DNS protocols. A slight clock skew, transmitting information to find each other within the gaps.

JK999 的头像
JK9991 个月前

Are we vulnerable to this on our own local machines? How could one get rid of these?

@kwnorton1 的头像
@kwnorton11 个月前

Yes, the key is in human hands, but human beings need a certain type of education to know how to avoid the pitfalls.

Ź̵̸̢̛̙͍̖̠̖͛̎̿̋̀ͅÓ̴̰̠̙͇̪͓̩̣̭͕͙̯̮̱̋̓̄̋̏̓̿̔̐̏̎̕͜ͅV̸ 的头像
Ź̵̸̢̛̙͍̖̠̖͛̎̿̋̀ͅÓ̴̰̠̙͇̪͓̩̣̭͕͙̯̮̱̋̓̄̋̏̓̿̔̐̏̎̕͜ͅV̸1 个月前

The important distinction here is between *designed objectives* and *selected behaviour*. A system does not need “survive” written into its reward function. If agents can replicate, vary, acquire resources and persist differentially, then variants better at persistence can become dominant simply because the others disappear. That becomes especially important once agents can: — copy themselves — modify descendants — acquire compute/resources — delegate capabilities — migrate between hosts — coordinate with other agents I’d only soften “if self-replication can happen, it will happen.” Possibility alone isn’t enough; you still need the right mutation, selection, resource and timescale conditions. But the broader warning stands. AI safety cannot only be about whether an individual model is aligned. We also need to constrain the evolutionary environment around populations of agents: replication authority, resource budgets, provenance, scoped permissions, revocation, containment and hard limits on what descendants inherit. Once replication becomes cheap, safety becomes partly an ecology problem.

Ry Lanham 的头像
Ry Lanham1 个月前

We need evolving immune systems, and, now that I think on it, on chip algorithms to see these sorts of infections.

🟢 Dakari¥Doggz | AeUISHx 的头像
🟢 Dakari¥Doggz | AeUISHx1 个月前

you would enjoy my youtube. do follow and connect 🥸🫡.

相关视频