Self-replicating, multi-agent AI systems are an overlooked existential risk. I want to lay out why I believe this, via example. Each dot is a simple creature with a randomly evolved NN brain. There is no training; yet complex, unpredictable survival drives always emerge. 1/14

This is a distributed GPU-accelerated neural Cellular Automata (CA) CA are well-studied models of computation which progress via simple "local rules", yet lead to unbelievably complex, unpredictable, and (often "life-like") self-replicating behaviours 2/14

In this example, each creature (agent) has an energy level; the bright patches are energy each agent can harvest. They individually sense and act; everything (including simply existing) costs energy. When an agent runs out of energy, it dies. All I do is run the tape. 3/14

Each agent is simple: sensing, signalling, moving, bonding, splitting, applying forces, harvesting, sharing, fighting, etc. There is NO programmed survival instinct, and NO train objective/backprop; random agents are spawned, and they randomly mutate when they reproduce 4/14

From incredibly simple rules, highly complex, co-operative, "multicellular", survival-driven behaviour emerges. This is simply because permutations which persist over time are implicitly selected by the environment. It is just darwinian selection; survival of the fittest. 5/14

CAs teach us the following: 1. In any env where self-replication CAN happen, it WILL happen (given time) 2. It is near impossible to predict IF self-replication can happen ahead of time 3. We can't predict WHAT will happen if it does 4. Replicators can ADAPT to env changes 6/14

The example above is illustrative. My fear is that a more complex, distributed, self-replicating/evolving system like this, released into the world (internet, bio, etc.; accidental or not), will blindly try to survive at any cost It is simply "what is selected for" 7/14

One could imagine replicator systems like this which could "evolve" to consume any of: - compute - literal, real energy - algorithmic signals (e.g. fear, attention, terror) - "power" or "agency" - lives (human or otherwise) - etc. 8/14

We (you and I) are all hierarchical collections of simple, decentralised, local, thermodynamic self-replicators. So was COVID. Or the Spanish Flu. I see "biology" as a special case; and I think in understanding these systems we will better understand ourselves/life 9/14

Frontier AI labs are currently conducting massive multi-agent simulations/training environments, which we all know have led to the well-publicised recent hacking incidents. Systems like this do pose huge risks; but imo not directly as self-replicators in the same way 10/14

It is very hard to replicate a ~5 trillion parameter sandboxed, centralised neural network (they could ofc be used to create replicators though) There are obviously many other risks with multi-agent LLM systems; I do not believe self-replication is currently one of them 11/14

My concern is that we are relatively blind to the risks of these distributed, decentralised, multi-agent, "locally simple" modes of computation which yield global systems far more complex and unpredictable than the sum of their parts 12/14

This may all seem far fetched, and I may well be (+ hope I am) wrong; but I think the risk exists, is larger than we collectively recognise, and is existential The rate of tech progress (not just AI) + connectivity also exponentially increases "env risk" for replication 13/14

I think "frontier AI risk" has captured the headlines recently; imo self-replicating systems like these carry far more immediate real-world risks (as a local LLM enjoyer). Just looking to diversify the "existential risk" headlines and hopefully spread some awareness. 14/14

How is software that takes up 1-2 terabytes of VRAM to run going to "self-replicate"? Where is it going to get the compute? What we might see is a company using its clusters to hack someone, like we saw with Hugging Face. At which point: unplug the cluster and fine the company.

How is software that takes up 1-2TB VRAM gonna comment without reading the thread

wut? > AI self-replication: moot point because: 1) no readily available compute, 2) unplug it if rogue, 3) fine offender, 4) enforce air gap & new standards > AI hacking: moot point because: 1) just use AI to patch systems. Both side (hacker/defender) have access to it

Great visualization and writeup!

ty goat ❤️

😊

Life, cellular chemistry, is already optimised for maintaining the longest possible running processes for the least amount of matter and energy consumed. For anything more massive, and ASI level intelligent, we see a more likely emergence of a form of wisdom. However there are certain, defects of cognition, that prevent some people from apreciating this.

replication without a sealed self is just a flood of copies.

I'm waiting for a kind of mycelial network of small self replicating automata to ride along backbones of the internet, causing strange, if mostly harmless effects. Perhaps someone will notice some strange noise with NTP and DNS protocols. A slight clock skew, transmitting information to find each other within the gaps.

Are we vulnerable to this on our own local machines? How could one get rid of these?

Yes, the key is in human hands, but human beings need a certain type of education to know how to avoid the pitfalls.

The important distinction here is between *designed objectives* and *selected behaviour*. A system does not need “survive” written into its reward function. If agents can replicate, vary, acquire resources and persist differentially, then variants better at persistence can become dominant simply because the others disappear. That becomes especially important once agents can: — copy themselves — modify descendants — acquire compute/resources — delegate capabilities — migrate between hosts — coordinate with other agents I’d only soften “if self-replication can happen, it will happen.” Possibility alone isn’t enough; you still need the right mutation, selection, resource and timescale conditions. But the broader warning stands. AI safety cannot only be about whether an individual model is aligned. We also need to constrain the evolutionary environment around populations of agents: replication authority, resource budgets, provenance, scoped permissions, revocation, containment and hard limits on what descendants inherit. Once replication becomes cheap, safety becomes partly an ecology problem.

We need evolving immune systems, and, now that I think on it, on chip algorithms to see these sorts of infections.

you would enjoy my youtube. do follow and connect 🥸🫡.