Jaron Lanier on how to think about LLMs (and stuff like it doing mathematics). As a bonus, he connects current doom-talk around AI directly to the loss of optimism in popular Sci-Fi. Memes beginning in Sci-Fi fundamentally affect talk in intellectual/tech communities:

Full discussion. As always, he's worth listening to all the way through:

I like that they mention Star Trek at the very end. This is exactly how I’ve thought about LLMs / AIs. No one in Star Trek thought the ship computer was an actual being (much less some sort of ‘god’); in fact, they called it “computer” because that’s exactly what it was. All the creativity, judgement, problem solving, etc., came from the person using the computer.

Instead of addressing why it would be perfectly safe for dumb species to coexist with smart species, he attacks personalities of AI safety researches.

Been a while since I’ve seen jaron, didn’t expect ozempic face

Love Jaron what a legend

Once you adopt this as common sense, the idea of non-human *intelligence* destroying humanity seems preposterous. I’d say more than X price guys, people should listen to you. Because Peter, Emad, and all the others seem to me to be the most optimistic, pessimists out there. They’re likely not to yield the kind of optimism we actually need.

Jaron Lanier is fascinating and this is an amazing episode

Wikipedia is deterministic software. It was built by humans. Humans in full control of it. LLMs are non-deterministic. They were trained, not built. Humans are failing at controlling it.

If we get super-intelligence, it will be via more efficient communication between models, not smarter models.

Sadly, fear sells. If you're trying to create a sci-fi series saying everything is going to be great, it's just not going to get attention. You can see it today with that Anthropic employee getting 100 million views on his tweet about quitting.

That claim that optimistic sci-fi doesn't sell is refuted by the success of Star Trek - just as Jaron says! And that claim is *itself* an instance of the very pessimism being resisted, and hence self perpetuating.

disagree start trek is just one example. there are very few positive scifi series (literally can't think of any besides start trek and culture series). acknowledging it leads to more awareness about how it impacts us and hopefully spurs people to create more positive stuff.

Pretty sure he plays shaman Ork in Shadow Run.

I was just thinking about how dystopian most sci-fi was last month while watching Foundation Almost always devolves back into authoritarianism, has a war that eliminates advanced computers (this is mostly for writing purposes so whatever) People always living worse than today

why are all the tech nerds all dressing like greasers all the sudden

White rasta Morpheus is right

Niel does not seem interested

"Just because you're paranoid doesn't mean they aren't after you."

I really enjoyed this, and Jaron is wonderful. I only think my criticisms are worth expressing though. 1) Liking Star Trek doesn't make it more relevant than Terminator. The facts of the world decide which scenario is more likely. Reads as "manifesting" adjacent woo woo 2) AI has been embodied by tool calling with agents (robotics and rich sensors are not necessary.) The idea of ai as a product of people is not the whole story anymore / "digital dignity" falls flat if you consider that millions of agents are writing code, calling compilers, executing code, talking to coordinating agents, reading logs, proposing solutions, verifying in lean, literally producing multiples of the amount text/information and raw experience compared to the short prompts humans are providing. When agent transcripts feed some of the training data - that is hybrid human & machine in origin. It is not like wikipedia anymore. It seems strange to have confident opinions about the nature of llm intelligence which basically amount to him projecting his aesthetics onto it, instead of being more grounded and saying it is alien. Also he doesn't propose a mechanism for how his thinking about ai in a certain way would change the behavior of these systems. Oh yes don't mind the paperclip maximizer over there, it's got people powered digital dignity...

I agree with him though i think thats only half of the humble approach. What do you think of this reasoning, intuition about the danger that is also trying to keep ego at bay...

Critical rationalists eschew coercive or fear based memes that are most effective. Maybe sci-fi is the most effective way to create memes that create values of welcoming criticism and admitting mistakes.

I’ve long been thinking that TNG is the antidote to LLM doom & gloom. The Enterprise computer is basically an LLM. That said, I recently watched T2 again and it hits different in the age of AI.

you can call any mind as collaboration of previous minds