Neural Software talk I gave at GTC this year. The demo from this presentation is a basic Neural OS I wrote in July of 2024 inspired by Andrej Karpathy's 'LLM OS' post. Lambda's secret mission is to replace all of world's software with neural networks.

To really get there, we'll eventually need a few more things: 1. Faster and more powerful multi-modal models. 2. Byte-level encoding instead of token-based encoding. 3. Explicitly encoded sense of time in the neural network. 4. More work on both longer context windows and shared context between systems. The first is obvious: this stuff needs to run at 120Hz, not 0.1Hz and we need it to be able to generate text, image/video, audio waveforms, maybe even some form of direct access to generating radio signals. Byte-level encoding means that we can simplify the development of hardware / device drivers. Simply blit a bytestream from the device into the temporal context of the Neural OS and it should be able to figure everything out just from how the bytestream changes over time. You might prompt something like "this bytestream is coming from my mouse, please let me use it as a mouse, it has two buttons and a scroll wheel". A true sense of time seems important. Right now every 'tick' of the Neural OS clock is based on either user input or the LLM requesting an action to take place. It would be great to dis-entangle this explicit 'tick' and make the Neural OS feel more like a high FPS video game instead of a tick-by-tick text based dungeon crawler. People can do things like 'wait for 10 seconds' by counting aloud in their head, we need our Neural OS to be able to do the same thing. In our first demo of this, we networked multiple neural operating systems together and created a chat room. There is a lot of space to explore around building shared context between multiple networked neural operating systems. Finally, it's clear that there are some things that make a lot of sense to encode as squishy neural software, some things make no sense at all. For example, we probably shouldn't attempt to implement encryption in the mind of the LLM. Generally, I've found that software that is easy for you to 'run' in your own mind with only a piece of paper as a scratch pad is software that 'runs well' in a neural OS--and vice-versa. We do a good job of running the 'game of chess' program in our heads when we play chess. But try doing an MD5 hash in you head, it's not as easy. :)

Galit Perez of KSM – Maccabi Healthcare Services joined the PANCAID General Assembly at the European Parliament to advance early pancreatic cancer detection. As an Israeli partner, KSM provides biosamples and clinical insights to help develop a minimally invasive blood test.

@karpathy Looking forward to buying the Lambda workstation running a Neural OS.

@karpathy @takumi_mastery さんのアドバイス、実践して大成功。 投資タイミングの見極めが上手い。 焦らずに投資できるようになった。 これからも一緒に相場と向き合いたい。

@karpathy Sounds like Lambda's trying to make all our apps as moody as my ex! Good luck debugging that!

@karpathy the desktop themes 😂 cool stuff! Bullish for OS companies in the AI race?

@karpathy it is interesting how that conflicts with the 'model inference cost going to 0, all hail the software layer' narrative.

@trsohmers @karpathy Loved this! Made me think of how humans are constantly upgrading their cognitive abilities and then applying to their known capabilities. Prompt for walking at 5yo or 30yo may be the same but executed differently

@karpathy Maybe the future you're describing us 10 to 20 months away. Accelerate.

@karpathy Great presentation, and very interesting proof of concepts demonstrated! This takes aligning software to work with agentic AI, that Andrew Ng spoke of, to a whole new level – using AI as the software.

@karpathy I don’t understand why you would want the neural os? What makes it better or useful to have nn as an os?