the Andrej Karpathy code:
swyx
Visit on X167K followers
53,917 views
the Andrej Karpathy code:
- be Team Human
- choose things that scale
- scale them to all of humanity
things = { Tesla camera vision | humanoid robots | transformers | AI education Eureka Labs}
Favorite pull quotes from Andrej Karpathy this morning:
- Elon Musk was right on self driving:
- "Waymo looks like it's winning right now but I think when we look in 10 years and who's actually at scale and where most of the revenue is coming from I still think [Tesla is] ahead"
- "Tesla has a software problem, Waymo has a hardware problem"... "a waymo car has a lot of very expensive LIDAR and other sort of sensors built into the car so it can do what it does...[but] if you can just use cameras which is the Tesla approach then you effectively get rid of enormous cost complexity and you can do it in in many different types of cars".
- on Optimus:
- "cars are robots"
- Tesla isn't a car company, it "is a robotics at scale company"
- "early versions of Optimus thought it was a car" - same computer, same cameras, it was walking but thought it was driving
- first applications will be in factories where it doesnt "crush grandma"
- excited for Optimus to solve the Nat Friedman challenge of the quiet leaf blower robot
- on transformers:
- "Transformers are this beautiful like blob of tissue you can just get just arbitrary tasks and you just need the data you need to put it in the right form"
- "the scaling laws are actually to a large extent a property of the Transformer. Before the Transformer, people were playing with LSTMs and stacking them, you don't actually get clean scaling laws... the Transformer was the first thing that actually just scales.
- Architecture is no longer a bottleneck, its now just dataset and objectives
- Internet data is "not what you want for your transformer, it's just a nearest neighbor that gets you really far"... "what you want is the inner thought monologue of your brain.. if we had a billion trajectories [of your brain as you're doing problem solving] then AGI is here". "the Internet is like 0.001% cognition and 99.99% of information and most of it is not useful for thinking"
- Synthetic data is largely about "refactoring the dataset into these inner monologue formats". Cites the Tencent 1 billion persona paper
- Transformers > Humans: much better at learning/memory "if you give it a sequence and you do a single forward-backward pass in that sequence then if you give it the first few elements it will complete the rest of the sequence... it memorized that sequence!" Human brain working memory is very small, transformers "are much more efficient learners".
- Most LLMs memorize useless information - a "cognitive core" LLM OS could be as smol as 1b params - just needs to think, and then use tools to look stuff up
- Bullish on an AI CEO supervising a swarm (or crew?) of smaller specialist agents
- on Education
- LLM101n will be "an undergrad level course" coming "early next year"
- "I'm always more interested in anything that empowers people... I'm on Team Human"
- cites Bloom's Two Sigma problem: "I find very interesting is like how far can a person go if they have the perfect tutor for all the subjects" - people with 1:1 tutoring get 2 stdev better results
- "I taught 231n at Stanford and that was the first deep learning class and was pretty successful but the question is how do you really scale these classes — like, how do you make it so that your target audience is maybe 8 billion people on Earth" - for different languages and different capability levels
- languages and transfer learning (from previously known domains) are low hanging fruit
- "the demo is near but the product is far"
- "so the question is how do you use AI to do the scaling of a really good teacher and so the way I'm thinking about it is the teacher is doing a lot of the course creation and the curriculum"
- "at current AI capability the models are not good enough to create a good course but I think they're good to become the front end to the student and interpret the course to them"
- Learning is supposed to be hard.. but he will "make it easier for people to learn" and in a post AGI society, learning can be entertainment if people want
- Kids today should study Math, Physics, CS - "symbol manipulation heavy tasks, not memory heavy"
Comments
Comments from the original post on X (Twitter)
No comments yet
