Inspired by Andrej Karpathy 's NotebookLM project, I gave the codebase of Llama-3 Architecture to NLM and used Rag to find the perfect images to sync with the generated audio. The result exceeded my expectations. Google's NotebookLM is truly amazing :) Here is a youtube link as well:

@karpathy Have you used multi modal rag for getting the exact image for the context? Recently I designed and basic level architecture where its working good when compared to copali rag. Please check out that and provide any valuable suggestions and feedback on it

@karpathy This is interesting, Thanks for sharing, will check it out.

@karpathy This is a new type of hell, being forced to endlessly learn new things in podcast form.

@karpathy If the images are accurate, youtube learning channels are going to be awesome!

@lukedw888999 @karpathy Good metadata is all you need :)

@karpathy the real question is how to replicate NotebookLM and add more languages

@karpathy Yes, Adding local language support will be next level creativity.

@karpathy I just wish if you can/the listener/the user can interact with them in between to ask doubt. That would be 🤯 super realistic and interactive like asking/talking to NPC in the game and they explaining it customised according to user endlessly.

@karpathy Can you share the code for "used Rag to find the perfect images to sync with the generated audio."?

Sure will do in sometime, meanwhile pipeline is - It matches the metadata of images with the transcript. You can either use Serper ( more search options ) or tag the images (few quality sources as I used ) using a good multimodal LLM (Claude etc) , and a simple vector search will provide the image metadata matching for each chunk ( transcript ) I also edited few images to keep the flow in sync.