Loading video...

Video Failed to Load

Go Home

We just discovered faster sorting algorithms! Here is a 60-second explanation of what happened: DeepMind turned the sorting problem into a puzzle, where the goal of an agent is to organize a list of instructions to find a correct and efficient algorithm. Think Tetris, but using lines of code....

407,783 views • 3 years ago •via X (Twitter)

8 Comments

Santiago's profile picture
Santiago3 years ago

Here is a link to DeepMind's paper:

Santiago's profile picture
Santiago3 years ago

Isn't this a new algorithm?

Santiago's profile picture
Santiago3 years ago

Thanks for the link! I forgot to include it.

RedEyed's profile picture
RedEyed3 years ago

It should be noted that it's 70% faster on sequences of length less than 5 elements, and 1.7% speedup on sequences more than 250k elements

Santiago's profile picture
Santiago3 years ago

Correct.

Santiago's profile picture
Santiago3 years ago

Yup, I read that post. I don't feel comfortable mixing their marketing with their technical accomplishments. I understand those who argue they leaned too much into hype. I do not understand those minimizing the importance of what they did.

𝗝 𝟯 𝟯 𝗣 𝟰 | 𝗷𝟯𝟯𝗽𝟰.𝗲𝘁𝗵's profile picture
𝗝 𝟯 𝟯 𝗣 𝟰 | 𝗷𝟯𝟯𝗽𝟰.𝗲𝘁𝗵3 years ago

This is important in terms of what could be achieved in other problems, but this is definitely overhyped by Deepmind. Nvidia is doing stuff like this with hardware optimisation and they don’t try to publish it in Nature. They just do it and make faster chips faster.

Santiago's profile picture
Santiago3 years ago

I think we can argue about their marketing, but it's not fair to subtract importance from their work.

Related Videos

The most interesting part for me is where Andrej Karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.” A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer. > “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.” So what do humans do instead? > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” > “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.” Why can’t we just add this training to LLMs today? > “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.” > “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.” > “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.” How do humans get around model collapse? > “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.” In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by Erik Hoel. I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization? > “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”

Dwarkesh Patel

1,050,069 views • 8 months ago

☝️🏾Michelle Obama is right! MAGA is NOT full of racist people; it's full of desperate people. And of course, who was the first to attack her? Black liberals. Why? Because racism feeds the black liberal. If Michelle Obama, the queen of black politics, says that it's not racism, how will the black liberal eat? Without racism, how will the black liberal gain the attention of blacks? How will they maintain their sense of purpose? The black liberal is simply an errand dog for white liberals. Without racism, the black liberal is of NO value to the white liberal and immediately becomes disposable. As you can see, they will even attack black women; Barack's wife is not exempt. Anyone can get it! But Michelle is right. These people aren't racist. They truly are desperate. Public schools and universities did not equip them with survival skills. It taught them how to be a slave (we'll talk about the white slaves of America at a later date). The schools didn't teach them how to think. The schools didn't make them more intelligent. In fact, it gave many whites a false sense of intelligence. School taught them that if you have knowledge, then you are intelligent. But that is only half of the puzzle. Knowledge is just mental masturbation...when you don't have the skill of problem-solving. And when you grow up spoon-fed, there aren't many problems to solve. So, what we have is a population of educated fools who think they are intelligent but are not. And many of them are realizing this now, and their ego is shattered. So, what do they do? They look for the low-hanging fruit. Racism. They know that rage bait can get them the attention that their minds can't. Because they have nothing positive to contribute to society, they must resort to the lowest of tactics to survive. That's why they don't anger me. All I see is a sea of pitiful dolts...like someone who is mentally challenged. There's no reason to have high expectations for the intellectually disabled. What we are witnessing from this group of whites is acts of desperation. Hope is almost gone in their minds. The America they learned to love was only a veneer to mask its dark side. If only they studied black history. They would see the evil that the government is capable of. But no, they were happy because the target was not them. They were happy not to be black. But now...white conservatives are the new niggas. How long will they remain suckled at the teat of Lady Liberty?

Hotep Jesus

13,644 views • 26 days ago