Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

We just discovered faster sorting algorithms! Here is a 60-second explanation of what happened: DeepMind turned the sorting problem into a puzzle, where the goal of an agent is to organize a list of instructions to find a correct and efficient algorithm. Think Tetris, but using lines of code....

407,785 Aufrufe • vor 3 Jahren •via X (Twitter)

8 Kommentare

Profilbild von Santiago
Santiagovor 3 Jahren

Here is a link to DeepMind's paper:

Profilbild von Santiago
Santiagovor 3 Jahren

Isn't this a new algorithm?

Profilbild von Santiago
Santiagovor 3 Jahren

Thanks for the link! I forgot to include it.

Profilbild von RedEyed
RedEyedvor 3 Jahren

It should be noted that it's 70% faster on sequences of length less than 5 elements, and 1.7% speedup on sequences more than 250k elements

Profilbild von Santiago
Santiagovor 3 Jahren

Correct.

Profilbild von Santiago
Santiagovor 3 Jahren

Yup, I read that post. I don't feel comfortable mixing their marketing with their technical accomplishments. I understand those who argue they leaned too much into hype. I do not understand those minimizing the importance of what they did.

Profilbild von 𝗝 𝟯 𝟯 𝗣 𝟰 | 𝗷𝟯𝟯𝗽𝟰.𝗲𝘁𝗵
𝗝 𝟯 𝟯 𝗣 𝟰 | 𝗷𝟯𝟯𝗽𝟰.𝗲𝘁𝗵vor 3 Jahren

This is important in terms of what could be achieved in other problems, but this is definitely overhyped by Deepmind. Nvidia is doing stuff like this with hardware optimisation and they don’t try to publish it in Nature. They just do it and make faster chips faster.

Profilbild von Santiago
Santiagovor 3 Jahren

I think we can argue about their marketing, but it's not fair to subtract importance from their work.

Ähnliche Videos

The most interesting part for me is where Andrej Karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.” A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer. > “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.” So what do humans do instead? > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” > “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.” Why can’t we just add this training to LLMs today? > “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.” > “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.” > “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.” How do humans get around model collapse? > “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.” In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by Erik Hoel. I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization? > “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”

Dwarkesh Patel

1,050,387 Aufrufe • vor 8 Monaten

☝️🏾Michelle Obama is right! MAGA is NOT full of racist people; it's full of desperate people. And of course, who was the first to attack her? Black liberals. Why? Because racism feeds the black liberal. If Michelle Obama, the queen of black politics, says that it's not racism, how will the black liberal eat? Without racism, how will the black liberal gain the attention of blacks? How will they maintain their sense of purpose? The black liberal is simply an errand dog for white liberals. Without racism, the black liberal is of NO value to the white liberal and immediately becomes disposable. As you can see, they will even attack black women; Barack's wife is not exempt. Anyone can get it! But Michelle is right. These people aren't racist. They truly are desperate. Public schools and universities did not equip them with survival skills. It taught them how to be a slave (we'll talk about the white slaves of America at a later date). The schools didn't teach them how to think. The schools didn't make them more intelligent. In fact, it gave many whites a false sense of intelligence. School taught them that if you have knowledge, then you are intelligent. But that is only half of the puzzle. Knowledge is just mental masturbation...when you don't have the skill of problem-solving. And when you grow up spoon-fed, there aren't many problems to solve. So, what we have is a population of educated fools who think they are intelligent but are not. And many of them are realizing this now, and their ego is shattered. So, what do they do? They look for the low-hanging fruit. Racism. They know that rage bait can get them the attention that their minds can't. Because they have nothing positive to contribute to society, they must resort to the lowest of tactics to survive. That's why they don't anger me. All I see is a sea of pitiful dolts...like someone who is mentally challenged. There's no reason to have high expectations for the intellectually disabled. What we are witnessing from this group of whites is acts of desperation. Hope is almost gone in their minds. The America they learned to love was only a veneer to mask its dark side. If only they studied black history. They would see the evil that the government is capable of. But no, they were happy because the target was not them. They were happy not to be black. But now...white conservatives are the new niggas. How long will they remain suckled at the teat of Lady Liberty?

Hotep Jesus

13,644 Aufrufe • vor 1 Monat

Bill Maher immediately tried to cast doubt on the Iran deal, claiming he’s “heard it so many times before.” The problem for Maher was that the man leading the negotiations, JD Vance, was sitting right across from him to shut it down in real time. Vance said America wins either way — whether they make the final deal or not. MAHER: “You came out of these meetings, I heard sort of the same thing I’ve heard, a lot of talk about progress and that. I've heard it so many times before. Why is this different, why isn't it bullsh*t this time?” VANCE: “Well...the most important thing Bill is that the people who judge whether the oil is actually flowing, they judge this as success.” “If you look at oil right now, it’s back down to $73 a barrel, got up to $126 a barrel. There's a signal that something real is going on here.” “I think the second, Bill, is whether we make the final deal...then great. If we don't make the final deal, their nuclear program is still destroyed, they are still much weaker as a country so my attitude is, America wins either way.” “But I do think that what the president has done is asked us to do something that frankly nobody in 47 years of dealing with the Iranians has done, which is offer them an opportunity to fundamentally transform how they behave with the West.” “They've been the largest state sponsor of terrorism basically since they began as a nation or at least as an Islamic republic 47 years ago.” “He’s saying, look, if they are willing to change, we are willing to change too. If they are not willing to change, we still fundamentally have all the cards and I think that's a good place for us to be.”

Overton

95,379 Aufrufe • vor 3 Tagen