正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

🚨 Grant Sanderson DID IT AGAIN Language compressibility is not just a neat math trick: it is the core engine of modern LLMs. Grant's latest video boils Shannon's entropy down to a single, powerful idea: Prediction IS compression. → Predict the next word better, use fewer bits to store... show more

Charly Wargnier

175,228 subscribers

160,113 次观看 • 1 个月前 •via X (Twitter)

教育新闻政治科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

As someone who's been shipping LLMs since the GPT-2 days, this lecture on cross-entropy from a Stanford math grad is the closest thing to an ML PhD qualifying exam I've ever seen released publicly for free. Everyone thinks language models predict the next word. They don't. They compress language. Once you see the math, you can't unsee it. 33 minutes. Bookmark & watch today.

As someone who's been shipping LLMs since the GPT-2 days, this lecture on cross-entropy from a Stanford math grad is the closest thing to an ML PhD qualifying exam I've ever seen released publicly for free. Everyone thinks language models predict the next word. They don't. They compress language. Once you see the math, you can't unsee it. 33 minutes. Bookmark & watch today.

fintex

352,576 次观看 • 26 天前

3Blue1Brown’s new video explains why every LLM is actually a compression machine. everyone describes pre-training as “next token prediction” but that’s just the surface-level objective. in reality it is a means to making the most efficient text compressor. prediction and compression are two sides of the same coin. when you train the model to predict the next token you’re not just teaching it to guess the next word but how to best encode the human knowledge it sees. better compression means better abstraction means better reasoning at some point, compression stops looking like storage or a database (as some like to call it on X) and looks like an approximation of understanding.

3Blue1Brown’s new video explains why every LLM is actually a compression machine. everyone describes pre-training as “next token prediction” but that’s just the surface-level objective. in reality it is a means to making the most efficient text compressor. prediction and compression are two sides of the same coin. when you train the model to predict the next token you’re not just teaching it to guess the next word but how to best encode the human knowledge it sees. better compression means better abstraction means better reasoning at some point, compression stops looking like storage or a database (as some like to call it on X) and looks like an approximation of understanding.

ℏεsam

120,027 次观看 • 2 个月前

Is entropy subjective? Entropy is core to the laws of thermodynamics and the arrow of time. And yet, some physicists argue that entropy actually depends on who is measuring it. Could it really be the case that such a fundamental aspect of the laws of physics is subjective?! I addressed this question in my talk at the Rationalist Festival (Rat Fest) 2025, organised by Conjecture Institute. Watch here or on the YouTube link below!

Is entropy subjective? Entropy is core to the laws of thermodynamics and the arrow of time. And yet, some physicists argue that entropy actually depends on who is measuring it. Could it really be the case that such a fundamental aspect of the laws of physics is subjective?! I addressed this question in my talk at the Rationalist Festival (Rat Fest) 2025, organised by Conjecture Institute. Watch here or on the YouTube link below!

Dr Maria Violaris

54,834 次观看 • 6 个月前

In 1948, Claude Shannon invented the math behind every LLM you use today. He tested it by making his wife guess the next letter in a book. A Stanford-trained mathematician just released a 32-minute walkthrough of this exact history - and why the "next-token prediction" story of GPT-5 is actually wrong. Bookmark & watch this weekend. The alternative is a graduate info-theory course + 3 semesters of your life.

In 1948, Claude Shannon invented the math behind every LLM you use today. He tested it by making his wife guess the next letter in a book. A Stanford-trained mathematician just released a 32-minute walkthrough of this exact history - and why the "next-token prediction" story of GPT-5 is actually wrong. Bookmark & watch this weekend. The alternative is a graduate info-theory course + 3 semesters of your life.

fintex

285,466 次观看 • 24 天前

Ilya Sutskever, co-founder and chief scientist of OpenAI: "The better a neural network can predict the next word, the more it understands it." Everyone calls it autocomplete. Ilya explained why that is a comforting lie. To guess the final word of a murder mystery, the name of the killer, the model has to reason through every clue in the book. Prediction that good is not statistics. It is understanding. Then he said the part that should stop you. The text these models train on is "a projection of the world," and to compress it they learn more and more about people, "the human condition, their hopes, dreams, and motivations." The machine is not memorizing our words. It is quietly building a model of us. That is the whole premise of the fear. The machine will never hate you. It understands you far better than that. The frightening variable was never the intelligence. It was the aim. I wrote the full story: the one fear about AI everyone gets wrong, and the one already here.

Ilya Sutskever, co-founder and chief scientist of OpenAI: "The better a neural network can predict the next word, the more it understands it." Everyone calls it autocomplete. Ilya explained why that is a comforting lie. To guess the final word of a murder mystery, the name of the killer, the model has to reason through every clue in the book. Prediction that good is not statistics. It is understanding. Then he said the part that should stop you. The text these models train on is "a projection of the world," and to compress it they learn more and more about people, "the human condition, their hopes, dreams, and motivations." The machine is not memorizing our words. It is quietly building a model of us. That is the whole premise of the fear. The machine will never hate you. It understands you far better than that. The frightening variable was never the intelligence. It was the aim. I wrote the full story: the one fear about AI everyone gets wrong, and the one already here.

Casper

91,838 次观看 • 25 天前

🚨 | Lewis Hamilton on Mercedes’ compression ratio: “What’s clear is that they didn’t show the engine power through any of the practice because of the whole talk on the compression issue. They’ve done a really solid job with their engine, which we have as well but I’m trying to understand why it's 2 tenths or more just through power per sector... And so, if it is the compression thing, I wanna understand why the FIA haven't done anything [or] what’s being done to rectify it.”

🚨 | Lewis Hamilton on Mercedes’ compression ratio: “What’s clear is that they didn’t show the engine power through any of the practice because of the whole talk on the compression issue. They’ve done a really solid job with their engine, which we have as well but I’m trying to understand why it's 2 tenths or more just through power per sector... And so, if it is the compression thing, I wanna understand why the FIA haven't done anything [or] what’s being done to rectify it.”

deni

960,177 次观看 • 5 个月前

This 1-hour tutorial on building a Football AI is the foundation traders use to build their own sports prediction systems one of them made $504K in 1 month using this exact approach Python + YOLO + OpenCV Detection, tracking, team classification from scratch Same stack powers the prediction systems in the article below Bookmark this & give it 1 hour today

This 1-hour tutorial on building a Football AI is the foundation traders use to build their own sports prediction systems one of them made $504K in 1 month using this exact approach Python + YOLO + OpenCV Detection, tracking, team classification from scratch Same stack powers the prediction systems in the article below Bookmark this & give it 1 hour today

Paone

200,600 次观看 • 3 个月前

"It all comes down to compute." Dmitri Alperovitch from Silverado Policy Accelerator explains: "The single most important input to winning is compute the processing power used to train and run AI models. Let me say that plainly, because it is defining that everything that congress and this committee do on AI policy. The binding constraint in this competition is not talent, it is not data, and it is not cash. Both the United States and China have talented researchers and financial resources, and we have learned that frontier models trained on generic data routinely outperform smaller models, trained on more specialized inputs. It all comes down to compute."

"It all comes down to compute." Dmitri Alperovitch from Silverado Policy Accelerator explains: "The single most important input to winning is compute the processing power used to train and run AI models. Let me say that plainly, because it is defining that everything that congress and this committee do on AI policy. The binding constraint in this competition is not talent, it is not data, and it is not cash. Both the United States and China have talented researchers and financial resources, and we have learned that frontier models trained on generic data routinely outperform smaller models, trained on more specialized inputs. It all comes down to compute."

Select Committee on China

41,578 次观看 • 4 个月前

This 1-hour MIT Probability Theory lecture is the mathematical foundation behind the 3 formulas top traders use > Law of Large Numbers - a tiny edge compounds into profit over enough trades > log-normal distribution - why modelling markets with normal distribution fails every time > Shannon entropy - how to detect insider information before it moves the price 87% of wallets on prediction markets lose the ones who win understand this math watch this

This 1-hour MIT Probability Theory lecture is the mathematical foundation behind the 3 formulas top traders use > Law of Large Numbers - a tiny edge compounds into profit over enough trades > log-normal distribution - why modelling markets with normal distribution fails every time > Shannon entropy - how to detect insider information before it moves the price 87% of wallets on prediction markets lose the ones who win understand this math watch this

Mr. Buzzoni

38,237 次观看 • 3 个月前

Jensen Huang says agentic AI is not a chatbot. It is a "digital robot" -- and the next bottleneck is not the model. It is first-person data. "Agentic AI is just a digital robot. It understands, it reasons, it plans, and it acts and uses tools." Then he makes the bigger jump: "The world's video data is from a third person, not first person. And so agentic systems, robotic systems, physical AI -- the data is the hardest problem." That is the hidden constraint in robotics. LLMs got the internet. Physical AI needs demonstrations, synthetic worlds, reinforcement learning, and first-person experience. This is why Jensen keeps calling it a stack: robots are not waiting for one magical model. They need the data engine, the simulator, the reward system, the runtime, and the computer. The next wave of AI is not just software that talks. It is software that sees, plans, acts, and eventually moves through the world.

Jensen Huang says agentic AI is not a chatbot. It is a "digital robot" -- and the next bottleneck is not the model. It is first-person data. "Agentic AI is just a digital robot. It understands, it reasons, it plans, and it acts and uses tools." Then he makes the bigger jump: "The world's video data is from a third person, not first person. And so agentic systems, robotic systems, physical AI -- the data is the hardest problem." That is the hidden constraint in robotics. LLMs got the internet. Physical AI needs demonstrations, synthetic worlds, reinforcement learning, and first-person experience. This is why Jensen keeps calling it a stack: robots are not waiting for one magical model. They need the data engine, the simulator, the reward system, the runtime, and the computer. The next wave of AI is not just software that talks. It is software that sees, plans, acts, and eventually moves through the world.

Karl Mehta

14,140 次观看 • 1 个月前

Alan Oppenheim, MIT professor: "Jump Trading pays quants $650K to pull one real signal out of market noise. I wrote the textbook on that exact math and taught it for free." this free MIT lecture is the math the whole "quant edge" is built on, and almost nobody selling you a course will name it. the man teaching it wrote the signal-processing textbook every engineer on earth learned from, and he put the lectures online for nothing. here is the idea. a price chart is a signal buried in noise. most of what you see is random static, and hidden inside is a faint, repeating pattern with real mathematics for pulling it out. that is not a metaphor. jim simons was a codebreaker who did exactly this to enemy transmissions, then pointed the same math at the market and made a hundred billion dollars. none of it is new. oppenheim taught this in 1987, the math is older, and every lecture is free. same story i told in the article above. the edge was never a secret formula. it was knowing a market is a noisy signal, and having the tools to read it. here is what the gurus skip. the math to separate signal from noise is free and everyone has it. the edge is cleaner data than the next desk, and the discipline to trust a faint signal across thousands of trades instead of chasing the static. the filter is free. the patience to use it is not.

Alan Oppenheim, MIT professor: "Jump Trading pays quants $650K to pull one real signal out of market noise. I wrote the textbook on that exact math and taught it for free." this free MIT lecture is the math the whole "quant edge" is built on, and almost nobody selling you a course will name it. the man teaching it wrote the signal-processing textbook every engineer on earth learned from, and he put the lectures online for nothing. here is the idea. a price chart is a signal buried in noise. most of what you see is random static, and hidden inside is a faint, repeating pattern with real mathematics for pulling it out. that is not a metaphor. jim simons was a codebreaker who did exactly this to enemy transmissions, then pointed the same math at the market and made a hundred billion dollars. none of it is new. oppenheim taught this in 1987, the math is older, and every lecture is free. same story i told in the article above. the edge was never a secret formula. it was knowing a market is a noisy signal, and having the tools to read it. here is what the gurus skip. the math to separate signal from noise is free and everyone has it. the edge is cleaner data than the next desk, and the discipline to trust a faint signal across thousands of trades instead of chasing the static. the filter is free. the patience to use it is not.

Rossst.03

281,905 次观看 • 1 个月前

The Bible is the one book that dares to predict the future. Not once or twice, but hundreds times with 100% accuracy. That is because it is the very word of God. All that is happening with Israel and the Middle East right now? It was all spoken of thousands of years ago by the Hebrew Prophets. Check out this video I just did for a Biblical perspective on the headlines of today.

The Bible is the one book that dares to predict the future. Not once or twice, but hundreds times with 100% accuracy. That is because it is the very word of God. All that is happening with Israel and the Middle East right now? It was all spoken of thousands of years ago by the Hebrew Prophets. Check out this video I just did for a Biblical perspective on the headlines of today.

Greg Laurie

158,454 次观看 • 2 年前

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.

Aakash Gupta

508,254 次观看 • 3 个月前

Iran is not just another geopolitical rival. The regime in Tehran was built on a THEOLOGICAL revolution designed to export its ideology across the globe. It is EVIL convinced it is righteousness. The NYC bombing attempt was just the latest example of this.

Iran is not just another geopolitical rival. The regime in Tehran was built on a THEOLOGICAL revolution designed to export its ideology across the globe. It is EVIL convinced it is righteousness. The NYC bombing attempt was just the latest example of this.

Glenn Beck

85,999 次观看 • 5 个月前

Yann LeCun just told the most well-funded industry in human history it is solving the wrong problem. LeCun: “Babies learn this around the age of eight or nine months, that objects don’t float, they fall.” No dataset. No labels. No reward signal. A nine month old drops a spoon and builds a physics engine no machine can match. LeCun: “Most of us can learn to drive in about 20 or 30 hours of training without ever crashing, causing any accident.” Twenty hours. Tesla has built the most capable driving system on the road. It took billions of miles of data to get there. A sixteen year old gets there over a long weekend. Not because the teenager is the better driver. Because the teenager is not learning to drive. They are deploying a model of reality they have been building since birth. LeCun: “If we drive next to a cliff, we know that if we turn the wheel to the right, the car is going to run off the cliff and nothing good is going to come out of this.” You simulate the crash. You see the wreckage. You feel the fall. You turn the wheel. None of it was real. All of it was intelligence. Every AI has to crash a thousand times to learn what you imagined once and never did. That is not a performance gap. That is an architecture gap. LeCun: “The main problem we need to solve is how do we learn models of the world.” Not bigger models. Not more compute. Not another trillion tokens. World models. A machine that can run reality forward before it acts. The industry is scaling language. LeCun says language is a compression of thought. Not thought itself. You understood gravity before you could say the word. You grasped cause and effect before your first sentence. The deepest intelligence you will ever possess was built in total silence. And every lab on Earth is trying to reconstruct the mind from words alone. Physics does not care about your context window. A baby who learns that cups fall in a kitchen already knows that rocks fall off cliffs. No retraining. No fine-tuning. One model. Every environment. That is what intelligence actually is. Not prediction. Not pattern matching. Not scale. A simulation of reality so precise you rehearse the future before it exists. Every infant on Earth builds one. No machine ever has.

Yann LeCun just told the most well-funded industry in human history it is solving the wrong problem. LeCun: “Babies learn this around the age of eight or nine months, that objects don’t float, they fall.” No dataset. No labels. No reward signal. A nine month old drops a spoon and builds a physics engine no machine can match. LeCun: “Most of us can learn to drive in about 20 or 30 hours of training without ever crashing, causing any accident.” Twenty hours. Tesla has built the most capable driving system on the road. It took billions of miles of data to get there. A sixteen year old gets there over a long weekend. Not because the teenager is the better driver. Because the teenager is not learning to drive. They are deploying a model of reality they have been building since birth. LeCun: “If we drive next to a cliff, we know that if we turn the wheel to the right, the car is going to run off the cliff and nothing good is going to come out of this.” You simulate the crash. You see the wreckage. You feel the fall. You turn the wheel. None of it was real. All of it was intelligence. Every AI has to crash a thousand times to learn what you imagined once and never did. That is not a performance gap. That is an architecture gap. LeCun: “The main problem we need to solve is how do we learn models of the world.” Not bigger models. Not more compute. Not another trillion tokens. World models. A machine that can run reality forward before it acts. The industry is scaling language. LeCun says language is a compression of thought. Not thought itself. You understood gravity before you could say the word. You grasped cause and effect before your first sentence. The deepest intelligence you will ever possess was built in total silence. And every lab on Earth is trying to reconstruct the mind from words alone. Physics does not care about your context window. A baby who learns that cups fall in a kitchen already knows that rocks fall off cliffs. No retraining. No fine-tuning. One model. Every environment. That is what intelligence actually is. Not prediction. Not pattern matching. Not scale. A simulation of reality so precise you rehearse the future before it exists. Every infant on Earth builds one. No machine ever has.

Dustin

121,810 次观看 • 1 个月前

🇺🇸KJP: BIDEN DID NOT USE FEMA FUNDS FOR MIGRANTS, TRUMP DID “I mean, it's just categorically false. It is not true. It is a false statement. And, look, the fact of the matter is, I think Washington Post Fact Checker, did a piece and the headline recently, just moments ago, not too long ago, and the headline was ‘No Biden did not take FEMA relief money to use on migrants but Trump did. I'll leave it there.” Source: The Post Millennial

🇺🇸KJP: BIDEN DID NOT USE FEMA FUNDS FOR MIGRANTS, TRUMP DID “I mean, it's just categorically false. It is not true. It is a false statement. And, look, the fact of the matter is, I think Washington Post Fact Checker, did a piece and the headline recently, just moments ago, not too long ago, and the headline was ‘No Biden did not take FEMA relief money to use on migrants but Trump did. I'll leave it there.” Source: The Post Millennial

Mario Nawfal

670,402 次观看 • 1 年前

🚨| Sir Lewis Hamilton on Mercedes' engine: "What's clear is that they didn't show the engine power through any of the practice, because there's the whole talk of compression ratio, and they've obviously done a really solid job with their engine, which we have as well, but it would be interesting to... I want to understand why it's like two-tenths or more just through power, per sector. And so, if it is a compression thing, I want to understand why the FIA haven't done anything, what's being done to rectify it, but if it's not, and it's just pure [ped], then we have to do a better job." #F1 || #AustralianGP

🚨| Sir Lewis Hamilton on Mercedes' engine: "What's clear is that they didn't show the engine power through any of the practice, because there's the whole talk of compression ratio, and they've obviously done a really solid job with their engine, which we have as well, but it would be interesting to... I want to understand why it's like two-tenths or more just through power, per sector. And so, if it is a compression thing, I want to understand why the FIA haven't done anything, what's being done to rectify it, but if it's not, and it's just pure [ped], then we have to do a better job." #F1 || #AustralianGP

sim

38,390 次观看 • 5 个月前

Professor Marcos López de Prado at Cornell - the man Shannon and Thorp's framework eventually became He personally managed $13 billion at Guggenheim Partners with an audited risk-adjusted return of 2.3 - institutional Sharpe-equivalent that less than 1% of fund managers ever hit Then became the first head of machine learning at AQR Capital, a $226B fund. Then went to Cornell to teach this Shannon used information theory to beat Buffett. Thorp used it to beat Vegas. López de Prado uses it to manage billions for institutions today The article above is that exact same lineage applied to Polymarket - KL-divergence, max-entropy, entropy collapse, three tools you can use today 1 hour from one of three people on Earth qualified to teach this ↓

Professor Marcos López de Prado at Cornell - the man Shannon and Thorp's framework eventually became He personally managed $13 billion at Guggenheim Partners with an audited risk-adjusted return of 2.3 - institutional Sharpe-equivalent that less than 1% of fund managers ever hit Then became the first head of machine learning at AQR Capital, a $226B fund. Then went to Cornell to teach this Shannon used information theory to beat Buffett. Thorp used it to beat Vegas. López de Prado uses it to manage billions for institutions today The article above is that exact same lineage applied to Polymarket - KL-divergence, max-entropy, entropy collapse, three tools you can use today 1 hour from one of three people on Earth qualified to teach this ↓

0xDipper

111,860 次观看 • 3 个月前

Oligarchy: a word I've been told not to use because it's too confusing. I looked it up, and it's just a description of what the hell is going on.

Oligarchy: a word I've been told not to use because it's too confusing. I looked it up, and it's just a description of what the hell is going on.

Graham Platner for Senate

46,879 次观看 • 9 个月前

CNN: He claimed that China builds a lot of wind turbines and manufactures them for others, but refuses to use it itself, barely uses wind power. In fact, China is the world leader in the use of wind power. It is building additional wind power in China far faster than the pace at which the U.S. Is building in the United States itself. So the idea that China is just, you know, foisting this terrible source of energy on other countries while refusing to use it is a reversal of reality.

CNN: He claimed that China builds a lot of wind turbines and manufactures them for others, but refuses to use it itself, barely uses wind power. In fact, China is the world leader in the use of wind power. It is building additional wind power in China far faster than the pace at which the U.S. Is building in the United States itself. So the idea that China is just, you know, foisting this terrible source of energy on other countries while refusing to use it is a reversal of reality.

Acyn

213,926 次观看 • 10 个月前