Loading video...

Video Failed to Load

Go Home

Terence Tao says that we still cannot predict how many weights a model needs, or how long training takes, to model a data set well. It is done empirically, with rules of thumb, because some very basic concepts are missing. Communication has Shannon's theory: information comes in bits, with...

14,653 views • 5 days ago •via X (Twitter)

7 Comments

Swordwaker 7749 (play/acc)'s profile picture
Swordwaker 7749 (play/acc)5 days ago

I mean... he had a point. The scaling law was largely empirical or with limited rigor in mathematical evidence.

Maclain Hunter (Logical Concept)'s profile picture
Maclain Hunter (Logical Concept)5 days ago

Tao is a know nothing hack and you’re a fucking idiot for giving his noise more attention and respect than it deserves. He’s showing why it’s not Asian dudes doing math that actually advance the world but white men, and logicians like me who can prove he knows nothing. Fuck him.

SunStar Systems's profile picture
SunStar Systems5 days ago

@burny_tech He’s an imbecile if he doesn’t know who Yann Lecun is, or has never used an open weight model.

QuietlyDisruptive's profile picture
QuietlyDisruptive5 days ago

"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."-- John von Neumann

Nicholas Perry's profile picture
Nicholas Perry5 days ago

Absolutely no reason to believe this

Adam Barnett's profile picture
Adam Barnett5 days ago

Isn’t that what validation is for. It helps you gauge accuracy.

Ada Wan's profile picture
Ada Wan4 days ago

i. There shouldn't be "AI" when there is already tech and ML. There shouldn't be "AI" at all. The limits of "AI" lie in its lack of definitions, its redundancy, its lack of ethics, and its unlawfulness. ii. When it comes to ML, correct, these are mostly questions empirical in 1/

Related Videos

Terence Tao, UCLA mathematics professor and Fields Medallist, on why AI may be succeeding at science while scientists get worse at it: Tao starts by naming the tension directly: "There is this paradox that on the one hand AIs are becoming more powerful and more capable and making fewer mistakes, and they are ostensibly achieving a lot of the goals that we think scientists are trying to do. They're running experiments. They're analyzing data. They're writing papers." Experiments run, data analyzed, papers written — the exact outputs any university or funding body would cite as proof that science is working. Tao's concern is what those outputs stop telling you: "It may be that it comes at the cost of the AI picks up some skill but no human scientist gets any better at doing the science." The skill still accumulates, just in the wrong place. The system gets more capable while the people operating it stand still, because the work that used to build a researcher is now the work being handed off. And the cost shows up in the one thing scientists are supposed to be able to do: "No human can communicate exactly what just happened and why. This scientific discovery is interesting, why this proof is new and what features it has and how it connects." That final clause is the sharp end of it. Tao is describing a result nobody can place — a proof that arrives with no one able to say what makes it new, how it's built, or how it connects to the rest of the field. Understanding is a separate achievement from getting the answer, and it can quietly disappear while the answers keep arriving on schedule. Which is why he thinks the target itself needs re-examining: "We may have to sort of redesign our conception of what science is and what we actually want out of science. What exactly is science for and what are we trying to do? And is there a danger that we are optimizing the wrong thing when we are pointing our AI tools at science?"

Big Brain AI

42,751 views • 2 months ago

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

56,782 views • 2 months ago

Terence Tao, Professor of Mathematics at UCLA and Fields Medalist, on why nobody can fully explain why LLMs work: Tao starts with the mechanics, which are no mystery at all. You gather an enormous amount of text and you fit a curve to it. "The magic of LLMs is that if you train these LLMs on enough data — so trillions and trillions of data points — and you really try to fit as good a curve as possible, and this takes like millions and millions of dollars of computing power and months and months of time, then suddenly, even when you iterate, it stays coherent. It begins to sound not like monkeys but it actually sounds like a human speaking." That is the entire recipe: data, compute, time, curve-fitting. None of it obviously adds up to fluent English. Then Tao says the part that most people building these systems move past quickly: "And somehow we don't fully understand why that's the case." The admission comes from one of the most capable living mathematicians, and the gap he describes sits at the centre of the field. His best account of what's happening puts the mystery in the language rather than the machine: "What seems to be true is that language, like English or other natural languages, contains a lot of hidden patterns that we're not consciously aware of. I mean, we know some of the laws of English, there's laws of grammar and things, but there are sort of unspoken, unwritten rules of language that humans pick up." It relocates the question: the structure was always latent in the text, and the model found it. Why enough curve-fitting surfaces that structure is still unanswered. Tao reaches for a child to explain it: "A human child, even though they're not taught what a noun is, what a verb is or whatever, they can pick up what order English words go in just by continual exposure to the language." Which is honest about the limits of the explanation, because we can't fully account for how children do it either. From there, the unexplained behaviour compounds. Exposure to language turns out to be enough to produce something that looks like reasoning: "It seems like you can teach these models to also pick up patterns in language to the point where you can give them math questions. The answer to 2 plus 3 is — and they will say five." And once a model handles language at all, you can push it into resembling self-correction: "Once you have a little bit of ability to speak English, you can kind of go in loops and sort of check your work and make fewer mistakes, and you can prompt these models to proceed step by step and not say something unless it's been double checked and so forth. And so they become a little bit smarter, quote unquote, to the point where they can solve many, many complicated tasks." The scare quotes around "smarter" carry the whole argument. Tao does not concede that the unexplained fluency implies anything underneath it: "But they're still just guessing the next word to say. It's not really grounded in any deep understanding of the real world. It's just that they have seen the patterns in the English language or other language that they've absorbed so well."

Big Brain AI

192,177 views • 2 months ago

WATCH CHARLIE'S SHIRT. This is a very interesting video to say the least. Look how far it pulls back behind him and is it moving before anything else does in the frame? What does this erratic movement honestly tell you? Does this lean more of a bullet or projectile of some sort pulling the shirt before or after impact or when it is exiting? Or does this tell you that maybe something exploded and pulled his shirt because of the split second movement before anything else seems to move? What is your honest opinion and theory about this video? The shirt movement is very odd. The 2 most important pieces of information we need to narrow down the speculation is a video with Tyler in the prone position with the rifle taking the shot. And we also need to know if the wound on Charlie's neck is an ENTRANCE or an EXIT, and where the 2ND hole is IF THERE EVEN IS A 2ND HOLE. Candace recently disclosed that Charlie was adding a DOGE type team to TPUSA because "money" was missing or being spent too fast. She said he honestly didn't know where it was going and also where all the money was really even COMING FROM... More data points and more evidence to help us find the truth. FBI Dir. Kash Patel said that they have crucial information that they WILL NOT release yet, and DID NOT rule out a 2ND SHOOTER. They are also looking into 20+ more people involved or associated in some way. We will continue to to dig. The truth will eventually reveal itself. NEVER GIVE UP. EVER. cc: MJTruthUltra

The SCIF

299,051 views • 1 year ago