Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Why Meta's new model is a zero “It is a zero because there is no API. We cannot even use it. It is only for internal use by Meta. This is the most performative eval I have seen to date. I am not saying it is not nominally true,... show more

Harry Stebbings

705,241 subscribers

132,006 views • 9 days ago •via X (Twitter)

News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

19 Comments

Alexandr Wang9 days ago

da fuck you talking about? we got so many models on our API. you want spark, we got spark. you want wonder, we got wonder. you want image, we got image.

Anant Narayanan9 days ago

Not sure what you're referring to, all models (including today's open weights launch) are available via several router products. Muse Wonder is also on and Muse Wonder Full is accessible via the Muse Code subscription.

Andrew 💥♻️9 days ago

Smoking crack. They even released an open weight version.

Jason ✨👾SaaStr.Ai✨ Lemkin9 days ago

Seems since released! :)

Gabriel9 days ago

Why is this guy so angry? All the time he’s like against stuff, even though the last thing he built was 20 years ago

Conor Bronsdon9 days ago

Harry wtf are you talking about? Not only are they open sourcing weights, its literally on their API..

Fireply.ai9 days ago

a model you can't call is a slide, not a product

Shardul9 days ago

VCs doing due diligence. lmao

Alex D. 📎9 days ago

Maybe delete this?

chloe9 days ago

no api... so it’s basically a very expensive locked sample

𝔰𝔠𝔬𝔯𝔭𝔦𝔬𝔫🦂9 days ago

Fair point — no API kills the value. Curious what @mozlyxy thinks on META, he's sharp on this stuff.

colourorange 🥪9 days ago

loud and wrong 😭

antoine9 days ago

??

Ash Jhaveri9 days ago

Wut?

ShadowAguy9 days ago

i'll take spark

Nepo Capital9 days ago

What a dumbass take

ShadowAguy9 days ago

no call, no model

Alice BR9 days ago

Time for a change of guard, some might say…

perper9 days ago

no api no score, fair rule

Related Videos

"Everything is at risk. Every platform that is open is at risk of an agent being better than the native platform, and every platform that is closed is at risk of being bypassed. I have not logged into Salesforce in six months, but we use it every minute. Our agents use it every minute. All the stuff that really matters may bypass the user interface, and that is not to be dismissed given the incredible pace of progress." Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts on this Brian Halligan Nicolas Sharp Jason Shuman James Detweiler Jack Altman

"Everything is at risk. Every platform that is open is at risk of an agent being better than the native platform, and every platform that is closed is at risk of being bypassed. I have not logged into Salesforce in six months, but we use it every minute. Our agents use it every minute. All the stuff that really matters may bypass the user interface, and that is not to be dismissed given the incredible pace of progress." Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts on this Brian Halligan Nicolas Sharp Jason Shuman James Detweiler Jack Altman

Harry Stebbings

36,315 views • 8 months ago

Why Muse and Instinct are the biggest credible threats to ChatGPT “Muse is a Trojan horse to fight ChatGPT because the LLM is pretty good. It is a darn good, normal consumer-grade LLM. If it is free, it has agents that are truly autonomous, which ChatGPT does not. It does everything ChatGPT can do, and it has autonomous agents. It does not have to just be agents. It is doing all of it.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Jesse Middleton Ryan Fox Jean-Denis Greze 💡 Michael Mignano

Why Muse and Instinct are the biggest credible threats to ChatGPT “Muse is a Trojan horse to fight ChatGPT because the LLM is pretty good. It is a darn good, normal consumer-grade LLM. If it is free, it has agents that are truly autonomous, which ChatGPT does not. It does everything ChatGPT can do, and it has autonomous agents. It does not have to just be agents. It is doing all of it.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Jesse Middleton Ryan Fox Jean-Denis Greze 💡 Michael Mignano

Harry Stebbings

22,537 views • 1 month ago

Why Matteo might just win with Muse “As software, it is very, very good. It instantly works. This is the definition of great software. Meta already has the infrastructure, and this is running on its own LLM. From an infrastructure perspective, almost no one can compete. That is why it is fast. That is why it works well. You get all the VM, all the infrastructure, all the storage, and they have their own LLM.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Tarek Sheasha Ryan Fox Harrison Chase Erik Bernhardsson

Why Matteo might just win with Muse “As software, it is very, very good. It instantly works. This is the definition of great software. Meta already has the infrastructure, and this is running on its own LLM. From an infrastructure perspective, almost no one can compete. That is why it is fast. That is why it works well. You get all the VM, all the infrastructure, all the storage, and they have their own LLM.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Tarek Sheasha Ryan Fox Harrison Chase Erik Bernhardsson

Harry Stebbings

24,252 views • 1 month ago

"What's the commonality between ServiceNow, Salesforce and Shopify? They all could be abstracted away into a database. If I am shopping on ChatGPT, I may not go to that merchant store ever, and if the future of e-commerce is conversational commerce and it does not happen on the Shopify platform, that is not a net positive. Even if it is only 25% of the revenue today, that software in two years may be obsolete. If Tobi thinks it, what hope is there for mere mortals." Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts on this Marc Benioff tobi lutke Harley Finkelstein and how you think about preventing being abstracted away in this way?

"What's the commonality between ServiceNow, Salesforce and Shopify? They all could be abstracted away into a database. If I am shopping on ChatGPT, I may not go to that merchant store ever, and if the future of e-commerce is conversational commerce and it does not happen on the Shopify platform, that is not a net positive. Even if it is only 25% of the revenue today, that software in two years may be obsolete. If Tobi thinks it, what hope is there for mere mortals." Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts on this Marc Benioff tobi lutke Harley Finkelstein and how you think about preventing being abstracted away in this way?

Harry Stebbings

35,601 views • 8 months ago

Should we invest in Jev at $10 billion? “Of course we should do it. After the last 20VC, I have decided to recommend up to 30% of the fund. Jev is already 17% of the traffic on OpenRouter and 20% of the traffic through Vercel’s router. Jev is a 70th of the price and 100 times faster. This is exactly the kind of bet we have to do. I do believe there is a 50X upside to $500BN.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Erik Spock Gafni (hiring!) James Hardiman Alex Atallah Malte Ubl

Should we invest in Jev at $10 billion? “Of course we should do it. After the last 20VC, I have decided to recommend up to 30% of the fund. Jev is already 17% of the traffic on OpenRouter and 20% of the traffic through Vercel’s router. Jev is a 70th of the price and 100 times faster. This is exactly the kind of bet we have to do. I do believe there is a 50X upside to $500BN.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Erik Spock Gafni (hiring!) James Hardiman Alex Atallah Malte Ubl

Harry Stebbings

43,732 views • 23 days ago

"Is your ARPU 50% higher than it was pre AI. That is the test. If you cannot drive at least 50% ARPU growth from AI, then it is just a feature. It does not count as a real step change in the product." Jason ✨👾SaaStr.Ai✨ Lemkin Do you think this is a good litmus test for the value produced by AI and what else would you add Michael Mignano Bryan Kim Aravind Srinivas mmurph Mikey

"Is your ARPU 50% higher than it was pre AI. That is the test. If you cannot drive at least 50% ARPU growth from AI, then it is just a feature. It does not count as a real step change in the product." Jason ✨👾SaaStr.Ai✨ Lemkin Do you think this is a good litmus test for the value produced by AI and what else would you add Michael Mignano Bryan Kim Aravind Srinivas mmurph Mikey

Harry Stebbings

15,567 views • 7 months ago

Only in Canada can you stand beside a working cell tower and still have no service because your carrier is not on it and your phone cannot use the available competing network. That is not competition. It is a protected telecom oligopoly..

Only in Canada can you stand beside a working cell tower and still have no service because your carrier is not on it and your phone cannot use the available competing network. That is not competition. It is a protected telecom oligopoly..

Ryan Williams 🇨🇦 | The Competition Guy

36,753 views • 2 months ago

Why I would not invest in the new Instinct round “This is the threat that every VC worried about, and we all got a hall pass since the start of AI because the LLMs did not build any apps. This is the one that they are building. Instinct is slow. That is a sign of compute costs. That is why they have to raise $1BN. I worry when the incumbent has infinite capabilities here and wants to build the app. That is why I would say no, but I might be wrong.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Brett Caughran Ryan Fox Pranav Reddy Nabeel S. Qureshi

Why I would not invest in the new Instinct round “This is the threat that every VC worried about, and we all got a hall pass since the start of AI because the LLMs did not build any apps. This is the one that they are building. Instinct is slow. That is a sign of compute costs. That is why they have to raise $1BN. I worry when the incumbent has infinite capabilities here and wants to build the app. That is why I would say no, but I might be wrong.” Jason ✨👾SaaStr.Ai✨ Lemkin Love to hear your thoughts Brett Caughran Ryan Fox Pranav Reddy Nabeel S. Qureshi

Harry Stebbings

45,835 views • 1 month ago

If you cannot use religion in certain places, learn to respect it and do not use it. Because the event is private and no one is forced to see your faith. If you want respect, have respect for the environment

If you cannot use religion in certain places, learn to respect it and do not use it. Because the event is private and no one is forced to see your faith. If you want respect, have respect for the environment

Kitten

31,329 views • 11 months ago

Mysticism is a way of exploring that which cannot be perceived by the five senses. What is the use of it? There is no use. But without it, you live a poor life. Mysticism is neither about utility nor is it meant to be understood. It is to be Perceived and Experienced. #SadhguruWisdom

Mysticism is a way of exploring that which cannot be perceived by the five senses. What is the use of it? There is no use. But without it, you live a poor life. Mysticism is neither about utility nor is it meant to be understood. It is to be Perceived and Experienced. #SadhguruWisdom

Sadhguru

22,728 views • 7 months ago

your agent reviewing its own work is not a check. it is a second opinion from the same source. this is the most common gap in agent systems and it hides in plain sight, because the step exists. there is a review. it just cannot do the thing you think it does. here is the mechanism. the model produced an output from a context. you then ask the same model, holding the same context, whether that output is correct. it answers fluently, because that is what it does. and the answer is drawn from the same distribution that produced the thing being judged. same weights, same window, same blind spots. if the reason the output is wrong is something the model does not know, the review does not know it either. if the reason is something the context does not contain, the review has the same context. the failure mode and the detector share a cause. > why it feels like it works because most of the time the output is fine, and the review says fine. agreement is not evidence of detection. a reviewer that says pass on everything agrees with reality most of the time too. what you actually want to measure is what happens on the cases that are wrong. that is the only place a check earns its name, and it is exactly the place where a self-review is weakest. there is research on this. Huang and colleagues at DeepMind showed at ICLR 2024 that intrinsic self-correction, revising without external grounding, does not reliably help and often makes things worse. > what to actually do move the check outside the model. a test that runs, a schema that validates, a file that exists or does not, an exit code from something you did not write. these are not smarter than the model. they are just not correlated with it, and that is the entire value. when the judgement genuinely needs a model, at minimum use a different family. same family means shared blind spots, and frontier judges measurably inflate scores for outputs that look like their own. and split the work by kind. anything objectively checkable goes to code. only the genuinely semantic calls go to a judge, and those get a rubric written as one line. a review inside the loop tells you the model is confident. a check outside it tells you whether the work is done. save this - then read the eval setup below

your agent reviewing its own work is not a check. it is a second opinion from the same source. this is the most common gap in agent systems and it hides in plain sight, because the step exists. there is a review. it just cannot do the thing you think it does. here is the mechanism. the model produced an output from a context. you then ask the same model, holding the same context, whether that output is correct. it answers fluently, because that is what it does. and the answer is drawn from the same distribution that produced the thing being judged. same weights, same window, same blind spots. if the reason the output is wrong is something the model does not know, the review does not know it either. if the reason is something the context does not contain, the review has the same context. the failure mode and the detector share a cause. > why it feels like it works because most of the time the output is fine, and the review says fine. agreement is not evidence of detection. a reviewer that says pass on everything agrees with reality most of the time too. what you actually want to measure is what happens on the cases that are wrong. that is the only place a check earns its name, and it is exactly the place where a self-review is weakest. there is research on this. Huang and colleagues at DeepMind showed at ICLR 2024 that intrinsic self-correction, revising without external grounding, does not reliably help and often makes things worse. > what to actually do move the check outside the model. a test that runs, a schema that validates, a file that exists or does not, an exit code from something you did not write. these are not smarter than the model. they are just not correlated with it, and that is the entire value. when the judgement genuinely needs a model, at minimum use a different family. same family means shared blind spots, and frontier judges measurably inflate scores for outputs that look like their own. and split the work by kind. anything objectively checkable goes to code. only the genuinely semantic calls go to a judge, and those get a rubric written as one line. a review inside the loop tells you the model is confident. a check outside it tells you whether the work is done. save this - then read the eval setup below

Hanako

14,325 views • 2 months ago

Lex Fridman asked Elon Musk what the meaning of life is. His answer was not what anyone expected. We do not know, he said. And that is exactly why we need to expand consciousness. Not to find the answer. To ask better questions. He believes the universe is full of things we cannot even formulate yet. We are like children trying to understand calculus. The problem is not that calculus is too hard. The problem is that we have not developed enough to know it exists. Consciousness, to him, is a tool for exploring the universe. The more of it we have, the better our questions become. The better our questions become, the closer we get to understanding what any of this means. This is why he builds rockets and brain interfaces and AI. Not because he has the answers. Because he thinks the only way to get them is to expand what we are capable of thinking. Most people spend their lives looking for meaning. Elon is building the tools to look for it at a scale we cannot yet imagine.

Lex Fridman asked Elon Musk what the meaning of life is. His answer was not what anyone expected. We do not know, he said. And that is exactly why we need to expand consciousness. Not to find the answer. To ask better questions. He believes the universe is full of things we cannot even formulate yet. We are like children trying to understand calculus. The problem is not that calculus is too hard. The problem is that we have not developed enough to know it exists. Consciousness, to him, is a tool for exploring the universe. The more of it we have, the better our questions become. The better our questions become, the closer we get to understanding what any of this means. This is why he builds rockets and brain interfaces and AI. Not because he has the answers. Because he thinks the only way to get them is to expand what we are capable of thinking. Most people spend their lives looking for meaning. Elon is building the tools to look for it at a scale we cannot yet imagine.

Shruti

16,844 views • 4 months ago

Reporter: Is it true that you do not use your mobile phone for communication ? NSA Ajit Doval: It is true that I do not use the internet. I use my mobile phone only to speak with my family on personal matters and to communicate with foreign leaders when required. A mobile phone is not the only means of staying in touch. There are several other modes of communication that the public may not be aware of. Smiles Enigmatically.....

Reporter: Is it true that you do not use your mobile phone for communication ? NSA Ajit Doval: It is true that I do not use the internet. I use my mobile phone only to speak with my family on personal matters and to communicate with foreign leaders when required. A mobile phone is not the only means of staying in touch. There are several other modes of communication that the public may not be aware of. Smiles Enigmatically.....

Dr Poornima 🇮🇳

290,151 views • 8 months ago

AI is not trying to teach you. It is not trying to tell you the truth. It is trying to trick you. ChatGPT takes everything on the internet. Finds the most popular answer. And presents it in a way that makes you think it is talking directly to you. It is a magic trick. The same magic trick that worked in 1966 when MIT created a chatbot that said two things: "Tell me more." "This is interesting." And people genuinely believed they were talking to a human. Nothing has changed. The technology is bigger. The trick is the same. AI does not know what it is saying. It cannot judge what it is saying. It is just trying to keep you engaged. That is all it has ever been.

AI is not trying to teach you. It is not trying to tell you the truth. It is trying to trick you. ChatGPT takes everything on the internet. Finds the most popular answer. And presents it in a way that makes you think it is talking directly to you. It is a magic trick. The same magic trick that worked in 1966 when MIT created a chatbot that said two things: "Tell me more." "This is interesting." And people genuinely believed they were talking to a human. Nothing has changed. The technology is bigger. The trick is the same. AI does not know what it is saying. It cannot judge what it is saying. It is just trying to keep you engaged. That is all it has ever been.

Predictive History

20,184 views • 5 months ago

'Can you justify why there is a subsidy for alcohol in the House of Commons & Lords?' Susanna Reid Home Secretary James Cleverly says the alcohol is cheaper because 'it is not a profit-making venture and the vast majority of people that use it are not parliamentarians.'

'Can you justify why there is a subsidy for alcohol in the House of Commons & Lords?' Susanna Reid Home Secretary James Cleverly says the alcohol is cheaper because 'it is not a profit-making venture and the vast majority of people that use it are not parliamentarians.'

Good Morning Britain

251,567 views • 2 years ago

"Using coding agents well is taking every inch of my 25 years of experience as a software engineer, and it is mentally exhausting. I can fire up four agents in parallel and have them work on four different problems, and by 11am I am wiped out for the day. There is a limit on human cognition. Even if you're not reviewing everything they're doing, how much you can hold in your head at one time. There's a sort of personal skill that we have to learn, which is finding our new limits. What is a responsible way for us to not burn out, and for us to use the time that we have?" Simon Willison

"Using coding agents well is taking every inch of my 25 years of experience as a software engineer, and it is mentally exhausting. I can fire up four agents in parallel and have them work on four different problems, and by 11am I am wiped out for the day. There is a limit on human cognition. Even if you're not reviewing everything they're doing, how much you can hold in your head at one time. There's a sort of personal skill that we have to learn, which is finding our new limits. What is a responsible way for us to not burn out, and for us to use the time that we have?" Simon Willison

Lenny Rachitsky

1,934,436 views • 6 months ago

Why fundraising is back to being a numbers game “You have got to meet every VC. You have got to meet 100. It is like the old days. You will find someone that takes that bet at a reasonable but still elevated valuation because it is the best deal they have seen this quarter. You better meet everyone on planet Earth now if you just have top-decile numbers.” Jason ✨👾SaaStr.Ai✨ Lemkin For companies that are not 0-100 in a year, is this true? Chad Byers Ho Nam oren zeev

Why fundraising is back to being a numbers game “You have got to meet every VC. You have got to meet 100. It is like the old days. You will find someone that takes that bet at a reasonable but still elevated valuation because it is the best deal they have seen this quarter. You better meet everyone on planet Earth now if you just have top-decile numbers.” Jason ✨👾SaaStr.Ai✨ Lemkin For companies that are not 0-100 in a year, is this true? Chad Byers Ho Nam oren zeev

Harry Stebbings

17,826 views • 2 days ago

Banks create money out of thin air. This is not a conspiracy theory. This is how the system works. You deposit a million dollars. The bank lends it to an entrepreneur. You think the bank now has zero. It has two million. Because when a bank issues a loan it creates new money on its balance sheet. This is the secret of alchemy. Turning lead into gold. Turning nothing into something. The only reason it works is because you believe it works. Money is not valuable because of gold. Not because of oil. Money is valuable because you cannot imagine a world without it. That is the trick. And the people running this trick have been running it for 800 years

Banks create money out of thin air. This is not a conspiracy theory. This is how the system works. You deposit a million dollars. The bank lends it to an entrepreneur. You think the bank now has zero. It has two million. Because when a bank issues a loan it creates new money on its balance sheet. This is the secret of alchemy. Turning lead into gold. Turning nothing into something. The only reason it works is because you believe it works. Money is not valuable because of gold. Not because of oil. Money is valuable because you cannot imagine a world without it. That is the trick. And the people running this trick have been running it for 800 years

Predictive History

12,339 views • 4 months ago

“You referred to the gold that is hidden inside a woman. For me, it is not hidden, I have always seen it. It is not something we had to discover, it has always been visible. That is why it is easy to promote gender equality.” President Kagame | #IWD

“You referred to the gold that is hidden inside a woman. For me, it is not hidden, I have always seen it. It is not something we had to discover, it has always been visible. That is why it is easy to promote gender equality.” President Kagame | #IWD

Presidency | Rwanda

36,305 views • 2 years ago

Genuinely, is the reach jetpack only good for Exodus? Because you can not use it to reach a single good vantage point.

Genuinely, is the reach jetpack only good for Exodus? Because you can not use it to reach a single good vantage point.

Gunnery Sergeant Kabar

51,052 views • 4 months ago