Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Vals AI CEO Rayan Krishnan on why defensible evals are critical for enterprises' survival: "The labs side of it is very clear. If you're raising lots of money, investing heavily in building models, it's essential for you to show why your model is getting better and why the customer...

a16z's profile picture

a16z

1,075,411 subscribers

34,017 Aufrufe • vor 2 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

12 Kommentare

Profilbild von Arnold
Arnoldvor 2 Tagen

A firm is its evals is a sharp framing. The tension is that the most defensible evals are often too proprietary to become trusted external proof.

Profilbild von Lynn Welch
Lynn Welchvor 2 Tagen

The independent testing layer can't come fast enough. In diligence rooms I still get handed the vendor's own benchmark deck as proof the model works, and nobody asks who ran the test. Who pays the evaluator decides whether any of it holds.

Profilbild von SAI
SAIvor 2 Tagen

Thanks for making it so that the regular investor cant invest

Profilbild von icefrog.◎
icefrog.◎vor 2 Tagen

token spend vs salary is wild. audit it or die

Profilbild von asa
asavor 2 Tagen

Recording the first sign that my range estimate was off gave me something concrete to work on in reviews.

Profilbild von sophs b
sophs bvor 2 Tagen

Defensible evals are a nice story, but enterprise survival depends on actual deployment, not benchmarks. What’s your threshold for trusting an eval over real-world results?

Profilbild von Weaver Labs
Weaver Labsvor 2 Tagen

Absolutely. Once AI becomes a meaningful operating cost, being able to measure performance and ROI becomes essential for every enterprise.

Profilbild von Saminu suleiman
Saminu suleimanvor 2 Tagen

A habit that stayed with me: writing a reason before returning to an old idea.

Profilbild von Apoorv Agarwal
Apoorv Agarwalvor 2 Tagen

The problem with most companies is their workflows are not stable. AI produces different verdict every time. Companies are focusing of accuracy but not stability. Just throwing bigger and bigger models on your workflows won't help. Developers have to build pass^k and pass@k evals effectively.

Profilbild von AI Mastery Guide
AI Mastery Guidevor 2 Tagen

evals really are the moat now

Profilbild von Modelplane
Modelplanevor 2 Tagen

The credit-ratings analogy is the part worth stressing: raters got paid by issuers and drifted. If eval vendors are paid by the labs they grade, the same conflict shows up, just faster. Independent funding is the whole ballgame.

Profilbild von Rosetina Degen 🚢🎒
Rosetina Degen 🚢🎒vor 2 Tagen

Defensible evals are crucial for survival

Ähnliche Videos

Vals AI co-founder and CEO Rayan Krishnan with a16z's Ben Horowitz and Jennifer Li on grading AI, what it costs, and who gets to make the rules: Every big industry eventually grows an independent testing layer. AI has credit ratings to learn from and Enron to avoid. Model capability today is still mostly self-reported. As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations. The harder problem is geopolitical. Reagan's "trust but verify" worked during the Cold War because you could fly over and count the missiles. No simple equivalent for AI models exists. In this conversation with Erik Torenberg, they get into how you measure a model's ability to improve itself, why every good benchmark eventually has to be retired, and what happens when token spend begins to rival employee salaries. 00:00 Intro 02:20 Llama 4 on public vs private benchmarks 05:24 Nobody agreed how to test humans either 06:55 What movie ratings teach us about AI 08:55 The Enron problem in benchmarking 11:36 Why a good benchmark has to be retired 13:22 Evals that run for weeks, not seconds 16:20 Where the real workday starts at 4pm 18:08 A firm really is just its evals 20:35 Why Sonnet can cost more than Opus 22:42 One engineer, 6 billion tokens in a day 25:05 Who should set the rules for models 28:55 Public sector enforces, private verifies 33:32 Why sovereign AI is inefficient and happening anyway 35:00 The AI version of trust-but-verify 37:15 Where cyber evals have to go next YouTube: Rayan Krishnan Vals AI benahorowitz.eth Jennifer Li Erik Torenberg
39:14

Vals AI co-founder and CEO Rayan Krishnan with a16z's Ben Horowitz and Jennifer Li on grading AI, what it costs, and who gets to make the rules: Every big industry eventually grows an independent testing layer. AI has credit ratings to learn from and Enron to avoid. Model capability today is still mostly self-reported. As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations. The harder problem is geopolitical. Reagan's "trust but verify" worked during the Cold War because you could fly over and count the missiles. No simple equivalent for AI models exists. In this conversation with Erik Torenberg, they get into how you measure a model's ability to improve itself, why every good benchmark eventually has to be retired, and what happens when token spend begins to rival employee salaries. 00:00 Intro 02:20 Llama 4 on public vs private benchmarks 05:24 Nobody agreed how to test humans either 06:55 What movie ratings teach us about AI 08:55 The Enron problem in benchmarking 11:36 Why a good benchmark has to be retired 13:22 Evals that run for weeks, not seconds 16:20 Where the real workday starts at 4pm 18:08 A firm really is just its evals 20:35 Why Sonnet can cost more than Opus 22:42 One engineer, 6 billion tokens in a day 25:05 Who should set the rules for models 28:55 Public sector enforces, private verifies 33:32 Why sovereign AI is inefficient and happening anyway 35:00 The AI version of trust-but-verify 37:15 Where cyber evals have to go next YouTube: Rayan Krishnan Vals AI benahorowitz.eth Jennifer Li Erik Torenberg

a16z

154,318 Aufrufe • vor 3 Tagen

Scott on why conviction in an idea is dangerous as an investor but required as a founder: "From the investor seat, I don't think you want to be in love with an idea. That's a risky thing because you'll find a way to express that by investing and make some compromises on the team because you know this idea is just so good, its time has come. But if the team isn't there, 90% of the time it doesn't work. Then there's a whole thing can you swap out the team? But the whole Founders Fund thesis was always no, you need the founder to run the company. But on the company side, you have to be in love with the idea. It's not that rational to start a company. There's a lot easier, more comfortable ways to make money if that's the goal. So it's got to be about what the company is doing specifically. If you want to actually start a company, it better be something that you're really passionate about, or you think that the problem is really important."
1:15

Scott on why conviction in an idea is dangerous as an investor but required as a founder: "From the investor seat, I don't think you want to be in love with an idea. That's a risky thing because you'll find a way to express that by investing and make some compromises on the team because you know this idea is just so good, its time has come. But if the team isn't there, 90% of the time it doesn't work. Then there's a whole thing can you swap out the team? But the whole Founders Fund thesis was always no, you need the founder to run the company. But on the company side, you have to be in love with the idea. It's not that rational to start a company. There's a lot easier, more comfortable ways to make money if that's the goal. So it's got to be about what the company is doing specifically. If you want to actually start a company, it better be something that you're really passionate about, or you think that the problem is really important."

Patrick OShaughnessy

33,016 Aufrufe • vor 5 Monaten

Vals AI CEO Rayan Krishnan on why sovereign AI makes Cold War-style verification necessary: "From my very idealistic perspective, I'm surprised to see so much investment in sovereign AI." "You could probably consolidate a lot of these efforts. But it seems like that's not the world we're in or the one we're headed towards. There are actually increased efforts to build AI in a sovereign way." "So that takes having a shared language to communicate about what the framework for evaluations is, and where we're going to collectively align around the risks." "There's a lot to learn from nuclear here. Reagan had this line, 'Trust but verify.' We're starting to see signs of trust in that Xi Jinping and Trump are going to be meeting... But there is no clear way to actually do the verification part of this." "Having the shared language of evals will allow us to say things like, you have the right number of nuclear warheads. In that example, there were also flyovers, a mechanism by which a country could audit another country's nuclear stockpile." "If there's concern about the societal or even existential risk of AI, it will necessitate us constructing this shared language of evaluations to do the verification process." Rayan Krishnan Jennifer Li
2:12

Vals AI CEO Rayan Krishnan on why sovereign AI makes Cold War-style verification necessary: "From my very idealistic perspective, I'm surprised to see so much investment in sovereign AI." "You could probably consolidate a lot of these efforts. But it seems like that's not the world we're in or the one we're headed towards. There are actually increased efforts to build AI in a sovereign way." "So that takes having a shared language to communicate about what the framework for evaluations is, and where we're going to collectively align around the risks." "There's a lot to learn from nuclear here. Reagan had this line, 'Trust but verify.' We're starting to see signs of trust in that Xi Jinping and Trump are going to be meeting... But there is no clear way to actually do the verification part of this." "Having the shared language of evals will allow us to say things like, you have the right number of nuclear warheads. In that example, there were also flyovers, a mechanism by which a country could audit another country's nuclear stockpile." "If there's concern about the societal or even existential risk of AI, it will necessitate us constructing this shared language of evaluations to do the verification process." Rayan Krishnan Jennifer Li

a16z

34,808 Aufrufe • vor 2 Tagen

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." Rayan Krishnan Jennifer Li
1:02

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." Rayan Krishnan Jennifer Li

a16z

34,164 Aufrufe • vor 2 Tagen

$GOOG's Ruth Porat says the biggest risk in the AI buildout is the ROI question, and focusing on short-term tactics at the cost of building a great company. "I think the biggest risk for everyone is when the question is asked, which it often is, how do you get to an ROI? That's going to force our teams to do something short-term and tactical. You can always do a cost cut to get something to drop to the bottom line, and that is not a way to build a great company." "So I think the risk is don't be so in the moment maniacally focused on what is it, but have a right mix of an engine that's continuing to drive the business, which also explains why you need to start building early for growth so that you can grow into it." ____ Follow Fireside Alpha for more daily highlights of key business and technology conversations.
0:32

$GOOG's Ruth Porat says the biggest risk in the AI buildout is the ROI question, and focusing on short-term tactics at the cost of building a great company. "I think the biggest risk for everyone is when the question is asked, which it often is, how do you get to an ROI? That's going to force our teams to do something short-term and tactical. You can always do a cost cut to get something to drop to the bottom line, and that is not a way to build a great company." "So I think the risk is don't be so in the moment maniacally focused on what is it, but have a right mix of an engine that's continuing to drive the business, which also explains why you need to start building early for growth so that you can grow into it." ____ Follow Fireside Alpha for more daily highlights of key business and technology conversations.

Fireside Alpha

38,443 Aufrufe • vor 1 Monat

I love this idea from Jason Fried "Your only competition is your costs." Keep costs low, keep the team small, make stuff you want to use. You don't need the whole world: “A business is very simple. You got to make more than you spend. If you're making more than you spend, then your competition is your cost. That's what you're really in business against, how much it costs you to stay in business. It's not all the other alternatives that are on the market. You can't control what they're going to put out there, what they're going to price it at, all the things they're going to do. They're going to do what they're going to do. What I can control is how much it costs me to run my business, how much I sell my product for, and as long as I make more than I spend, I get to stay in business. And isn't that what this is all about, staying in business? That's what it's all about because I like this. I want to keep doing this. I can't keep doing it if I don't stay in business. I can't keep doing it if I make less than it costs me to make the things that I make. So I'm always thinking about the only competition I really have on an annual basis is to make sure that we make more as a company than it costs us to run the company. That's my real competition.”
0:57

I love this idea from Jason Fried "Your only competition is your costs." Keep costs low, keep the team small, make stuff you want to use. You don't need the whole world: “A business is very simple. You got to make more than you spend. If you're making more than you spend, then your competition is your cost. That's what you're really in business against, how much it costs you to stay in business. It's not all the other alternatives that are on the market. You can't control what they're going to put out there, what they're going to price it at, all the things they're going to do. They're going to do what they're going to do. What I can control is how much it costs me to run my business, how much I sell my product for, and as long as I make more than I spend, I get to stay in business. And isn't that what this is all about, staying in business? That's what it's all about because I like this. I want to keep doing this. I can't keep doing it if I don't stay in business. I can't keep doing it if I make less than it costs me to make the things that I make. So I'm always thinking about the only competition I really have on an annual basis is to make sure that we make more as a company than it costs us to run the company. That's my real competition.”

David Senra

103,326 Aufrufe • vor 4 Monaten

"We look for people that are students of history, that have studied the markets and all prior attempts." David Haber on what great founders do before building a company: "If you're going to spend the next 10 years of your life building a company, why wouldn't you spend time studying every prior attempt?" "It always gives me much more discomfort when somebody is saying, 'All those people are idiots. We're the smarter team.'" "I think having respect for prior attempts, and in some cases even having spoken to the former entrepreneurs in that category." "And then having a very clear articulation of what you're going to do differently or why now is different." David Haber on The High Flyers Podcast with Vidit Agarwal
1:02

"We look for people that are students of history, that have studied the markets and all prior attempts." David Haber on what great founders do before building a company: "If you're going to spend the next 10 years of your life building a company, why wouldn't you spend time studying every prior attempt?" "It always gives me much more discomfort when somebody is saying, 'All those people are idiots. We're the smarter team.'" "I think having respect for prior attempts, and in some cases even having spoken to the former entrepreneurs in that category." "And then having a very clear articulation of what you're going to do differently or why now is different." David Haber on The High Flyers Podcast with Vidit Agarwal

a16z

53,612 Aufrufe • vor 5 Monaten

"There should be as many models in the world as firms in the world." Satya and I dig into when to own vs. rent your intelligence, why every company should be building and climbing its own private evals, and what makes for a stable frontier.
32:39

"There should be as many models in the world as firms in the world." Satya and I dig into when to own vs. rent your intelligence, why every company should be building and climbing its own private evals, and what makes for a stable frontier.

Yash Patil

237,060 Aufrufe • vor 2 Monaten

Marc Andreessen says AI is hyper democratizing. “Is the future of the world going to be one or a small number of companies, or for that matter, governments or super AIs that kind of own and control everything?” “Or is this a technology that's going to diffuse out and be in everybody's hands?” “This is the time in which AI is actually hyper democratizing. It has spread to more individuals, both in the country and around the world in the shortest period of time of any new technology I think in history.” “The best AIs in the world are in the consumer products.” “This technology is going to be in everybody's hands. And everybody is going to be able to use it to optimize the things that they do, have it be a thought partner, assistant for building companies.” Marc Andreessen 🇺🇸
1:34

Marc Andreessen says AI is hyper democratizing. “Is the future of the world going to be one or a small number of companies, or for that matter, governments or super AIs that kind of own and control everything?” “Or is this a technology that's going to diffuse out and be in everybody's hands?” “This is the time in which AI is actually hyper democratizing. It has spread to more individuals, both in the country and around the world in the shortest period of time of any new technology I think in history.” “The best AIs in the world are in the consumer products.” “This technology is going to be in everybody's hands. And everybody is going to be able to use it to optimize the things that they do, have it be a thought partner, assistant for building companies.” Marc Andreessen 🇺🇸

a16z

79,935 Aufrufe • vor 10 Monaten

$PLTR this analyst gets it: - Palantir is the AI Operating System - Palantir provides the environment and context layer (ontology + security model) for enterprises to deploy AI models and agents - This is a structural growth opportunity - The ROI for enterprises to deploy AI and use Palantir’s service will become increasingly attractive - (Regarding living up to today’s PE) Revenue growth and margin is unique. “You’ve never seen that type of growth and margin previously for any other companies. Enterprise is a huge market. AI is really shifting from consumer to the enterprise world. Enterprise offers a larger TAM. This is early stage for enterprise to use AI. The TAM could be trillion $$ level…. The path towards that will never be easy, it will be uneven. If you have that long term view, you will still be long term bullish on the company.” - Enterprise opportunities across the world will totally change the business of Palantir in the coming decades I don’t think he’s bullish enough on government based on what we see coming from Gokarp and DV, but at least he acknowledges the importance of the government sector to the company.
3:37

$PLTR this analyst gets it: - Palantir is the AI Operating System - Palantir provides the environment and context layer (ontology + security model) for enterprises to deploy AI models and agents - This is a structural growth opportunity - The ROI for enterprises to deploy AI and use Palantir’s service will become increasingly attractive - (Regarding living up to today’s PE) Revenue growth and margin is unique. “You’ve never seen that type of growth and margin previously for any other companies. Enterprise is a huge market. AI is really shifting from consumer to the enterprise world. Enterprise offers a larger TAM. This is early stage for enterprise to use AI. The TAM could be trillion $$ level…. The path towards that will never be easy, it will be uneven. If you have that long term view, you will still be long term bullish on the company.” - Enterprise opportunities across the world will totally change the business of Palantir in the coming decades I don’t think he’s bullish enough on government based on what we see coming from Gokarp and DV, but at least he acknowledges the importance of the government sector to the company.

Jack Prescott

112,551 Aufrufe • vor 1 Jahr

Gavin Baker says the future for the world's biggest companies is open models and private context: "I think the future is an ensemble of models. There's a Pareto curve. No one model is going to be the best at everything." "For the global 1,000 biggest companies, you're going to take whatever the best open-source model is. I think probably in the very near future, that's going to be an Nvidia model." "Everybody says 'Well, in a world where open-source wins, who funds the training?' Chip companies can fund the training." "It's trivial to do a $50 to $100 billion training run for Jensen, but I do think you're going to see American open-source led by Nvidia get really close to the frontier." "Sharing your own enterprise context, that's truly your IP, that's truly the value of your company, the context embedded in all of your data, sharing that with a frontier lab may be hazardous for your financial health." "I think what you'll see these companies do is they'll have their own model on their data, and it will work with one or two other frontier models, checking each other... I think that feels like a very likely future to me." Gavin Baker David George
1:37

Gavin Baker says the future for the world's biggest companies is open models and private context: "I think the future is an ensemble of models. There's a Pareto curve. No one model is going to be the best at everything." "For the global 1,000 biggest companies, you're going to take whatever the best open-source model is. I think probably in the very near future, that's going to be an Nvidia model." "Everybody says 'Well, in a world where open-source wins, who funds the training?' Chip companies can fund the training." "It's trivial to do a $50 to $100 billion training run for Jensen, but I do think you're going to see American open-source led by Nvidia get really close to the frontier." "Sharing your own enterprise context, that's truly your IP, that's truly the value of your company, the context embedded in all of your data, sharing that with a frontier lab may be hazardous for your financial health." "I think what you'll see these companies do is they'll have their own model on their data, and it will work with one or two other frontier models, checking each other... I think that feels like a very likely future to me." Gavin Baker David George

a16z

39,881 Aufrufe • vor 11 Tagen

"We treat software like it is so precious. And in the new world we're going to, it's going to be much more fungible. But that's the good news." Garry Tan on where the moats will be for future YC companies: "That's where taste, agency, and trust matter a lot. This is something that's only crystallized recently. We've been talking about agency and taste for the longest time. Agency is being able to prompt, and taste is being able to do evals." "Trust is the third thing. It's probably the most important. When you have an enterprise company that is selling to real businesses that are relying on you — that's your moat. Your moat is actually trust. They went through the effort to try you, get you onboarded, and incorporate you into how they work."
1:48

"We treat software like it is so precious. And in the new world we're going to, it's going to be much more fungible. But that's the good news." Garry Tan on where the moats will be for future YC companies: "That's where taste, agency, and trust matter a lot. This is something that's only crystallized recently. We've been talking about agency and taste for the longest time. Agency is being able to prompt, and taste is being able to do evals." "Trust is the third thing. It's probably the most important. When you have an enterprise company that is selling to real businesses that are relying on you — that's your moat. Your moat is actually trust. They went through the effort to try you, get you onboarded, and incorporate you into how they work."

TBPN

43,619 Aufrufe • vor 5 Monaten

Sam Altman: “One of the few arbitrage opportunities left in the market is time” “I don’t think you can beat the market in a lot of ways, but the one way I do is by making a long-term commitment to something. My new belief on how long I should hold stock in the best companies I invest in is forever. And I think in a world where people are increasingly focused on the tick and the quarterly earnings cycle, you should try to go in the other direction. This is a great way to generate value and wealth.” Sam continues: “And so I think that when you’re thinking about a startup, it’s really worthwhile to think about something you’re willing to make a very long-term commitment to because that is where the current void in the market is. You get paid as a founder for the wealth you create for other people… The best companies just create massive amounts of value for the world, and then they capture some of that for themselves. But they capture far less than they create, and they do it over a very long period of time. So if you’re going to be one of those companies, I think you want to make a super long-term commitment to yourself and others that are going to work on it with you… It is worth waiting because you will make far more money over the long term by doing this company really well than a bunch of short term things along the way.” Video source: Y Combinator (2016)
1:44

Sam Altman: “One of the few arbitrage opportunities left in the market is time” “I don’t think you can beat the market in a lot of ways, but the one way I do is by making a long-term commitment to something. My new belief on how long I should hold stock in the best companies I invest in is forever. And I think in a world where people are increasingly focused on the tick and the quarterly earnings cycle, you should try to go in the other direction. This is a great way to generate value and wealth.” Sam continues: “And so I think that when you’re thinking about a startup, it’s really worthwhile to think about something you’re willing to make a very long-term commitment to because that is where the current void in the market is. You get paid as a founder for the wealth you create for other people… The best companies just create massive amounts of value for the world, and then they capture some of that for themselves. But they capture far less than they create, and they do it over a very long period of time. So if you’re going to be one of those companies, I think you want to make a super long-term commitment to yourself and others that are going to work on it with you… It is worth waiting because you will make far more money over the long term by doing this company really well than a bunch of short term things along the way.” Video source: Y Combinator (2016)

Startup Archive

49,283 Aufrufe • vor 1 Jahr

David Sacks says companies are trapped paying OpenAI & Anthropic because they can't figure out how to use open source models "I think enterprise CTOs would like to shift their token consumption to cheaper models for the obvious reason that it would be more efficient. They are seeing compute costs or token costs skyrocket right now, so everyone's trying to figure this out." "You also have the AI sovereignty issue that Alex Karp talked about. They're worried about giving up the secret sauce or the alpha in their business to a frontier lab that may one day be competing with them. "The problem is, I think in most cases, they don't have the technical ability to do it. Coinbase figured out how to do it. DoorDash figured out how to do it. They built a token routing system that allows them to send frontier tasks to frontier models and non frontier tasks to more mundane models. But I don't think your average enterprise has the technical capability to do that." "This is why the share of wallet of closed models, it actually increased. I think that open source went from 19% last year to 11% this year. So open source as a share of enterprise spending is actually decreasing." "I don't think that means usage is decreasing. I think usage is skyrocketing. It also may be the case that because the whole point of using an open model is you just pay for the compute costs, you don't have to pay a lab, so it may be that it's hard to measure that usage in terms of spend." "But nonetheless, anyone who's saying that these closed models are going to lose or are somehow losing, you're just not seeing it in the data."
2:06

David Sacks says companies are trapped paying OpenAI & Anthropic because they can't figure out how to use open source models "I think enterprise CTOs would like to shift their token consumption to cheaper models for the obvious reason that it would be more efficient. They are seeing compute costs or token costs skyrocket right now, so everyone's trying to figure this out." "You also have the AI sovereignty issue that Alex Karp talked about. They're worried about giving up the secret sauce or the alpha in their business to a frontier lab that may one day be competing with them. "The problem is, I think in most cases, they don't have the technical ability to do it. Coinbase figured out how to do it. DoorDash figured out how to do it. They built a token routing system that allows them to send frontier tasks to frontier models and non frontier tasks to more mundane models. But I don't think your average enterprise has the technical capability to do that." "This is why the share of wallet of closed models, it actually increased. I think that open source went from 19% last year to 11% this year. So open source as a share of enterprise spending is actually decreasing." "I don't think that means usage is decreasing. I think usage is skyrocketing. It also may be the case that because the whole point of using an open model is you just pay for the compute costs, you don't have to pay a lab, so it may be that it's hard to measure that usage in terms of spend." "But nonetheless, anyone who's saying that these closed models are going to lose or are somehow losing, you're just not seeing it in the data."

dnap

110,354 Aufrufe • vor 2 Monaten

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A
34:33

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

8VC

104,829 Aufrufe • vor 29 Tagen

Collison Brothers Explain Why Stripe is Better Off Staying Private John's (John Collison) take: " People generally make the argument that public companies run in a more disciplined fashion. I think that's hogwash." "If you need a 25-year old Fidelity analyst asking you to double click on your CapEx, blah, blah, blah, to run the company with discipline, something is horribly wrong at the company and you need new management." "It used to be the case that to do any return of capital to shareholders, or if you needed any kind of large sums of money, you needed the public markets." "That's obviously not true today, where stable private markets exist." " This is our life's work. We're not going anywhere. We'll be very happily running Stripe in 10 years time, in 20 years time." Patrick's (Patrick Collison) take: "We're a company at the intersection of financial services and technology. Being private for a long time is the norm." "What matters is less kind of the returns in a given year, and more duration. And so the question is, what enables the best compounding on a 10 year time horizon?" "And what's best for shareholders as you really take the longer-term perspective? And then what's best for customers?" " At this juncture, with the business growing at this rate, we want to spend the marginal hour with some customer."
1:02

Collison Brothers Explain Why Stripe is Better Off Staying Private John's (John Collison) take: " People generally make the argument that public companies run in a more disciplined fashion. I think that's hogwash." "If you need a 25-year old Fidelity analyst asking you to double click on your CapEx, blah, blah, blah, to run the company with discipline, something is horribly wrong at the company and you need new management." "It used to be the case that to do any return of capital to shareholders, or if you needed any kind of large sums of money, you needed the public markets." "That's obviously not true today, where stable private markets exist." " This is our life's work. We're not going anywhere. We'll be very happily running Stripe in 10 years time, in 20 years time." Patrick's (Patrick Collison) take: "We're a company at the intersection of financial services and technology. Being private for a long time is the norm." "What matters is less kind of the returns in a given year, and more duration. And so the question is, what enables the best compounding on a 10 year time horizon?" "And what's best for shareholders as you really take the longer-term perspective? And then what's best for customers?" " At this juncture, with the business growing at this rate, we want to spend the marginal hour with some customer."

The All-In Podcast

407,507 Aufrufe • vor 1 Jahr

Rogin: We give them the weapons, we're funding this war, we should be able to have the more than a 50% say in whether American blood and treasure is spent to perpetuate this war for months and months. Who's going to win that disagreement? Is it going to be Trump and we're going to end the war soon? Or is it going to be Bibi? And the war is going to go on for a very long time. In the end, we’re going to find out who wears the pants in this bromance.
1:24

Rogin: We give them the weapons, we're funding this war, we should be able to have the more than a 50% say in whether American blood and treasure is spent to perpetuate this war for months and months. Who's going to win that disagreement? Is it going to be Trump and we're going to end the war soon? Or is it going to be Bibi? And the war is going to go on for a very long time. In the end, we’re going to find out who wears the pants in this bromance.

Acyn

33,709 Aufrufe • vor 5 Monaten

“What I think SPX is, is a wake-up call. And I think for the normie, the average person, it literally is best described as being the next Bitcoin, because that's literally what it is. Bitcoin was a wake-up call to show people that value is subjective. And SPX is the next iteration of that. And once you get enough people to understand that the value of the dollar is subjective. It only has value because people believe in it. Bitcoin only has value because people believe in it. SPX has value because people believe in it. And the next step is Well, it's not just money that is subjective in value. It is literally everything is subjective. The entire universe is this subjective entity that is created from our own consciousness. And that's what Like, the purpose of this vehicle, yes, it's going to make a lot of people a lot of money, but the point of that is to wake people up to the fact that the entire universe is this subjective entity. It is not just this, like, empirical machine with interlocking gears and like this orderly machine that proceeds from just like the sum of its parts like no like we are literally the universe creating the universe and that's what spx is meant to wake people up to.” — Crypto Maxi 💹🧲
1:36

“What I think SPX is, is a wake-up call. And I think for the normie, the average person, it literally is best described as being the next Bitcoin, because that's literally what it is. Bitcoin was a wake-up call to show people that value is subjective. And SPX is the next iteration of that. And once you get enough people to understand that the value of the dollar is subjective. It only has value because people believe in it. Bitcoin only has value because people believe in it. SPX has value because people believe in it. And the next step is Well, it's not just money that is subjective in value. It is literally everything is subjective. The entire universe is this subjective entity that is created from our own consciousness. And that's what Like, the purpose of this vehicle, yes, it's going to make a lot of people a lot of money, but the point of that is to wake people up to the fact that the entire universe is this subjective entity. It is not just this, like, empirical machine with interlocking gears and like this orderly machine that proceeds from just like the sum of its parts like no like we are literally the universe creating the universe and that's what spx is meant to wake people up to.” — Crypto Maxi 💹🧲

Tanz Cho 💹🧲

12,294 Aufrufe • vor 1 Monat