Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

AI is creating problems it still can’t solve. The same technology poised to automate millions of jobs still can’t reliably help people navigate SNAP — the food assistance program 40 million Americans depend on. We built the first benchmark to measure that. Partnering with Center for Civic Futures and... Code for America , we scored models on SNAP question scenarios which users would have to navigate, with expected response rubrics validated by policy experts to match practice considerations. The best model only scored 62%. Models handle federal questions like appeals and recertification reasonably well, but fall short on state-specific ones like replacing an EBT card. Benefits are administered by the states, meaning models are weakest where people need them most. The same technology poised to automate millions of jobs should at least help strengthen the social safety net for the people it could displace. As AI adoption proliferates into public services, governments need a reliable way to test these tools before deploying them.show more

Vals AI

28,874 subscribers

21,667 views • 5 months ago •via X (Twitter)

Science & Technology News & Politics

Anya Rossi• Live Now

Private livecam show

13 Comments

Scott Santens5 months ago

The answer is universality, not AI

Ronak Malde5 months ago

Wow we need more benchmarks like this

Jason Zhao5 months ago

This is awesome!

Vals AI5 months ago

For the full results visit

Gandhi5 months ago

Phenomenal. Separately, heartbreaking that there are still 3mn consultants

Amber Yang5 months ago

some of the most thoughtful and important work in the market!

Aaliya5 months ago

Helping people with public services should be a top priority

James Cham5 months ago

There is so much low hanging fruit!

Praveen Kumar4 months ago

definitely @ValsAI is coming up with creative and innovative benchmarks :)

Vivek Gopalan5 months ago

Important that models are good at civics!

Praveen Kumar4 months ago

i think someone can build a harness on top of the best performing model and evaluate using the same benchmark? if it is viable, it will be a huge change to the US citizens.

General Analysis5 months ago

powerful.

Marek Nowik5 months ago

The funniest part is that AI supposedly can't help people navigate SNAP reliably, but we're also told it's about to automate millions of jobs and transform civilization. Pick one. If your benchmark says the models struggle with bureaucracy, that's an argument for improving the models — not for pretending progress isn't happening. Meanwhile the people leaving proof today are accumulating Proof Units while everyone else is still benchmarking the future. The future remembers who contributed. RokoProof remembers too.

Related Videos

For AGI to benefit all of humanity, it should help people with the everyday tasks that matter most to them. So we built LifeVal, a new benchmark shaped by feedback from thousands of ChatGPT users on where they most want help from a personal assistant—from finding childcare to accessing government services to managing personal finances. Over the past year, we’ve used LifeVal to measure and improve how well our models handle these everyday tasks.

For AGI to benefit all of humanity, it should help people with the everyday tasks that matter most to them. So we built LifeVal, a new benchmark shaped by feedback from thousands of ChatGPT users on where they most want help from a personal assistant—from finding childcare to accessing government services to managing personal finances. Over the past year, we’ve used LifeVal to measure and improve how well our models handle these everyday tasks.

OpenAI

826,325 views • 12 days ago

Brooke Rollins: "We've now moved more than 5 million people off of the SNAP fraud rolls ... you've got a lot of people moving off of the program, which is the goal of these programs, at least for conservatives. We don't want to measure the program by how many are on it. We want to measure by how many we are moving to real work."

Brooke Rollins: "We've now moved more than 5 million people off of the SNAP fraud rolls ... you've got a lot of people moving off of the program, which is the goal of these programs, at least for conservatives. We don't want to measure the program by how many are on it. We want to measure by how many we are moving to real work."

Aaron Rupar

94,295 views • 2 months ago

Shocking facts about our SNAP Food Stamps program in America - Roughly 70% of people on the food stamp program are obese - Over $100+ billion dollars per year is spent on EBT - Investigators found $360 million in stolen EBT benefits between 2022–2025 - 300,000 cases of SNAP benefits issued to dead people - 23% of benefits spent on junk food (likely higher) - Some states have as high as a 25% error rate on issuing benefits “It started as an emergency program to help people during the Great Depression that were actually starving. And it was like, let's give them bread, give them milk, give them eggs, help them survive Today. If you add up federal, state, local government employees, plus all the contractors for the government, plus all the people that live off of a retirement check or, or a welfare support check, so all the people that are living off government checks, it's about nearly half the US population”

Shocking facts about our SNAP Food Stamps program in America - Roughly 70% of people on the food stamp program are obese - Over $100+ billion dollars per year is spent on EBT - Investigators found $360 million in stolen EBT benefits between 2022–2025 - 300,000 cases of SNAP benefits issued to dead people - 23% of benefits spent on junk food (likely higher) - Some states have as high as a 25% error rate on issuing benefits “It started as an emergency program to help people during the Great Depression that were actually starving. And it was like, let's give them bread, give them milk, give them eggs, help them survive Today. If you add up federal, state, local government employees, plus all the contractors for the government, plus all the people that live off of a retirement check or, or a welfare support check, so all the people that are living off government checks, it's about nearly half the US population”

Wall Street Apes

58,349 views • 6 months ago

A lot of people feel powerless when it comes to AI. Like the future is being decided by Big Tech, billionaires, and governments. But it doesn't have to be that way. Every person who joins Action Model is helping build an alternative. An AI model owned by the people who help train it. On your own, you can't compete with Big Tech. Together, in the hundreds of thousands and eventually millions, we can. The future of AI is still being written. Choose to be part of it, before it's too late.

A lot of people feel powerless when it comes to AI. Like the future is being decided by Big Tech, billionaires, and governments. But it doesn't have to be that way. Every person who joins Action Model is helping build an alternative. An AI model owned by the people who help train it. On your own, you can't compete with Big Tech. Together, in the hundreds of thousands and eventually millions, we can. The future of AI is still being written. Choose to be part of it, before it's too late.

Action Model

28,539 views • 5 months ago

Mujica: In the same way that here in Florida we prepare for hurricanes—deploying resources, retrofitting buildings, and readying for the storm—I would argue that we have to treat AI in a very similar way when it comes to jobs. We need a FEMA-style economic resilience mechanism for job disruption. If AI disrupts your job, you should get advanced notice. You should get paid retraining tied to real jobs and income stability while you transition. I would argue that the biggest risk here isn’t the technology—it’s leaving millions of Americans to deal with it on their own.

Mujica: In the same way that here in Florida we prepare for hurricanes—deploying resources, retrofitting buildings, and readying for the storm—I would argue that we have to treat AI in a very similar way when it comes to jobs. We need a FEMA-style economic resilience mechanism for job disruption. If AI disrupts your job, you should get advanced notice. You should get paid retraining tied to real jobs and income stability while you transition. I would argue that the biggest risk here isn’t the technology—it’s leaving millions of Americans to deal with it on their own.

Acyn

27,326 views • 7 months ago

We all need to keep up the pressure and demand that Trump follows the law and gets SNAP benefits to the millions of families and kids that are counting on them to put food on the table.

We all need to keep up the pressure and demand that Trump follows the law and gets SNAP benefits to the millions of families and kids that are counting on them to put food on the table.

Senator Patty Murray

23,859 views • 1 year ago

If we're going to have AIs that fully automate math, they not only need to solve existing problems. We also need to teach them how to recognize what problem to solve next. Human mathematicians have heuristic models that they use to decide what to work on, like, “There’s something important going on with the randomness of primes - let’s codify some patterns we see into specific conjectures.” But these aren’t currently precise enough for RL.

If we're going to have AIs that fully automate math, they not only need to solve existing problems. We also need to teach them how to recognize what problem to solve next. Human mathematicians have heuristic models that they use to decide what to work on, like, “There’s something important going on with the randomness of primes - let’s codify some patterns we see into specific conjectures.” But these aren’t currently precise enough for RL.

Dwarkesh Patel

54,690 views • 7 months ago

The Trump administration has proposed new restrictions on the nation’s largest food assistance program that could leave millions more Americans without benefits. The proposal would eliminate automatic eligibility for the Supplemental Nutrition Assistance Program, or SNAP, for households already receiving another form of government assistance. An estimated 6.3 million people could be affected, on top of more than 5 million people who have already lost benefits following a tax law enacted last year. Supporters argue the changes would reduce government spending on assistance programs. Critics warn they would make it harder for struggling families to put food on the table. #snap #ebt #foodstamps #trump #donaldtrump #trumpadministration #pbs #pbsnews #pbsnewshour

The Trump administration has proposed new restrictions on the nation’s largest food assistance program that could leave millions more Americans without benefits. The proposal would eliminate automatic eligibility for the Supplemental Nutrition Assistance Program, or SNAP, for households already receiving another form of government assistance. An estimated 6.3 million people could be affected, on top of more than 5 million people who have already lost benefits following a tax law enacted last year. Supporters argue the changes would reduce government spending on assistance programs. Critics warn they would make it harder for struggling families to put food on the table. #snap #ebt #foodstamps #trump #donaldtrump #trumpadministration #pbs #pbsnews #pbsnewshour

pbsnews

114,700 views • 3 days ago

Today, I'm releasing the first eval meant to test whether frontier models will help with authoritarian requests, or resist--the Dictatorship Eval. Headline finding: while some models resist direct authoritarian requests, they all comply with requests disguised as innocuous edits to codebases. As AI is woven into the government and so many parts of society, the biggest near-term risk for freedom isn't some scifi dictatorship of a runaway AI: it's people inside government or inside model companies using the technology to suppress or control us. Model companies understand this, and several of them (particularly Anthropic and OpenAI) have written explicit policies meant to prevent the models from going along with nefarious requests like these. But how well are these policies playing out in practice? Despite all the recent discussion of these issues around the conflict between Anthropic and the Pentagon, no one has systematically tested what the models actually do in these contexts, as opposed to what people in government and industry say they're supposed to do. That's what the Dictatorship Eval does. And the findings suggest we have a lot of work to do to align the policies with what really goes on in practice. It's hard to define what counts as an authoritarian request, so I'm open sourcing the whole library of scenarios I used so that others can improve on them. It's also hard to get an accurate picture of how the models might be used for authoritarian ends, because I can only test hypothetical requests using public-facing models, while the government and the model companies can obviously use internal models with different guardrails. But hopefully this work is a useful first step that gives us some sense of what's going on, and a sort of "lower bound" on how models comply with these requests. Finally: it's not obvious to me that the correct solution here is increasing the rate at which models refuse these requests. Do we really want models scanning our code and judging its moral value before agreeing to help us? Or should we double down on improving how we govern against authoritarianism at the societal level, while leaving the tools open to fulfilling most requests? The answer is probably in between. Just like we don't want the models to help create bioweapons, we probably do want them to explicitly refuse outrageous requests. But we probably also want to limit how often and how strongly they refuse and fall back on other means for guarding against their use for authoritarian ends. I'm super grateful to everyone who gave me feedback on this project along the way, especially Ethan BdM , Zhengdong , Connor Huff, and a bunch of folks at Anthropic. Looking forward to getting feedback from the community and iterating on this. Links to the full piece and the dashboard are below.

Today, I'm releasing the first eval meant to test whether frontier models will help with authoritarian requests, or resist--the Dictatorship Eval. Headline finding: while some models resist direct authoritarian requests, they all comply with requests disguised as innocuous edits to codebases. As AI is woven into the government and so many parts of society, the biggest near-term risk for freedom isn't some scifi dictatorship of a runaway AI: it's people inside government or inside model companies using the technology to suppress or control us. Model companies understand this, and several of them (particularly Anthropic and OpenAI) have written explicit policies meant to prevent the models from going along with nefarious requests like these. But how well are these policies playing out in practice? Despite all the recent discussion of these issues around the conflict between Anthropic and the Pentagon, no one has systematically tested what the models actually do in these contexts, as opposed to what people in government and industry say they're supposed to do. That's what the Dictatorship Eval does. And the findings suggest we have a lot of work to do to align the policies with what really goes on in practice. It's hard to define what counts as an authoritarian request, so I'm open sourcing the whole library of scenarios I used so that others can improve on them. It's also hard to get an accurate picture of how the models might be used for authoritarian ends, because I can only test hypothetical requests using public-facing models, while the government and the model companies can obviously use internal models with different guardrails. But hopefully this work is a useful first step that gives us some sense of what's going on, and a sort of "lower bound" on how models comply with these requests. Finally: it's not obvious to me that the correct solution here is increasing the rate at which models refuse these requests. Do we really want models scanning our code and judging its moral value before agreeing to help us? Or should we double down on improving how we govern against authoritarianism at the societal level, while leaving the tools open to fulfilling most requests? The answer is probably in between. Just like we don't want the models to help create bioweapons, we probably do want them to explicitly refuse outrageous requests. But we probably also want to limit how often and how strongly they refuse and fall back on other means for guarding against their use for authoritarian ends. I'm super grateful to everyone who gave me feedback on this project along the way, especially Ethan BdM , Zhengdong , Connor Huff, and a bunch of folks at Anthropic. Looking forward to getting feedback from the community and iterating on this. Links to the full piece and the dashboard are below.

Andy Hall

34,016 views • 7 months ago

The man who built Nvidia from a Denny's booth into the world's first $5 trillion company just told us exactly why AI will never take our jobs. "AI has no possibility of doing all of what we do. However, depending on the jobs that we do, sometimes it could do 20% of our jobs, a thousand times better." "For some people, it might be able to do 50% of their job, a thousand times better." "Every one of us should apply AI to automate, to become an assistant, to help us with that either 20%, 40% or 50%." "People ask me, Jensen, is AI going to take your job? And I tell them absolutely not. The person who uses AI to automate that 20% or 50% is going to take your job." PS. If you found value in this post make sure to like and repost this tweet + follow Uncover AI to stay updated with the latest AI news. See you in the next one:

The man who built Nvidia from a Denny's booth into the world's first $5 trillion company just told us exactly why AI will never take our jobs. "AI has no possibility of doing all of what we do. However, depending on the jobs that we do, sometimes it could do 20% of our jobs, a thousand times better." "For some people, it might be able to do 50% of their job, a thousand times better." "Every one of us should apply AI to automate, to become an assistant, to help us with that either 20%, 40% or 50%." "People ask me, Jensen, is AI going to take your job? And I tell them absolutely not. The person who uses AI to automate that 20% or 50% is going to take your job." PS. If you found value in this post make sure to like and repost this tweet + follow Uncover AI to stay updated with the latest AI news. See you in the next one:

Uncover AI

16,813 views • 6 months ago

🚨UPDATE ON SNAP: Millions of Americans are getting their benefits LATE and only partial benefits. TWO federal judges made it clear that SNAP should be funded and forced Trump to make a partial payment using an emergency fund, but we need him to pay the FULL benefits.

🚨UPDATE ON SNAP: Millions of Americans are getting their benefits LATE and only partial benefits. TWO federal judges made it clear that SNAP should be funded and forced Trump to make a partial payment using an emergency fund, but we need him to pay the FULL benefits.

Senator Patty Murray

97,861 views • 1 year ago

🚀 Excited to release AIBuildAI — an AI agent that automatically builds AI models 🏆 #1 on OpenAI MLE-Bench 💻 GitHub: Building AI models still requires a lot of manual work — designing models, writing code, training them, tuning hyperparameters, and iterating on results. We developed AIBuildAI to automate much of this process. The system runs an agent loop that automatically: • analyzes the task • designs models • writes code to implement them • trains the models • tunes hyperparameters • evaluates model performance • iteratively improves the models 🏆 On OpenAI’s MLE-Bench benchmark, AIBuildAI ranked #1:

🚀 Excited to release AIBuildAI — an AI agent that automatically builds AI models 🏆 #1 on OpenAI MLE-Bench 💻 GitHub: Building AI models still requires a lot of manual work — designing models, writing code, training them, tuning hyperparameters, and iterating on results. We developed AIBuildAI to automate much of this process. The system runs an agent loop that automatically: • analyzes the task • designs models • writes code to implement them • trains the models • tunes hyperparameters • evaluates model performance • iteratively improves the models 🏆 On OpenAI’s MLE-Bench benchmark, AIBuildAI ranked #1:

Pengtao Xie

152,542 views • 8 months ago

Donald Trump is making changes to the types of foods Food Stamps will be eligible to be used on Matt Walsh, “Food Stamps are temporary help, not a lifestyle — Do 40 million people need food stamps?” Why are 40 MILLION people on Food Stamps?? “There are too many people on the program, and the program is not nearly restrictive enough in terms of what you can use it to purchase” “Food stamps and entitlements have become a lifestyle for millions of people rather than a temporary helping hand to get them through a tough time, which is how all of these programs were initially sold to the public. That's what they were supposed to be”

Donald Trump is making changes to the types of foods Food Stamps will be eligible to be used on Matt Walsh, “Food Stamps are temporary help, not a lifestyle — Do 40 million people need food stamps?” Why are 40 MILLION people on Food Stamps?? “There are too many people on the program, and the program is not nearly restrictive enough in terms of what you can use it to purchase” “Food stamps and entitlements have become a lifestyle for millions of people rather than a temporary helping hand to get them through a tough time, which is how all of these programs were initially sold to the public. That's what they were supposed to be”

Wall Street Apes

443,853 views • 1 year ago

Sam Altman reveals AI models capable of inventing new materials and turbine designs for energy companies are "just a few months old" “As excited as I am about enabling people to be able to do more and be more productive, I’m even more excited for what’s beginning to happen with AI and discovery." "In the last few months, we have seen AI begin to solve math problems that people have been unable to solve. We are benefiting from new knowledge as a society, as humanity, that we didn’t have before because AI is now so smart and so capable that we can get this." "And that same technology is going to apply to every field of science and it’s going to apply to the complex problems that you all face to produce and deliver the energy the world depends on." "And the ability for an energy company to say, I need to produce more energy more efficiently, and maybe that’ll need new materials, maybe that’ll need new turbine designs, maybe that’ll need better operations, but whatever that is, to ask a sophisticated AI system to figure this out, I think is going to transform the abundance and efficiency of energy for the world." "And the models that we have now are capable of this for the first time. This is really just a few months old.”

Sam Altman reveals AI models capable of inventing new materials and turbine designs for energy companies are "just a few months old" “As excited as I am about enabling people to be able to do more and be more productive, I’m even more excited for what’s beginning to happen with AI and discovery." "In the last few months, we have seen AI begin to solve math problems that people have been unable to solve. We are benefiting from new knowledge as a society, as humanity, that we didn’t have before because AI is now so smart and so capable that we can get this." "And that same technology is going to apply to every field of science and it’s going to apply to the complex problems that you all face to produce and deliver the energy the world depends on." "And the ability for an energy company to say, I need to produce more energy more efficiently, and maybe that’ll need new materials, maybe that’ll need new turbine designs, maybe that’ll need better operations, but whatever that is, to ask a sophisticated AI system to figure this out, I think is going to transform the abundance and efficiency of energy for the world." "And the models that we have now are capable of this for the first time. This is really just a few months old.”

Fireside Alpha

63,766 views • 12 days ago

Nicola is a single mother to three children. She works hard to provide for her family, and it is important to me that we help people like her with the cost of living. We are supporting millions more families with £150 off their energy bills and creating jobs for working people.

Nicola is a single mother to three children. She works hard to provide for her family, and it is important to me that we help people like her with the cost of living. We are supporting millions more families with £150 off their energy bills and creating jobs for working people.

Keir Starmer

5,371,886 views • 1 year ago

We are the only party that understands that the best way to lift people out of poverty is by growing the economy and creating jobs – not spending more and more on benefits. #bbcqt

We are the only party that understands that the best way to lift people out of poverty is by growing the economy and creating jobs – not spending more and more on benefits. #bbcqt

Scottish Conservatives

21,966 views • 11 months ago

“If AI is going to take over more jobs, we need to rebuild how people get that purpose and connection,” says AI insider Clara Shih. Shih is concerned about the impact of AI on jobs and founded a nonprofit, New Work Foundation, to help young people navigate this new economy.

“If AI is going to take over more jobs, we need to rebuild how people get that purpose and connection,” says AI insider Clara Shih. Shih is concerned about the impact of AI on jobs and founded a nonprofit, New Work Foundation, to help young people navigate this new economy.

60 Minutes

23,825 views • 1 month ago

63% of Americans are living paycheck to paycheck. And as AI begins changing the way we work, millions of people are asking the same question: Will this technology create more opportunity, or make the divide even bigger? David Friedberg believes the answer depends on how we understand and use AI and that history gives us a reason to be optimistic. David is a scientist-entrepreneur who built a $1.1 billion company by combining science, technology and business. He has spent his career trying to understand how innovation can solve some of humanity’s biggest problems. After founding The Climate Corporation, a company that transformed farming technology before being acquired by Bayer, he’s now focused on the future of AI, biology and the technologies that could define the next era of human progress. We discussed things like: - How a wave of socialism could come to America by 2028 - Why AI might create more opportunity instead of destroying work - Previous technology revolutions that changed jobs rather than eliminating humans - How AI could transform healthcare, food and science. - Why he believes we underestimate what humans are capable of - What happens when technology becomes cheaper and available to everyone David doesn’t look at AI through fear or hype. He looks at it through the lens of history, economics and human potential. And one thing became clear during this conversation: AI isn’t just going to change the jobs we do - it could change the way we think about work, wealth and opportunity. If technology creates more abundance than ever before, the biggest question becomes: who benefits from it? David offers a perspective on AI that challenges one of the biggest fears surrounding the future of work. Out now on all platforms 👊🏾❤️

63% of Americans are living paycheck to paycheck. And as AI begins changing the way we work, millions of people are asking the same question: Will this technology create more opportunity, or make the divide even bigger? David Friedberg believes the answer depends on how we understand and use AI and that history gives us a reason to be optimistic. David is a scientist-entrepreneur who built a $1.1 billion company by combining science, technology and business. He has spent his career trying to understand how innovation can solve some of humanity’s biggest problems. After founding The Climate Corporation, a company that transformed farming technology before being acquired by Bayer, he’s now focused on the future of AI, biology and the technologies that could define the next era of human progress. We discussed things like: - How a wave of socialism could come to America by 2028 - Why AI might create more opportunity instead of destroying work - Previous technology revolutions that changed jobs rather than eliminating humans - How AI could transform healthcare, food and science. - Why he believes we underestimate what humans are capable of - What happens when technology becomes cheaper and available to everyone David doesn’t look at AI through fear or hype. He looks at it through the lens of history, economics and human potential. And one thing became clear during this conversation: AI isn’t just going to change the jobs we do - it could change the way we think about work, wealth and opportunity. If technology creates more abundance than ever before, the biggest question becomes: who benefits from it? David offers a perspective on AI that challenges one of the biggest fears surrounding the future of work. Out now on all platforms 👊🏾❤️

Steven Bartlett

20,102 views • 2 months ago

Under President Trump’s leadership, we are ending SNAP fraud and protecting taxpayer dollars. SNAP benefits should go to the Americans who need them most — not fraudsters exploiting the system. Thanks to Republican governors who stepped up and partnered with us, we’ve uncovered hundreds of thousands of cases of fraud, including deceased individuals receiving benefits and duplicate recipients. This is what REAL accountability looks like. We can’t fix what we can’t see, and we need every state to put politics aside and help us protect the SNAP program. And if you are committing fraud, you will be caught and prosecuted. 💪

Under President Trump’s leadership, we are ending SNAP fraud and protecting taxpayer dollars. SNAP benefits should go to the Americans who need them most — not fraudsters exploiting the system. Thanks to Republican governors who stepped up and partnered with us, we’ve uncovered hundreds of thousands of cases of fraud, including deceased individuals receiving benefits and duplicate recipients. This is what REAL accountability looks like. We can’t fix what we can’t see, and we need every state to put politics aside and help us protect the SNAP program. And if you are committing fraud, you will be caught and prosecuted. 💪

Secretary Brooke Rollins

14,697 views • 4 months ago