跳过导航
AI is creating problems it still can’t solve.
0:00 / 1:49

AI is creating problems it still can’t solve.

Vals AI

@ValsAI2.9万 位关注者

在 X 上查看
111 个赞下载原帖

21,667 次观看

AI is creating problems it still can’t solve. The same technology poised to automate millions of jobs still can’t reliably help people navigate SNAP — the food assistance program 40 million Americans depend on. We built the first benchmark to measure that. Partnering with Center for Civic Futures and Code for America , we scored models on SNAP question scenarios which users would have to navigate, with expected response rubrics validated by policy experts to match practice considerations. The best model only scored 62%. Models handle federal questions like appeals and recertification reasonably well, but fall short on state-specific ones like replacing an EBT card. Benefits are administered by the states, meaning models are weakest where people need them most. The same technology poised to automate millions of jobs should at least help strengthen the social safety net for the people it could displace. As AI adoption proliferates into public services, governments need a reliable way to test these tools before deploying them.

13 条评论

评论来自 X (Twitter) 上的原帖

  • Scott Santens 的头像

    @scottsantens

    The answer is universality, not AI

    11 个赞

  • Ronak Malde 的头像

    @rronak_

    Wow we need more benchmarks like this

    7 个赞

  • Jason Zhao 的头像

    @jasonzhao

    This is awesome!

    5 个赞

  • Vals AI 的头像

    @ValsAI

    For the full results visit

    4 个赞

  • Gandhi 的头像

    @ishangandhixyz

    Phenomenal. Separately, heartbreaking that there are still 3mn consultants

    4 个赞

  • Amber Yang 的头像

    @theamberyang

    some of the most thoughtful and important work in the market!

    4 个赞

  • Aaliya 的头像

    @aaliya_va

    Helping people with public services should be a top priority

    3 个赞

  • James Cham 的头像

    @jamescham

    There is so much low hanging fruit!

    3 个赞

  • Praveen Kumar 的头像

    @itzzme_pk

    definitely @ValsAI is coming up with creative and innovative benchmarks :)

    1 个赞

  • Vivek Gopalan 的头像

    @vvkgopalan

    Important that models are good at civics!

    1 个赞

  • Praveen Kumar 的头像

    @itzzme_pk

    i think someone can build a harness on top of the best performing model and evaluate using the same benchmark? if it is viable, it will be a huge change to the US citizens.

  • General Analysis 的头像

    @gen_analysis

    powerful.

  • Marek Nowik 的头像

    @marek_novik84

    The funniest part is that AI supposedly can't help people navigate SNAP reliably, but we're also told it's about to automate millions of jobs and transform civilization. Pick one. If your benchmark says the models struggle with bureaucracy, that's an argument for improving the models — not for pretending progress isn't happening. Meanwhile the people leaving proof today are accumulating Proof Units while everyone else is still benchmarking the future. The future remembers who contributed. RokoProof remembers too.