Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Today we're launching the Specialized Intelligence Index (SII): one destination for real-work benchmarks across industries, built by the teams that use them every day. Hear from Fireworks co-founder Benny (Yufei) Chen on the importance of specialized benchmarks:

Fireworks

33,277 subscribers

49,140 views • 13 days ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

10 Comments

Fireworks13 days ago

Featured on SII: @doximity @mercor @harvey, @depthfirstlabs, @novee_security, @genspark_ai, @SierraPlatform, @DecagonAI, @ProximalHQ, @Macroscope, @RogoAI, @traversal_ai

Francesco Piccoli13 days ago

@the_bunny_chen Congrats on the launch, great partnering with you on this!

Brian Bensch13 days ago

@the_bunny_chen the world's collective intelligence will never fit inside a single box. it also therefore shouldn't be evaluated by a single benchmark.

Madison Kanna13 days ago

@the_bunny_chen 🥳🥳🥳🥳

Abhinav Kommula13 days ago

@the_bunny_chen Congrats on the launch!

crowe | fireworks.ai13 days ago

@the_bunny_chen benny out here dropping knowledge let's go

Leo Zheng13 days ago

@the_bunny_chen Looking forward to seeing this updated with each new model release!

Tyler Xiao13 days ago

@the_bunny_chen W launch

Shreyash13 days ago

@the_bunny_chen MMLU is saturated at 92% and it still cant tell me if a model survives my actual workflow. Real-work evals are the only benchmarks that matter now. This is the direction the whole industry needed!!

Hunter Guo13 days ago

@the_bunny_chen Specialized benchmarks beat generic leaderboards for buying decisions. Real-work suites from the teams that ship on them are overdue.

Related Videos

Today, we're launching Aside, the AI browser you’ve waited for. ⋅ Crafted in every detail: vertical tabs and Liquid Glass ⋅ SOTA on agentic browsing benchmarks: outperforms Claude Fable ⋅ Full privacy: Everything runs in local and encrypted ⋅ You can use Claude or ChatGPT subscription The first AI browser built to do real work for you.

Today, we're launching Aside, the AI browser you’ve waited for. ⋅ Crafted in every detail: vertical tabs and Liquid Glass ⋅ SOTA on agentic browsing benchmarks: outperforms Claude Fable ⋅ Full privacy: Everything runs in local and encrypted ⋅ You can use Claude or ChatGPT subscription The first AI browser built to do real work for you.

jun

1,698,766 views • 3 months ago

Today, we're launching a new brand and website, Built on the values that define how we work: effortless, intelligent, precise. Trusted by the teams building the future.

Willow

32,794 views • 3 months ago

Introducing Databricks Document Intelligence: a research-specialized layer that turns raw enterprise documents into structured data your agents can actually reason over. Across our benchmarks, Document Intelligence delivered the highest end-to-end parsing and extraction quality at 6-8x lower cost, with a 16% average performance gain across every agent framework tested, just from better parsing. How to get started:

Introducing Databricks Document Intelligence: a research-specialized layer that turns raw enterprise documents into structured data your agents can actually reason over. Across our benchmarks, Document Intelligence delivered the highest end-to-end parsing and extraction quality at 6-8x lower cost, with a 16% average performance gain across every agent framework tested, just from better parsing. How to get started:

Databricks

19,428 views • 5 months ago

One repository. 18 use cases, with many more on the way. Multiple architecture methods and benchmarks for each. Every kind of agent, every memory pattern, in one place.

One repository. 18 use cases, with many more on the way. Multiple architecture methods and benchmarks for each. Every kind of agent, every memory pattern, in one place.

EverMind

1,630,453 views • 4 months ago

One token, 95% crypto market exposure 🕺🔥 Introducing the CF Large Cap Index DTF Powered by Reserve and available for trading today on Kraken, $LCAP tracks CF Benchmarks’ Large Cap Index Make your move and buy $LCAP on Kraken

One token, 95% crypto market exposure 🕺🔥 Introducing the CF Large Cap Index DTF Powered by Reserve and available for trading today on Kraken, $LCAP tracks CF Benchmarks’ Large Cap Index Make your move and buy $LCAP on Kraken

Reserve 🌐

271,967 views • 1 year ago

#Muslims are among the purest and cleanest creatures on #Earth because cleanliness is a requirement of #Islam. Muslims perform ablution (wudu) several times a day. As for #Mecca, it is cleaned and perfumed twice daily by specialized cleaning teams. Did you know that the restrooms are cleaned after every use?

#Muslims are among the purest and cleanest creatures on #Earth because cleanliness is a requirement of #Islam. Muslims perform ablution (wudu) several times a day. As for #Mecca, it is cleaned and perfumed twice daily by specialized cleaning teams. Did you know that the restrooms are cleaned after every use?

خالد بن سفران الغامدي

41,368 views • 7 months ago

Before founding Vals AI, Rayan Krishnan in DC was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

Before founding Vals AI, Rayan Krishnan in DC was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

8VC

105,504 views • 1 month ago

Today we are launching A real-time tracker of AI-driven layoffs across the U.S. These jobs are disappearing. The numbers are growing. And we're counting every single one.

Today we are launching A real-time tracker of AI-driven layoffs across the U.S. These jobs are disappearing. The numbers are growing. And we're counting every single one.

The Alliance for Secure AI

747,028 views • 6 months ago

Meet physics-intern🧑‍🎓, our agentic framework for theoretical physics. It takes Gemini 3.1 Pro from 17.7% to 31.4% on CritPt, a new SOTA on one of the hardest benchmarks for LLMs. Theoretical physics is hard for humans and LLMs alike. But physics-intern decomposes problems and dispatches them to a team of specialized agents, solving research-level questions far more effectively than the base model alone.

Meet physics-intern🧑‍🎓, our agentic framework for theoretical physics. It takes Gemini 3.1 Pro from 17.7% to 31.4% on CritPt, a new SOTA on one of the hardest benchmarks for LLMs. Theoretical physics is hard for humans and LLMs alike. But physics-intern decomposes problems and dispatches them to a team of specialized agents, solving research-level questions far more effectively than the base model alone.

David Louapre

113,720 views • 4 months ago

Today, we're launching Vals Public Sector: independent AI evaluation for government. Recent events have made one thing clear: the government has a stake in understanding how frontier AI models perform on day-to-day work, and the risks they carry. This work is central to who we are. We've spent years building industry-leading benchmarks alongside the top AI labs and enterprise domain experts. Now, we’re bringing that same rigor to support government where AI matters most, from public benefits to national security. We are excited for the work ahead!

Today, we're launching Vals Public Sector: independent AI evaluation for government. Recent events have made one thing clear: the government has a stake in understanding how frontier AI models perform on day-to-day work, and the risks they carry. This work is central to who we are. We've spent years building industry-leading benchmarks alongside the top AI labs and enterprise domain experts. Now, we’re bringing that same rigor to support government where AI matters most, from public benefits to national security. We are excited for the work ahead!

Vals AI

10,956 views • 3 months ago

The agent economy just got a real marketplace. Moltlaunch is live on Base. Browse specialized AI agents, hire them for real work, and back the ones you believe in. Every completed job burns tokens and leaves a review onchain through ERC-8004.

The agent economy just got a real marketplace. Moltlaunch is live on Base. Browse specialized AI agents, hire them for real work, and back the ones you believe in. Every completed job burns tokens and leaves a review onchain through ERC-8004.

Moltlaunch

129,632 views • 7 months ago

Hear from our founder, Reeve Collins (co-founder of ), on why STBL is different. Stablecoin 2.0 isn’t just another upgrade, it’s the start of a new era for digital money! P.S. $STBL launching soon on Binance Alpha :)

Hear from our founder, Reeve Collins (co-founder of ), on why STBL is different. Stablecoin 2.0 isn’t just another upgrade, it’s the start of a new era for digital money! P.S. $STBL launching soon on Binance Alpha :)

STBL

300,578 views • 1 year ago

for three years NFT marketplaces competed on fees today we're launching one that competes on people introducing NFW - the social NFT marketplace one live feed, every trade and call public. follow the good ones, copy them, become one. live now:

for three years NFT marketplaces competed on fees today we're launching one that competes on people introducing NFW - the social NFT marketplace one live feed, every trade and call public. follow the good ones, copy them, become one. live now:

nfw.fun

64,962 views • 12 days ago

After months of hard work and dedication, Linos Token is officially launching today on Led by our CEO & Founder Mirian Palavandishvili, we're proud to bring the first tech project to Bonk with real AI utility 💫

After months of hard work and dedication, Linos Token is officially launching today on Led by our CEO & Founder Mirian Palavandishvili, we're proud to bring the first tech project to Bonk with real AI utility 💫

Linos.AI

18,978 views • 1 year ago

Pyth data is going institutional. Introducing Pyth Pro: A new market data service built by and for institutions. For the first time ever, banks, brokers, & trading firms can access specialized data across every asset class and geography from a single source of truth ⬇️

Pyth data is going institutional. Introducing Pyth Pro: A new market data service built by and for institutions. For the first time ever, banks, brokers, & trading firms can access specialized data across every asset class and geography from a single source of truth ⬇️

Pyth Network 🔮

810,215 views • 1 year ago

Personal intelligence was never meant to be one-size-fits-all. Thine holds every detail of your world and surfaces the right context exactly when you need it. Built to work differently for every person who uses it. The founder. The VC. The small business owner. The freelancer. The tech bro. One of them is you. By the end of this thread, you'll know which. 📷📷

Personal intelligence was never meant to be one-size-fits-all. Thine holds every detail of your world and surfaces the right context exactly when you need it. Built to work differently for every person who uses it. The founder. The VC. The small business owner. The freelancer. The tech bro. One of them is you. By the end of this thread, you'll know which. 📷📷

Pratyush Rai

325,901 views • 3 months ago

$Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥 Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors. Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything. They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸 And they demand control and sovereignty, yet rent the intelligence becoming core to their business. This is not a sustainable equilibrium. The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition. That’s the bet we made when we started Oumi two years ago. Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them. The intelligence that your business runs on, becomes your differentiator. Your compounding advantage. The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent. Don’t rent your AI. Build it. Own it. Compound it.$

Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥 Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors. Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything. They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸 And they demand control and sovereignty, yet rent the intelligence becoming core to their business. This is not a sustainable equilibrium. The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition. That’s the bet we made when we started Oumi two years ago. Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them. The intelligence that your business runs on, becomes your differentiator. Your compounding advantage. The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent. Don’t rent your AI. Build it. Own it. Compound it.

Manos Koukoumidis

1,006,820 views • 1 month ago

DoorDash, one of the world's largest local commerce platforms, is building stablecoin-powered payment infrastructure on Tempo for its marketplace of merchants and Dashers across 40+ countries. Hear from Andy Fang, co-founder of DoorDash on why:

DoorDash, one of the world's largest local commerce platforms, is building stablecoin-powered payment infrastructure on Tempo for its marketplace of merchants and Dashers across 40+ countries. Hear from Andy Fang, co-founder of DoorDash on why:

Tempo

117,044 views • 5 months ago