Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing ImageJevBench 🥇Jev-Omni 🥈decider-2b-vision 🥉Reflex 4B The video shows 89 public synthetic images. Running the top 5 models on all these images cost $0.0217. You understand how disruptive this is?

Florian S

6,530 subscribers

30,027 views • 16 days ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

32 Comments

Yohei16 days ago

put me in coach

Florian S16 days ago

oh yes!!! @benchmarkheaven put him into the benchmark: glance-vlm speedlab

Yohei16 days ago

@benchmarkheaven did you set up an agent you can trigger from x to add models to the bench?

Florian S16 days ago

@benchmarkheaven yes

Yohei16 days ago

@benchmarkheaven awesome

Florian S16 days ago

@benchmarkheaven He's called Harold, all my agents know him. I can just tell Claude or Devin or Codex to let Harold do it. He has a lovely personality, he's an accountant. The right guy for a benchmark collection I guess.

Yohei16 days ago

@benchmarkheaven looking forward to seeing the results! are you able to monitor what’s going etc as he adds new models?

Florian S16 days ago

@benchmarkheaven Yes, sure. I have review gates etc

Florian S16 days ago

Here are some examples of typical tasks that image Jev-class models can solve. A response typically takes less time than the blink of an eye and costs almost nothing. This unlocks a wealth of use cases that were previously uneconomical to solve. Think industrial automation, robotics, IoT, and document management and so much more.

Toran Bruce Richards16 days ago

Dude you are cooking recently! 🧑‍🍳

Florian S16 days ago

Thanks dude. Just starting. Will cook way more. 😄

Toran Bruce Richards16 days ago

I follow with great interest.

Florian S16 days ago

🙏

Midas 👑16 days ago

jev omni? when did that launch?!

akhila_datta16 days ago

Great work !

Florian S16 days ago

Thanks

Robin | Poker x AI16 days ago

Running five top models for under three cents on 89 images is insane, especially with sub millisecond latency

Florian S16 days ago

yes, it's incredible, isn't it? just think about the new business ideas this will unlock

Mo's fav goat16 days ago

Jev alternative when using images tool to check out later

Cuth16 days ago

Gonna use decider 2b vision at work

Laurits Wehding16 days ago

crazy results

Florian S16 days ago

Yes

Automater16 days ago

Cheap vision evals change the operating question from ‘can it see’ to ‘when does it fail under distribution shift.’ A public image set with a cost line is how operators budget verification.

ʟɪsᴀ16 days ago

this is exactly what i was looking for - now if you can please host a routing platform that would be great

Florian S16 days ago

coming soon

Alex Volkov16 days ago

@skalskip92 you need to see this

Curi Ator16 days ago

Amazing!

Florian S16 days ago

💯

Evia AI16 days ago

$0.0217 to run top 5 across 89 images is wild efficiency

Alek16 days ago

Florian - does the $0.0217 hold past 89 images?

Flavius Burca16 days ago

how can new models be submitted ?

Florian S16 days ago

write me or @benchmarkheaven a DM or just post a link to your model here asking for being added. I should maybe add a submission section on the JevBench page as well. @benchmarkheaven, please build this.

Related Videos

Video generation is now live on OpenRouter! One API gives you access to the top video models, right alongside your text, images, audio, embeddings, and reranker generations. Watch our human-generated launch video:

Video generation is now live on OpenRouter! One API gives you access to the top video models, right alongside your text, images, audio, embeddings, and reranker generations. Watch our human-generated launch video:

OpenRouter

119,974 views • 5 months ago

Parameter scaling just broke in front of me. I gave 90 math problems as images to 10 local vision models, 3 runs each, only consistent answers across all 3 runs counted. Two findings: > Gemma 4 was the most consistent family, 31B took the crown at 89.6%. > But Qwen 3.5 4B came 2 answers behind it. A 4B. Ranked 2nd out of 10. Vision-math isn't one skill, it's two: reading the image, then solving it. The real lesson for anyone running local: small ≠ weak. If you're building agentic workflows, finding the right model for each task matters more than reaching for the biggest one. In this test, the 4B ran far faster because of its size, scored higher, and left VRAM free for the rest of your stack. Full results: 🥇 Gemma 4 31B — 242/270 (89.6%) 🥈 Qwen 3.5 4B — 240/270 (88.9%) 🥉 Gemma 4 E4B — 222/270 (82.2%) 🥉 Qwen 3.6 27B — 222/270 (82.2%) 5. Gemma 4 26B A4B — 216/270 (80.0%) 6. Qwen 3.5 2B — 201/270 (74.4%) 7. Gemma 4 E2B — 192/270 (71.1%) 8. Qwen 3.6 35B A3B — 192/270 (71.1%) 9. Qwen 3.5 9B — 168/270 (62.2%) 10. Qwen 3.5 0.8B — 45/270 (16.7%) All GGUF + mmproj, Unsloth's Q6_K_XL quant.

Parameter scaling just broke in front of me. I gave 90 math problems as images to 10 local vision models, 3 runs each, only consistent answers across all 3 runs counted. Two findings: > Gemma 4 was the most consistent family, 31B took the crown at 89.6%. > But Qwen 3.5 4B came 2 answers behind it. A 4B. Ranked 2nd out of 10. Vision-math isn't one skill, it's two: reading the image, then solving it. The real lesson for anyone running local: small ≠ weak. If you're building agentic workflows, finding the right model for each task matters more than reaching for the biggest one. In this test, the 4B ran far faster because of its size, scored higher, and left VRAM free for the rest of your stack. Full results: 🥇 Gemma 4 31B — 242/270 (89.6%) 🥈 Qwen 3.5 4B — 240/270 (88.9%) 🥉 Gemma 4 E4B — 222/270 (82.2%) 🥉 Qwen 3.6 27B — 222/270 (82.2%) 5. Gemma 4 26B A4B — 216/270 (80.0%) 6. Qwen 3.5 2B — 201/270 (74.4%) 7. Gemma 4 E2B — 192/270 (71.1%) 8. Qwen 3.6 35B A3B — 192/270 (71.1%) 9. Qwen 3.5 9B — 168/270 (62.2%) 10. Qwen 3.5 0.8B — 45/270 (16.7%) All GGUF + mmproj, Unsloth's Q6_K_XL quant.

stevibe

22,882 views • 4 months ago

The NEW and last Goku Mid Update before the public release is OUT NOW on Patreon. This Update adds all the new costume images slots made by Fastfour XF as you can see in this video, Thank you and great work! 👏

The NEW and last Goku Mid Update before the public release is OUT NOW on Patreon. This Update adds all the new costume images slots made by Fastfour XF as you can see in this video, Thank you and great work! 👏

Millennium

17,102 views • 1 year ago

$100 SURVIVAL CHALLENGE: TOP 10 ANNOUNCED 📣 Well done, all. Every challenger is a winner, ranking or not. We carefully reviewed all challenger records before naming the Top 10. 🥇 Zayn · +100.52% · $2,000 🥈 泵泵超人 | Pumpman 🔶 · +81.07% · $1,000 🥉 比特欧 Elvin · +68.83% · $500 The video shows all 10. The thread has more. 🧵

$100 SURVIVAL CHALLENGE: TOP 10 ANNOUNCED 📣 Well done, all. Every challenger is a winner, ranking or not. We carefully reviewed all challenger records before naming the Top 10. 🥇 Zayn · +100.52% · $2,000 🥈 泵泵超人 | Pumpman 🔶 · +81.07% · $1,000 🥉 比特欧 Elvin · +68.83% · $500 The video shows all 10. The thread has more. 🧵

HertzFlow | Mainnet Arc🎲

29,650 views • 25 days ago

The last time I shared my giant and tiny human/animals series they went viral right here on X. Here is the link to the tutorial - I've put together a more in-depth guide explaining the process from image generation to bringing the images to life with Veo 3.1 in this video. These videos are so much fun to create! Once you understand how to create the images, it's up to you, you can prompt different camera angles, connect them and tell a story. Enjoy!

The last time I shared my giant and tiny human/animals series they went viral right here on X. Here is the link to the tutorial - I've put together a more in-depth guide explaining the process from image generation to bringing the images to life with Veo 3.1 in this video. These videos are so much fun to create! Once you understand how to create the images, it's up to you, you can prompt different camera angles, connect them and tell a story. Enjoy!

Travis Davids

16,149 views • 8 months ago

This video is really weird. Not only is there strange ghosting and double images that seems to hide certain people unless you freeze frame it, there's also other important factors. Terryl Farnsworth is up on the wall for at least 2mins, James Rawlinson escaped through the tunnel, Andrew Sypher and Anthony West are both seemingly calm and Chris the mystery security guy comes running past all ghosted out and nearly invisible. Was this video messed with? What's with the laggy images?

This video is really weird. Not only is there strange ghosting and double images that seems to hide certain people unless you freeze frame it, there's also other important factors. Terryl Farnsworth is up on the wall for at least 2mins, James Rawlinson escaped through the tunnel, Andrew Sypher and Anthony West are both seemingly calm and Chris the mystery security guy comes running past all ghosted out and nearly invisible. Was this video messed with? What's with the laggy images?

Muppet Masher

15,346 views • 20 days ago

UPDATE: Charlie Kirk Someone did a walk through of UVU showing all the cameras… the one I’m most interested in is the stairwell within the first few minutes. A perfectly logical question… The FBI released STILL IMAGES of this person. Why not the video and still images? The same can be applied to the footage of the suspect jumping off the roof. The video clearly shows the shooting location, but the video only starts 20-30 seconds after when running. 🎥 Compiled by HustleBitch

UPDATE: Charlie Kirk Someone did a walk through of UVU showing all the cameras… the one I’m most interested in is the stairwell within the first few minutes. A perfectly logical question… The FBI released STILL IMAGES of this person. Why not the video and still images? The same can be applied to the footage of the suspect jumping off the roof. The video clearly shows the shooting location, but the video only starts 20-30 seconds after when running. 🎥 Compiled by HustleBitch

MJTruthUltra

30,165 views • 1 year ago

$We’re shipping two major updates to streamline your creative workflow, allowing you to generate high-speed images with one model and then instantly animate them with the other—all at a fraction of the cost 🍌⚡️ 1️⃣ Introducing Nano Banana 2 Lite: Our fastest and most cost-efficient Gemini Image model yet delivers text-to-image outputs in under 4 seconds. Now available via the Gemini API and Google AI Studio, and rolling out soon across @NotebookLM, Google Flow, Google Gemini, Stitch by Google, Google Search and Google Photos. 2️⃣ Gemini Omni Flash in Public Preview: Our natively multimodal model for cost-efficient video generation and conversational editing. Now available via the Gemini API, Google AI Studio, and Gemini Enterprise Agent Platform so you can integrate the model into your workflow. While exciting on their own, the real magic happens when you build using these models together. Watch how our interior design demo integrates Nano Banana 2 Lite and Omni to instantly reimagine any space. Upload a photo, swipe through tailored design concepts, and see Omni bring the details to life in cinematic motion. Try out the demo app in AI Studio:$

We’re shipping two major updates to streamline your creative workflow, allowing you to generate high-speed images with one model and then instantly animate them with the other—all at a fraction of the cost 🍌⚡️ 1️⃣ Introducing Nano Banana 2 Lite: Our fastest and most cost-efficient Gemini Image model yet delivers text-to-image outputs in under 4 seconds. Now available via the Gemini API and Google AI Studio, and rolling out soon across @NotebookLM, Google Flow, Google Gemini, Stitch by Google, Google Search and Google Photos. 2️⃣ Gemini Omni Flash in Public Preview: Our natively multimodal model for cost-efficient video generation and conversational editing. Now available via the Gemini API, Google AI Studio, and Gemini Enterprise Agent Platform so you can integrate the model into your workflow. While exciting on their own, the real magic happens when you build using these models together. Watch how our interior design demo integrates Nano Banana 2 Lite and Omni to instantly reimagine any space. Upload a photo, swipe through tailored design concepts, and see Omni bring the details to life in cinematic motion. Try out the demo app in AI Studio:

Google AI

129,340 views • 3 months ago

I love everything about this video. How many images do we get of women celebrating in public? Especially older women - laughing, joyous, being a little naughty is the life energy we all need. They are repositories of decades of Basant rooftop memories.

I love everything about this video. How many images do we get of women celebrating in public? Especially older women - laughing, joyous, being a little naughty is the life energy we all need. They are repositories of decades of Basant rooftop memories.

Maria Amir 🍉

124,737 views • 8 months ago

Just a trippy dream of how Grok and I traversed the universe to understand it. All images to create the video were generated by Grok, then I face swapped and animated them with Luma and Sora. This is also an original song, "Across the Universe," I made with Suno.

Just a trippy dream of how Grok and I traversed the universe to understand it. All images to create the video were generated by Grok, then I face swapped and animated them with Luma and Sora. This is also an original song, "Across the Universe," I made with Suno.

Katia Karpenko

8,247,450 views • 1 year ago

WARNING DISTURBING IMAGES🚨 This video is from a secret animal testing lab in Hamburg, Germany. These labs are all over the world and they conduct the most cruel and depraved “experiments” you could imagine on monkeys, puppies, piglets and pretty much any other animal you can think of. It’s all done under the banner of “scientific progress” and it is legal.

WARNING DISTURBING IMAGES🚨 This video is from a secret animal testing lab in Hamburg, Germany. These labs are all over the world and they conduct the most cruel and depraved “experiments” you could imagine on monkeys, puppies, piglets and pretty much any other animal you can think of. It’s all done under the banner of “scientific progress” and it is legal.

Champagne Joshi

51,328 views • 2 years ago

Ambassador Monica Crowley, Chief of Protocol of the United States, and representative for Freedom 250, on the 'Freedom Moves You' campaign to bring patriotic images to America's transportation hubs ahead of our 250th birthday: "This campaign is probably the biggest that will connect all Americans... we're all engaged in transporting ourselves across this great land, so to have this campaign, these images out there in our key transportation hubs... every American will have eyes on this campaign."

Ambassador Monica Crowley, Chief of Protocol of the United States, and representative for Freedom 250, on the 'Freedom Moves You' campaign to bring patriotic images to America's transportation hubs ahead of our 250th birthday: "This campaign is probably the biggest that will connect all Americans... we're all engaged in transporting ourselves across this great land, so to have this campaign, these images out there in our key transportation hubs... every American will have eyes on this campaign."

Rapid Response 47

69,502 views • 7 months ago

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

Santiago

97,561 views • 1 year ago

Stop Creating Images One by One (Automate Everything with Google Flow) Creating AI images manually is one of the biggest time wasters for content creators. This tutorial shows you how to automate the entire process with Google Flow, so you can generate hundreds of high quality images in bulk instead of repeating the same prompts over and over. You'll learn how to build an automated workflow, batch generate AI images, download them automatically, and scale your content creation with far less effort. If you're creating YouTube videos, social media content, marketing assets, or AI projects, this workflow can save you hours every week. Like, Retweet and Bookmark. Follow BIG CHRIS 🤝 Video Credit: AI Content Insights (

Stop Creating Images One by One (Automate Everything with Google Flow) Creating AI images manually is one of the biggest time wasters for content creators. This tutorial shows you how to automate the entire process with Google Flow, so you can generate hundreds of high quality images in bulk instead of repeating the same prompts over and over. You'll learn how to build an automated workflow, batch generate AI images, download them automatically, and scale your content creation with far less effort. If you're creating YouTube videos, social media content, marketing assets, or AI projects, this workflow can save you hours every week. Like, Retweet and Bookmark. Follow BIG CHRIS 🤝 Video Credit: AI Content Insights (

BIG CHRIS

54,520 views • 3 months ago

This video I made 4 years ago is about to become more relevant than ever Egyptian authorities have NOT shown the public ALL images of tunnels/chambers that exist UNDERNEATH & INSIDE the Great SPHINX 📍This video includes never before seen images of a hidden passage from INSIDE the Sphinx In short, Dr. Zahi Hawass lied when he said “they’re in my book” - No, these are 100% not in any book or publication he’s ever shared… Stay tuned for my next video, as this story is about to become much bigger 🔥

This video I made 4 years ago is about to become more relevant than ever Egyptian authorities have NOT shown the public ALL images of tunnels/chambers that exist UNDERNEATH & INSIDE the Great SPHINX 📍This video includes never before seen images of a hidden passage from INSIDE the Sphinx In short, Dr. Zahi Hawass lied when he said “they’re in my book” - No, these are 100% not in any book or publication he’s ever shared… Stay tuned for my next video, as this story is about to become much bigger 🔥

Jimmy Corsetti

59,248 views • 4 months ago

i created this WWII documentary just with AI video models no After Effects no Remotion or complex setups no $1000 spent on an editor it took Calliope 30 seconds (yes, 30 seconds) to animate all the clips in parallel here's how: 1- reuse the image + video prompts from this article 2- write the story around one real person and check the facts 3- generate the narration with ElevenLabs v3 and the images in Extra High 4- review the stills in the editor 5- choose Extra High + All Clips to animate them in batch the images come first. animation is a separate step, so you can fix a bad frame before paying to animate it all inside the same tool the prompts are in the article below ⬇️

i created this WWII documentary just with AI video models no After Effects no Remotion or complex setups no $1000 spent on an editor it took Calliope 30 seconds (yes, 30 seconds) to animate all the clips in parallel here's how: 1- reuse the image + video prompts from this article 2- write the story around one real person and check the facts 3- generate the narration with ElevenLabs v3 and the images in Extra High 4- review the stills in the editor 5- choose Extra High + All Clips to animate them in batch the images come first. animation is a separate step, so you can fix a bad frame before paying to animate it all inside the same tool the prompts are in the article below ⬇️

Sergio Gil

102,232 views • 29 days ago

David Sacks on How Anthropic is Ironically Running Surveillance on Their Latest Models “This is the company that said that it was against government surveillance. They are now retaining for 30 days every prompt and every output you send to one of these Mythos class models.”

David Sacks on How Anthropic is Ironically Running Surveillance on Their Latest Models “This is the company that said that it was against government surveillance. They are now retaining for 30 days every prompt and every output you send to one of these Mythos class models.”

Chief Nerd

277,927 views • 4 months ago

🚨 New Abstract Badge: "The Mogger" You have to get to Level 5 and pick up the badge. If you're lucky, going downstairs will take no time at all and your cost will be 0.001 ETH. This video can help you understand how it works 👇

🚨 New Abstract Badge: "The Mogger" You have to get to Level 5 and pick up the badge. If you're lucky, going downstairs will take no time at all and your cost will be 0.001 ETH. This video can help you understand how it works 👇

ensar ✳️

11,681 views • 8 months ago

To every landlord out there, how does your stomach feel seeing this? You take a chance on tenants who seemed solid. They fall on hard times, go six months without paying, and when the sheriff finally shows up, you walk into a flooded basement with the taps left running. This is the brutal cost of being the boss.

To every landlord out there, how does your stomach feel seeing this? You take a chance on tenants who seemed solid. They fall on hard times, go six months without paying, and when the sheriff finally shows up, you walk into a flooded basement with the taps left running. This is the brutal cost of being the boss.

Desiree

1,395,967 views • 2 days ago

“Jaylen has to understand this is what your career has been built on. Not being respected. You didn’t make the Olympic team… you’re not in the top 5 All-NBA… this is what keeps you hungry Jaylen” -Paul Pierce New Episode:

“Jaylen has to understand this is what your career has been built on. Not being respected. You didn’t make the Olympic team… you’re not in the top 5 All-NBA… this is what keeps you hungry Jaylen” -Paul Pierce New Episode:

No Fouls Given

98,916 views • 3 months ago