Introducing ImageJevBench 🥇Jev-Omni 🥈decider-2b-vision 🥉Reflex 4B The video shows 89 public synthetic images. Running the top 5 models on all these images cost $0.0217. You understand how disruptive this is?

put me in coach

oh yes!!! @benchmarkheaven put him into the benchmark: glance-vlm speedlab

@benchmarkheaven did you set up an agent you can trigger from x to add models to the bench?

@benchmarkheaven yes

@benchmarkheaven awesome

@benchmarkheaven He's called Harold, all my agents know him. I can just tell Claude or Devin or Codex to let Harold do it. He has a lovely personality, he's an accountant. The right guy for a benchmark collection I guess.

@benchmarkheaven looking forward to seeing the results! are you able to monitor what’s going etc as he adds new models?

@benchmarkheaven Yes, sure. I have review gates etc

Here are some examples of typical tasks that image Jev-class models can solve. A response typically takes less time than the blink of an eye and costs almost nothing. This unlocks a wealth of use cases that were previously uneconomical to solve. Think industrial automation, robotics, IoT, and document management and so much more.

Dude you are cooking recently! 🧑‍🍳

Thanks dude. Just starting. Will cook way more. 😄

I follow with great interest.

🙏

jev omni? when did that launch?!

Great work !

Thanks

Running five top models for under three cents on 89 images is insane, especially with sub millisecond latency

yes, it's incredible, isn't it? just think about the new business ideas this will unlock

Jev alternative when using images tool to check out later

Gonna use decider 2b vision at work

crazy results

Yes

Cheap vision evals change the operating question from ‘can it see’ to ‘when does it fail under distribution shift.’ A public image set with a cost line is how operators budget verification.

this is exactly what i was looking for - now if you can please host a routing platform that would be great

coming soon

@skalskip92 you need to see this

Amazing!

💯

$0.0217 to run top 5 across 89 images is wild efficiency

Florian - does the $0.0217 hold past 89 images?

how can new models be submitted ?

write me or @benchmarkheaven a DM or just post a link to your model here asking for being added. I should maybe add a submission section on the JevBench page as well. @benchmarkheaven, please build this.