Introducing the Decision Index 0.1 ⚖️ a rigorous leaderboard comparing jev with 30+ open weights decision models 35+ benchmarks. asking 130K questions to each model testing knowledge 🧠, automation ⚙️, understanding 🤔and even creativity 🎨

jev still leads open models, with some margin in case of overall knowledge (probably a bigger model), however, on tool use/automation/retrieval and classification, it's close!

the immediate models after jev are inference techniques that allow for one-pass inference of Qwen3.8-27B and diffusion gemma, no fine tuning in 3rd place, Decider 35B-A3B, a strong fine-tune of Qwen3.5 35B base (!)

37 benchmars across 5 categories were applied for every model, 130K decisions per model all of them ran on the same hardware: 1x RTX 6000 PRO

The benchmark is meant to be relatively challenging, so that it's not saturated from the get-go. This is how jev fares on the different categories of tasks specially good for tools and automation (makes sense!)

Jev still struggles in puzzle & reasoning games, like chess. It also didn't do super well on super hard document retrieval tasks (that normally require reasoning models),recognizing chords from musical notes, and fine-grained sentiment analysis

jev does particularly well on knowledge based questions up to graduate level, python function choosing, tool calling and model routing

GLiNER 2.5 by @fastinoAI leads the tiny frontier of <350M models the Kev family by @jaredpalmer lead 2 categories: 350M-700M and 3B-10B Decider 2B by @notmapika is in the < 3B sweet spot Jevfire's Qwen3.8 27B single-pass implementation is the best open weights jev-like

@jaredpalmer @notmapika The performance x latency pareto frontier also contains openvons Qwen3-4B as a super fast yet mighty model, and @mmastrac already classic and legendary diffusiongemma jev-like implementation

@jaredpalmer @notmapika @mmastrac everything is open source and reproducible, the methodology is available here and the code is on github!

@jaredpalmer @notmapika @mmastrac calibration is up too now!

Nice! can you add this too:

will do!

Thank you!

I want to learn how to make this animation

playright + fable + human feedback loop!

Curious to see where this would rank.

A Decision Index that refuses truncation / prompt-tuning and treats unanswered as wrong is the kind of eval culture this category needs. If decision models are a real layer, we should score them like production routers — volume, failure modes, not vibes.

Been benchmarking Jev alts for a few days and I love yours! Feel freee to check mine out and steal. I consider all my benchmarking MIT

How to submit a model there for benchmarking ?

which model?

A Decision Index across 30+ open-weight models is useful. Buyers need more than leaderboard vibes when they pick what runs on their own GPUs. Which test surprised you most?

Thank you! So useful

For agent routing, I'd love a breakdown of errors that trigger the wrong tool versus errors that simply defer to a human. Two models with the same average score can have very different operating costs. Could the index expose that distinction for automation tasks?

Impressive. I am curious and challenges you had since models aren't very deterministic - a good answer on one question could be bad the next round. Hoping to solve be the volume of questions?

I don’t think they are decision modes. It is up to u to ask yourself what questions to put to Jev or an LLM to help make decisions better.

This is good man - I am dropping a new multimodal model soon.. I will use this for the benchmarks

Great work. Is it fully non-synthetic? looks like?

@multimodalart Love seeing rigorous open-weights evals. We put Mark 1x-9B out on HF + ModelScope for the self-host crowd; hosted chat is the same family. Live on PH:

testing 130k questions proves the index isn't just marketing fluff.

130,000 questions per model, 30 open models against Jev. The leaderboard is longer than most of the posts about it.