you can make any open source model behave like jev with just a bit of inference engineering. it's shockingly easy. to prove it, we built a new endpoint we're calling deepseek-v4.1-flash-jev. see the demo below. here's how it's done: sglang (an inference engine) offers a scoring endpoint in addition... to the normal generation one. in scoring mode, given an input & set of possible answers, it forces the model to produce probabilities for each one. example: > input: what is most common letter in abcccde? > possible answers: a, b, c > output: (c, 0.9), (b, 0.0.5), (a, 0.05) getting the above behavior instead of streamed output is as simple as using sglang's /v1/score endpoint instead of /generate. there's just one other trick required. for deepseek, you have to add a closing think tag before the response. this forces a direct answer instead of a reasoning trace. if you want reasoning, you can do that too, but imo that makes things too slow to be worth it. dsv4.1 flash is not as good as jev, but if we had enough spare compute to experiment with this same approach for a larger model then i think the decision quality would be at least as good, if not better. also, somewhat unrelated, i think decision-making models kill all prospecting & sourcing work. i would have absolutely killed to have jev or similar when i was recruiting Mintlify. absolutely incredible.show more

striking gold on my feed at 12am on a Saturday X, the slot machine app

I need to get back to my dog puzzle now lol

does the same approach work for structured outputs like JSON schemas, or only for extended thinking? asking because the jev demos all show reasoning traces, but a lot of the real value would be in reliable tool calls with the long CoT behavior

no it doesn't. you could hack it to, but it's not great

I'm digging the average linkedin user experience bench (directly breaking tos with bots talking to the other bots breaking tos)

demo is actually earnest tho 😅. sourcing is a really difficult and important problem

i'm not judging ive got deepseek running 3 kvms for linkedin on macbooks as we speak

exactly! the problem is certainty is less accurate (you can use logprobs but not quite as good) this was the original architecture for

SGLANG CAN DO THIS? I kinda scoffed at someones prompt where they asked for a "confidence" score on the answer in the prompt but now I have to try this out.

i think inference engineering is the new web dev a lot of things become possible for application development when you understand how the underlying inference stuff works the same way a lot became possible before in the web 2.0 saas era when you understood how databases and rest APIs worked

so few people bother with inference and just use api its kinda of sad. Most people in the "real world" cant even differentiate from model capabilities vs things that the harness does and end up building some very questionable stuff.

holy moly

right?

this thing is kinda insane

yea the repo is open source. probably a $1M ARR saas product pretty easily

gonna wake up tmrw and see this hella massive how do i invest in a post

ill bet you $100 it’s under 1k likes at 10am pacific tmrw

bet 🤝 it’s going over

ah fuck

Thanks for sharing, really interesting. Do you know is this closing think tag is a thing for other models, too? Would be interesting if that can work not only for open source models 🥹

not off the top of my head, but any coding agent could tell you

yeah, Vercel is giving free usage of jev for a week, so we can just stick to it for now, and see how it works, find cool use cases and then if needed map to other models.

@benchmarkheaven include it in the benchmarks, if this team wants to opt out they can tell us and we will of course adhere to it then.

Structured output was a thing for a long time now not to mention you can also force a function call which is structured by definition, where Jev wins is its speed and cost I have measured over 95% accuracy on a very difficult very unstructured dataset at the latency of 500ms

price and latency?

too cheap for us rn

pretty cool dang how does this compare to jev’s crazy speed and cheapness? does setting up inference in this way actually make the model run faster/cheaper, or is jev just way smaller, or other shenanigans entirely?

i think jev's larger and they have b200's or b300's. this is on a rtx pro. simply running it on a b300 would likely make the speed close to equal

open source just became a race to see who can weaponize it fastest

the nice thing is you could presumably get better cache reuse between a "normal" turn and a "decision" turn compared to some hybrid stuff I've been working on today

definitely. if this post is well received and I might explain how that all works. If you have cache then it’s almost an entirely free run because all you’re doing is pre-fill, so there’s basically no new work required.

Yeah for agents I work on I am really hoping something like this becomes a dual use interface on a model. I just need popular inference providers to realize the potential of this.

Hint hint btw ;)

brother what you really need are the model order books like OpenRouter to start getting more complex and offering more markets I think this is a secondary concern beyond there just being separate order books for fast inference versus regular speed

Yeah, that’s basically what I want. same model + same cache, but this turn is a low-latency scoring request. all day i've been working on using jev for tool decisions and deep seek for generation and honestly it's really promising but the economics would be insanely better if i was getting any cache reuse and that becomes possible with something like you're showing

yea this reuses cache already. get enough friends to pay the $7/hr for the b200 and we’ll do it lol

lol yeah somehow I think i'm early to the party but give it like 6 months

Pretty cool.

making an open source model act like jev via inference engineering is wild deepseek-v4.1-flash-jev as a demo endpoint is a sharp proof

I made it opensource version and called it OpenJev that works on consumer grade GPU.

very cool

Can I include your model into JvBench v1? The first Jev Benchmark. Will be ran and published by @benchmarkheaven. Planned first rollout of the leaderboard is toady.

@benchmarkheaven im about to sleep, but if you can reach out to me in the morning over DM’s, then I can hook you up right now we don’t have the model of the running since obviously we have other revenue generating use cases for the compute lol

@benchmarkheaven thanks, good night

🔥

Interesting that the scoring endpoint flips the model into a decision engine, the think tag trick feels like a cheap stopgap for pure GTO style

absolute banger

i aim to offer value

this is so cool

high praise coming from you 🫶

the scary eval here is candidate set quality. if the recruiter forgot the right person before /v1/score runs, a perfect probability distribution just makes the miss look scientific. log who entered the set, who got filtered, and why.

you admit the bigger-model version is untested, so 'any model' overstates it.

The problem is the calibration

the real unlock is that the decision boundary becomes replayable. explicit candidates plus scores turn every routing miss into a regression test instead of another pile of prose to grade.

But is the architecture as effective and fast/cheap as Jev’s which is optimized for producing a single probability vector, not a serial strings of tokens, one after another?

The best part is you get a confidence number, not just an answer. Auto-act above 0.9, send the rest to a human or a slower reasoning pass. Just test calibration first. Token probabilities are often more confident than they deserve.

The demo is clean. The useful cut is concurrency: does the simpler endpoint stay predictable when one model instance gets busy, or do queueing and memory become the real bottleneck?

the scoring mode with sglang sounds really powerful for getting precise outputs, thanks for sharing how it works