Tetris bench: Jev vs. Laya-mlx I run the Laya model on my MacBook. It's true that the model is really fast (~84ms), but it's also much more stupid. Losing to Jev 3 out of 3 rounds.

Source:

why only run 3 times, at these costs you should run it at least 100x just to confirm

You gotta try mine 👀 Give enough game sense to Laya and it performs great for sure.

Jev is just too good

Yes, I got the same impression. Faster but dumb

... that's a fail of comparison. Don't compare timing, compare how many pieces they dropped until the end.

same here:

Similar experience when trying to run on laya - gets stuck. still pretty cool tho!

Aww but hey Jev is punk I mean pink

macbook beating the datacenter is my favourite genre

Yes, I noticed the same.

This is a useful reminder that latency and capability are separate axes. For local agents, the product question is whether the fast model is good enough for routing and control loops, while a stronger model handles the hard turns.

84ms per move is useless if the move is wrong. Raw latency wins benchmarks; the interesting question is how far quality scales with a slightly bigger local model.

mlx quantizations drop reasoning precision pretty fast once the board gets crowded.

Noticed Laya gets 6 heuristic-ranked placements in the repo, while Jev sees the full set. Could you try both with the same 6 candidates, plus a heuristic-only baseline? Curious how much of the gap survives with matched inputs.

This is exactly what people need to see faster doesn’t always mean better

its kinda clear Laya dont know the game

84ms but 0-3. Classic speed vs quality tradeoff. My take: the winning setup is routing. I'd run Laya locally for the easy moves and call Jev for the hard ones. Best of both.

Will be interesting to see how much it can be optimised, or trained to specific tasks

Speed is ego; depth is the throne. A fast dummy still loses the game.

Speed gets attention, but the intelligence gap is the real constraint. Fast wrong moves only tighten the failure loop.

Kev-4B strikes the right balance for me. I agree laya just seems to make too many mistakes

84ms is fast, but if Jev solves the Tetris mechanics and Laya doesn't, you're comparing different games. What specific moves does Laya miss?

Tetris bench where Laya is fast on the MacBook but Jev still wins the actual game is the eval people needed.

54ms to stack it wrong faster

Sub-hundred millisecond generstion on Apple Silicon makes local quantized weights feel immediate for single-token prediction. Spatial games like Tetris break lightweight models because placing a tetromino requires multi-step lookahead across grid colume heights rather than forward syntax completion. Speed masks the lack of planning depth until the board state demands evaluating branching placements.

Here we go with another open weight model that competes with Jev seems!!

I guess slow and steady wins the race 🤣

Just to be sure, did you put that model remotely? Because of you're comparing Jev through API vs a local model... I don't see that as a fair comparison.

more benchmarks like ths

84ms is nothing if it plays worse. speed only matters once accuracy is close, otherwise its just losing fast

Still needs more training then. Thanks for testing

So far this Jev thing is no differenct than a classifier and perform even worse, im confused