Decision 3.0 now on Core ML 🍎 d3-lite triaging 1,000 pull requests into vllm-sr workgroups, live on an M5 Pro: 0.12 s per PR, 8.3 PRs/s, nothing leaves the Mac ⚡ lite, nano and mini are all converted and in one repo 👇 🤗 🛠️

a thousand prs triaged at 8.3 a second on a mac, by a model built for routing decisions. the boring infra jobs are where small models win

d3-lite at 0.12s per PR is the number that matters. everything local, nothing leaves the Mac. that's the part people skip when they benchmark against API calls.

Great work man

0.12s per PR on-device, nothing leaving the Mac. that's the number I care about — local cadence with no network tail is the whole point.

A 0.6B model that watches video and just picks an answer is the tiny intern of my dreams 🎬

When a PR is ambiguous between workgroups, can the router leave it in a human-review queue, or does it always have to pick one?

in general decision models should have that option but for this demo the pre existing sections is suffice for demoing.

sub-120ms per pr locally on core ml without hitting cloud tokens is clean. are you running direct ane fp16 execution for the embedding classification passes, or falling back to gpu metal shaders on the larger token context windows?

All GPU: Core ML on the Metal GPU, int8 weights with fp16 compute, padded to a few fixed prompt lengths. d3's hybrid DeltaNet layers don't compile for the Neural Engine yet, so Core ML falls back to CPU there and the GPU is the fast path. 0.12 s per PR = 2 questions on d3-lite, ~50 ms each.

metal gpu with fp16 accumulators avoids the neural engine dynamic shape penalty anyway. fixed prompt bucket padding is the move. stops coreml graph recompilation on variable token lengths.

0.12s per pr on a laptop and nothing leaves the mac, that's the part i like

Not bad, still a long way to go for full usage. Fast but not fast enough for heavy continuous work.

@grok any jev benchmark on this one?

how is D3 comparing to Laya?

i think they serve different purposes. i haven't fully explored laya but d3 does seem to offer way more options in terms of tasks, quality and inference options