We distilled clef-flash into a 0.6B model and supporting ANE option. 1,000 support tickets triaged on a M5 pro in 57 s: team, urgency and refund for each, 48 ms per ticket. Within a few points of the 9B on classification. 🤗 💻

@ClementDelangue multi-modal search feels underrated. text-only RAG breaks the moment someone asks about a chart or diagram buried in page 47 of a PDF.

Opensource?

yes

can i dm you man if you allow me ?

Fire!

how does the 0.6B compare on urgent tickets and refund requests separately? overall accuracy can hide the expensive misses in those two labels

thats a good point, i will do that as a follow up when d4 is released

If precision is still good that would be a real deal

The ANE option gets more interesting with a local LLM already using the GPU. Would love to see ticket latency with both running, rather than each benchmarked alone.

Nobody cares about saving 10 dollars bro

@ClementDelangue 0.6B triaging 1000 tickets in 57s is impressive, but what did accuracy look like compared to the original model? curious where the distillation trade-off landed

omni next? asking for a friend.

unfortunately it doesn't fit on my m5 pro ram

Looks promising, any detailed blogpost on particular examples anytime soon?

not at the moment but if this post gages enough traction i might do a follow up blog

crazy good, follow up blogpost on csv example please, with customers like tickets, let's say, from customer service of ecommerce website and how your model solves it at scale.

i will do that when d4 is released