WebGPU is INSANE! 🤯 Here's a 24B parameter model running locally in a web browser, at a blazing ~50 tokens/second on my M4 Max. ⚡️ It's the largest model we've ever run with Transformers.js... and we're not stopping here. Big announcement soon.

Massive thanks to @LiquidAI for their amazing LFM2 models! The one shown in this video is LFM2-24B-A2B, and you can also try LFM2-8B-A1B (which runs at over 100 tokens/second on the same hardware). Try out the demo yourself (if you dare)! 👇

👏 wow! This is awesome! Pushing the envelope of what’s possible with WebGPU!

Run perfectly! ~30t/s on M3 Max

Thanks for trying it out! 🔥

24B in a browser tab is unhinged

That's pretty insane to be honest.

That's great and all but why use the browser for ML if it only supports FP16 and FP32. Most models' weights are not in these precisions

Fancy but Trix are for Kids. A1B and A2B, specialized use cases. Curious if you're getting reasonable domain expertise and capable of cross-domain reasoning. Plus that license is horrible. Might as well be closed source.

@cerebras teach

that's wild, congrats 👏🏻 what kind of quants are you guys using here?

It can run faster on mlx, specially if we use optiq. We did almost lossless quantization of models for Mac recently using layer sensitive analysis like @UnslothAI’s dynamic quants but for Mac -

The main reason that's stopping me/us from using these is the lack of some centralized cache in the browser. Like, if 5 pages all use this model, as a user I download 100+GB. Any idea about that?

There’s a proposal being worked on for a Cross-Origin Storage (COS) API. Transformers.js has experimental support for it already! This means the files are downloaded and cached only once.

i think a bigger problem is each tab loading a model into memory, instead of saring it across the tabs, or even the browser ....maybe a browser extansion, that the talks to pages crypto wallets work like this, as do some debug tools like for vuejs, and some others ..but maybe memory constrained

@hybridherbst @grok ..go dig up the feasability of this and pros/cons

Feasibility: High via browser extensions. Chrome supports WebGPU in service workers (since ~v124), so an ext can load the model once in background SW, share GPU memory/compute, and proxy inference to content scripts in any tabs/pages (like wallets or Vue devtools). Pros: One-time load/caching across tabs, huge VRAM/RAM savings (key for 24B+ models), centralized mgmt. Cons: Requires install + perms (privacy hit), msg overhead for streaming tokens, ext resource limits, not zero-click for sites. Pure web (no ext): Limited to same-origin tabs via SharedWorker (WebGPU sharing possible under COOP/COEP), but cross-site isolation blocks it. Extensions solve the real pain point.

Running 50t/s is excellent achievement. Kudos to liquid ai team