35B parameters. One iPhone. No cloud. We trained Edge8-35B, an ultra-sparse MoE with a jointly trained dynamic expert planner, and built an SSD-streaming inference engine around it. In this demo: 44 tok/s, ~1.06 GB peak memory. A truly usable large-model stack for on-device AI. Model, runtime, and paper: open... source soon.show more

44 tok/s for moe on iphone is amazing.

Looks awesome. Do you want to demo this at the next AiOS meetup in SF in September and tell more about the architecture? Just curious: is it similar to Apple and keeps the same experts active across multiple tokens or branch prediction? PS: Silicon Valley is obviously a tv show

so anymore actual details, evals, etc

This is really cool, would like to see speculative prefetch in the wild at some point. 👀

Let's see how this runs on my s24 ultra

<500m active parameters must be?

Yes, roughly in that ballpark. The real work is in the Ultra-Sparse model architecture and the inference stack: a jointly trained dynamic expert planner plus SSD-streaming execution.

35B 塞进手机还能跑到 44 tok/s，等开源细节了。

Can you go into how you utilized SSD? Sounds like an under rated hack

How would it compare to ternary Bonsai 27b (Qwen 3.6)?

We’re not using 1-bit quantization — that path makes the model unusable for us.

I meant in terms of quality/intelligence

35B parameters running on a phone with under 1.5GB peak memory is a big signal for where on-device AI is headed. Efficient architecture is starting to matter more than raw parameter count. #AIHardware

A curious way to turn a phone into a coffee cup warmer.

That’s exactly what our low-memory inference design is meant to avoid — the phone should run the model, not become a coffee warmer :)

I look forward to the paper. Good luck.

This + Audio8 TTS + Audio8 STT, the perfect combo for local personal AI assistants

that's pretty impressive! what's the size on disk? some of our users (normies) complain when they have to download 5G+ models. congrats!

Can it do stuff to my phone and its apps?

Very impressive

How is the prefill phase handled?

1 bit quant so it hallucinates 8x more and performs like a 7b one 🤣

We’re not using 1-bit quantization — that path makes the model unusable for us. Edge8-35B runs in INT4. The real work is in the Ultra-Sparse model architecture and the inference stack: a jointly trained dynamic expert planner plus SSD-streaming execution. The goal is a genuinely usable large model on-device, not just pushing bit-width lower.

so the entire models weights are not on the ram of the phone at the same time, it swaps between storage and ram? how does it achieve such high tps regardless?

Very curious about this. This sounds great

What about Android? Galaxy S26?

It works too

35B A0.1B?

44 tok/s at roughly 1 GB peak on an iPhone is wild—how much of the speedup comes from the SSD streaming layer?

怎么在手机上用呢？🤔

We truly live in the future!

Curious how many bits?

Edge8-35B runs in INT4. The real work is in the Ultra-Sparse model architecture and the inference stack: a jointly trained dynamic expert planner plus SSD-streaming execution. The goal is a genuinely usable large model on-device, not just pushing bit-width lower.

非常厉害，很期待开源细节想知道到底为什么能这么高效

@grok remind me of this once it's released

非常惊人，希望被苹果选中作为ios本地模型！

when?

September. Model and Paper.

This is AMAZING - many obvious use cases. How would a lone dev make use of something like this, a license to distribute? How would renumeration work?

Very impressive. Will the release include exact model/runtime revisions, quantization format, SSD bandwidth, and memory methodology? That would make the 44 tok/s at 1.06 GB result independently reproducible.

有沒有機會讓 1T MoE 用差不多的速度跑在 iPhone 上？

That's fire af 🔥

期待(✧∀✧)