Real-time, local conversational AI is how we will democratize intelligence. We just ported LFM3-Omni-2B by @LiquidAI to run fully offline on my iPhone 18 Pro Max at ~80 tok/s, powered by custom WebGPU kernels in a React Native app. Should we release this on the App Store? 👀

My favorite part of this project is that it's a perfect example of WebGPU kernels running *outside* of a browser at native speeds! Huge thanks to @wcandillon for building react-native-webgpu! 🤗 Learn more about our WebGPU kernels:

@liquidai Please release it (also for Android)! 🙏

@liquidai so cool!

@liquidai Apple should just make it the new Siri

@liquidai So cooool, glad we will still have access to cookie recipes after the apocalypse ☺️

@liquidai Wait you did this with webgpu What kind of steroids are you giving Claude

@liquidai yes.

@liquidai Love this UI! Did you build the ui your own?

@liquidai How close does the same kernel get in mobile Safari? Curious how much of the 80 tok/s is the kernels and how much is just getting out of the browser.

@liquidai This is amazing! Already played with it in the browser: I wonder what it will take for this to work on a Raspberry Pi. I ran LFM3 1.7B at 3.1t/s on a raspi using a custom lama.cpp build, but I reckon this would be much trickier…

@liquidai What the temperature of the phone during inference?

@liquidai Apple Intelligence cooked ❌ Apple Intelligence is cooked ✅

@liquidai i know a founder who already put his own on-device iphone assistant through app store review and a public beta. couldn't get a dm through so tap below for the intro

@liquidai Yes, SI conversations that work in airplane mode deserve a spot on the iPhone home screen 🩷

@liquidai voice notes on a flight would be my first use. i’d want to dictate and ask about them with airplane mode still on.

@liquidai 80 tok/s on a 2B is the easy headline. The numbers I'd want: time to first audio, and tok/s at minute 10 once the phone is thermally throttled. Sustained decode on mobile usually lands well below the first-minute burst.

@liquidai the whole thread in one line: 2b weights, custom webgpu kernels, zero api bill. the other side, measured here: 124.7 tok/s on a remote flash model, billed per token, and a 27b in 8 gb that only fits because the weights are 4 bit. context breaks before memory does.

@liquidai Ship it! How big is the download with the model bundled?

@liquidai Absolutely

@liquidai Yes!

@liquidai How did you make the model download user friendly though?

@liquidai 80 tok/s fully offline on a phone is the part that matters, no server bill and nothing leaving the device 📱 the App Store crowd would eat that up

@liquidai ship it, and post how long the battery lasts in a real conversation

@liquidai Yes, ship it. 80 tok/s offline on a phone is great. How hot does it get after ten minutes of talking?

@liquidai 80 tok/s offline on a phone. yes, release it

@liquidai This kind of pipeline is already implemented by major players via the cloud. If you can build it on Omni and the tests show a decent UI without lag, latency, or iPhone meltdowns, then I'm all for it!

@liquidai 🫡

@liquidai Ship it, and keep the airplane icon in every screenshot. What does ten minutes of talking do to the battery?

@liquidai 80 tok/s fully offline on the phone is wild, does it stay that fast once you are actually off wifi?