Introducing Phi-3 WebGPU, a private and powerful AI chatbot that runs locally in your browser, powered by 🤗 Transformers.js and onnxruntime-web! 🔒 On-device inference: no data sent to a server ⚡️ WebGPU-accelerated (> 20 t/s) 📥 Model downloaded once and cached Try it out! 👇

Demo:

Update: Now with dark mode! 🔥

Is this from the v3 branch?

Yes it is! 🤗

I love how fast the video shows this going 😂

It’s loading from cache 😇

not working on chrome: An uncaught WebGPU validation error was raised: [Invalid ComputePipeline "BlockwiseMatMulNBits"] is invalid. - While Validating GetBindGroupLayout (0) on [Invalid ComputePipeline "BlockwiseMatMulNBits"]

I'm on a tablet w/out a robust GPU (surface pro) and it still works, albeit, 2 to 3 tokens a second (so not sure if it's eking out some small gains over just CPU, still impressive though)! Curious, are there ways to shorten delay before tokens start streaming?

Where is this 2.3 GB stored? How to clear it once testing is done?

Caching is done via the Web Cache API: Steps to clear the cache: 1. Open Dev tools (F12 in Chrome) 2. Go to "Application" tab 3. Open "Cache storage" in the "Storage" tab on the left 4. Right click "transformers-cache" and click "Delete"