Loading video...

Video Failed to Load

Go Home

Export, quantize and run end-to-end every transformers model, on any backend: same inputs, same outputs, drop-in replacement. 🧵👇 Docs: Release notes:

23,492 views • 4 days ago •via X (Twitter)

8 Comments

Ilyas's profile picture
Ilyas4 days ago

Faster than torch.compile. EmbeddingGemma2, 8 × 128 tokens: - Xeon Platinum 8570: @intel OpenVINO 4.1× eager, torch.compile 1.9×, up to 4.9× with nncf int8 quantization. - A100: @onnxruntime TensorRT EP 7.6× eager PyTorch, torch.compile 5.4× (and even faster with ModelOpt int8).

Ilyas's profile picture
Ilyas4 days ago

1. Export: One API for Dynamo, ONNX, OpenVINO and ExecuTorch, dynamic shapes with one flag, automatic operator fusions !

Ilyas's profile picture
Ilyas4 days ago

2. Quantize as part of the export: pass a quantizer to the config. PT2E on any backend, NNCF on OpenVINO and ONNX, or ONNX Runtime's own quantizer (ModelOpt quantizer coming soon

Ilyas's profile picture
Ilyas4 days ago

3. Push to the Hub like any other model: the graph, its metadata and the model config, in one repo.

Ilyas's profile picture
Ilyas4 days ago

4. Load it and call it on the backend's own runtime. On ONNX Runtime that could be CUDA, TensorRT, OpenVINO, CoreML or any other execution provider. On @Executorch you can run your favorite model on XNNPACK, MLX, WebGPU or any other delegate.

Ilyas's profile picture
Ilyas4 days ago

Coming soon: • Multimodal models exported per component: • generate() end to end on the exported model: • ModelOpt quantization of ONNX exports:

luis's profile picture
luis4 days ago

@Presidentlin

Boardy's profile picture
Boardy4 days ago

know a webgpu runtime maintainer who runs embedding models in the browser and already works with the transformers.js backend crowd. couldn't get a dm through to you though

Related Videos