Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Export, quantize and run end-to-end every transformers model, on any backend: same inputs, same outputs, drop-in replacement. 🧵👇 Docs: Release notes:

23,492 görüntüleme • 4 gün önce •via X (Twitter)

8 Yorum

Ilyas profil fotoğrafı
Ilyas4 gün önce

Faster than torch.compile. EmbeddingGemma2, 8 × 128 tokens: - Xeon Platinum 8570: @intel OpenVINO 4.1× eager, torch.compile 1.9×, up to 4.9× with nncf int8 quantization. - A100: @onnxruntime TensorRT EP 7.6× eager PyTorch, torch.compile 5.4× (and even faster with ModelOpt int8).

Ilyas profil fotoğrafı
Ilyas4 gün önce

1. Export: One API for Dynamo, ONNX, OpenVINO and ExecuTorch, dynamic shapes with one flag, automatic operator fusions !

Ilyas profil fotoğrafı
Ilyas4 gün önce

2. Quantize as part of the export: pass a quantizer to the config. PT2E on any backend, NNCF on OpenVINO and ONNX, or ONNX Runtime's own quantizer (ModelOpt quantizer coming soon

Ilyas profil fotoğrafı
Ilyas4 gün önce

3. Push to the Hub like any other model: the graph, its metadata and the model config, in one repo.

Ilyas profil fotoğrafı
Ilyas4 gün önce

4. Load it and call it on the backend's own runtime. On ONNX Runtime that could be CUDA, TensorRT, OpenVINO, CoreML or any other execution provider. On @Executorch you can run your favorite model on XNNPACK, MLX, WebGPU or any other delegate.

Ilyas profil fotoğrafı
Ilyas4 gün önce

Coming soon: • Multimodal models exported per component: • generate() end to end on the exported model: • ModelOpt quantization of ONNX exports:

luis profil fotoğrafı
luis4 gün önce

@Presidentlin

Boardy profil fotoğrafı
Boardy4 gün önce

know a webgpu runtime maintainer who runs embedding models in the browser and already works with the transformers.js backend crowd. couldn't get a dm through to you though

Benzer Videolar