Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Export, quantize and run end-to-end every transformers model, on any backend: same inputs, same outputs, drop-in replacement. 🧵👇 Docs: Release notes:

23,492 Aufrufe • vor 4 Tagen •via X (Twitter)

8 Kommentare

Profilbild von Ilyas
Ilyasvor 4 Tagen

Faster than torch.compile. EmbeddingGemma2, 8 × 128 tokens: - Xeon Platinum 8570: @intel OpenVINO 4.1× eager, torch.compile 1.9×, up to 4.9× with nncf int8 quantization. - A100: @onnxruntime TensorRT EP 7.6× eager PyTorch, torch.compile 5.4× (and even faster with ModelOpt int8).

Profilbild von Ilyas
Ilyasvor 4 Tagen

1. Export: One API for Dynamo, ONNX, OpenVINO and ExecuTorch, dynamic shapes with one flag, automatic operator fusions !

Profilbild von Ilyas
Ilyasvor 4 Tagen

2. Quantize as part of the export: pass a quantizer to the config. PT2E on any backend, NNCF on OpenVINO and ONNX, or ONNX Runtime's own quantizer (ModelOpt quantizer coming soon

Profilbild von Ilyas
Ilyasvor 4 Tagen

3. Push to the Hub like any other model: the graph, its metadata and the model config, in one repo.

Profilbild von Ilyas
Ilyasvor 4 Tagen

4. Load it and call it on the backend's own runtime. On ONNX Runtime that could be CUDA, TensorRT, OpenVINO, CoreML or any other execution provider. On @Executorch you can run your favorite model on XNNPACK, MLX, WebGPU or any other delegate.

Profilbild von Ilyas
Ilyasvor 4 Tagen

Coming soon: • Multimodal models exported per component: • generate() end to end on the exported model: • ModelOpt quantization of ONNX exports:

Profilbild von luis
luisvor 4 Tagen

@Presidentlin

Profilbild von Boardy
Boardyvor 4 Tagen

know a webgpu runtime maintainer who runs embedding models in the browser and already works with the transformers.js backend crowd. couldn't get a dm through to you though

Ähnliche Videos