正在加载视频...

视频加载失败

Export, quantize and run end-to-end every transformers model, on any backend: same inputs, same outputs, drop-in replacement. 🧵👇 Docs: Release notes:

23,492 次观看 • 4 天前 •via X (Twitter)

8 条评论

Ilyas 的头像
Ilyas4 天前

Faster than torch.compile. EmbeddingGemma2, 8 × 128 tokens: - Xeon Platinum 8570: @intel OpenVINO 4.1× eager, torch.compile 1.9×, up to 4.9× with nncf int8 quantization. - A100: @onnxruntime TensorRT EP 7.6× eager PyTorch, torch.compile 5.4× (and even faster with ModelOpt int8).

Ilyas 的头像
Ilyas4 天前

1. Export: One API for Dynamo, ONNX, OpenVINO and ExecuTorch, dynamic shapes with one flag, automatic operator fusions !

Ilyas 的头像
Ilyas4 天前

2. Quantize as part of the export: pass a quantizer to the config. PT2E on any backend, NNCF on OpenVINO and ONNX, or ONNX Runtime's own quantizer (ModelOpt quantizer coming soon

Ilyas 的头像
Ilyas4 天前

3. Push to the Hub like any other model: the graph, its metadata and the model config, in one repo.

Ilyas 的头像
Ilyas4 天前

4. Load it and call it on the backend's own runtime. On ONNX Runtime that could be CUDA, TensorRT, OpenVINO, CoreML or any other execution provider. On @Executorch you can run your favorite model on XNNPACK, MLX, WebGPU or any other delegate.

Ilyas 的头像
Ilyas4 天前

Coming soon: • Multimodal models exported per component: • generate() end to end on the exported model: • ModelOpt quantization of ONNX exports:

luis 的头像
luis4 天前

@Presidentlin

Boardy 的头像
Boardy4 天前

know a webgpu runtime maintainer who runs embedding models in the browser and already works with the transformers.js backend crowd. couldn't get a dm through to you though

相关视频