正在加载视频...
视频加载失败
Export, quantize and run end-to-end every transformers model, on any backend: same inputs, same outputs, drop-in replacement. 🧵👇 Docs: Release notes:
23,492 次观看 • 4 天前 •via X (Twitter)
8 条评论

Faster than torch.compile. EmbeddingGemma2, 8 × 128 tokens: - Xeon Platinum 8570: @intel OpenVINO 4.1× eager, torch.compile 1.9×, up to 4.9× with nncf int8 quantization. - A100: @onnxruntime TensorRT EP 7.6× eager PyTorch, torch.compile 5.4× (and even faster with ModelOpt int8).

1. Export: One API for Dynamo, ONNX, OpenVINO and ExecuTorch, dynamic shapes with one flag, automatic operator fusions !

2. Quantize as part of the export: pass a quantizer to the config. PT2E on any backend, NNCF on OpenVINO and ONNX, or ONNX Runtime's own quantizer (ModelOpt quantizer coming soon

3. Push to the Hub like any other model: the graph, its metadata and the model config, in one repo.

4. Load it and call it on the backend's own runtime. On ONNX Runtime that could be CUDA, TensorRT, OpenVINO, CoreML or any other execution provider. On @Executorch you can run your favorite model on XNNPACK, MLX, WebGPU or any other delegate.

Coming soon: • Multimodal models exported per component: • generate() end to end on the exported model: • ModelOpt quantization of ONNX exports:

@Presidentlin

know a webgpu runtime maintainer who runs embedding models in the browser and already works with the transformers.js backend crowd. couldn't get a dm through to you though
相关视频
Sensitive content
always getting on each other’s nerves only to end up in the same bed afterward
nini
68,733 次观看 • 4 个月前

