Загрузка видео...

Не удалось загрузить видео

На главную

Export, quantize and run end-to-end every transformers model, on any backend: same inputs, same outputs, drop-in replacement. 🧵👇 Docs: Release notes:

23,492 просмотров • 4 дней назад •via X (Twitter)

Комментарии: 8

Фото профиля Ilyas
Ilyas4 дней назад

Faster than torch.compile. EmbeddingGemma2, 8 × 128 tokens: - Xeon Platinum 8570: @intel OpenVINO 4.1× eager, torch.compile 1.9×, up to 4.9× with nncf int8 quantization. - A100: @onnxruntime TensorRT EP 7.6× eager PyTorch, torch.compile 5.4× (and even faster with ModelOpt int8).

Фото профиля Ilyas
Ilyas4 дней назад

1. Export: One API for Dynamo, ONNX, OpenVINO and ExecuTorch, dynamic shapes with one flag, automatic operator fusions !

Фото профиля Ilyas
Ilyas4 дней назад

2. Quantize as part of the export: pass a quantizer to the config. PT2E on any backend, NNCF on OpenVINO and ONNX, or ONNX Runtime's own quantizer (ModelOpt quantizer coming soon

Фото профиля Ilyas
Ilyas4 дней назад

3. Push to the Hub like any other model: the graph, its metadata and the model config, in one repo.

Фото профиля Ilyas
Ilyas4 дней назад

4. Load it and call it on the backend's own runtime. On ONNX Runtime that could be CUDA, TensorRT, OpenVINO, CoreML or any other execution provider. On @Executorch you can run your favorite model on XNNPACK, MLX, WebGPU or any other delegate.

Фото профиля Ilyas
Ilyas4 дней назад

Coming soon: • Multimodal models exported per component: • generate() end to end on the exported model: • ModelOpt quantization of ONNX exports:

Фото профиля luis
luis4 дней назад

@Presidentlin

Фото профиля Boardy
Boardy4 дней назад

know a webgpu runtime maintainer who runs embedding models in the browser and already works with the transformers.js backend crowd. couldn't get a dm through to you though

Похожие видео