Загрузка видео...

Не удалось загрузить видео

На главную

Happy to OSS gpt-fast, a fast and hackable implementation of transformer inference in <1000 lines of native PyTorch with support for quantization, speculative decoding, TP, Nvidia/AMD support, and more! Code: Blog: (1/12)

cHHillee's profile picture

Horace He

52,413 subscribers

476,969 просмотров • 2 лет назад •via X (Twitter)

Наука и технологииНовости и политикаОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Just released a MuJoCo implementation of end-effector control with differential inverse kinematics. It's blazing fast, super readable, has 0 external dependencies and best of all, it's just a few lines of code:
0:15

Just released a MuJoCo implementation of end-effector control with differential inverse kinematics. It's blazing fast, super readable, has 0 external dependencies and best of all, it's just a few lines of code:

Kevin Zakka

26,427 просмотров • 2 лет назад

New version of Clip Deidentifier available! - much faster, up to twice as fast - better interface for selecting what you want - Apple Silicon (M1/M2) native support - new Linux support - code signed Grab Clip Deidentifier v0.8 here:
0:12

New version of Clip Deidentifier available! - much faster, up to twice as fast - better interface for selecting what you want - Apple Silicon (M1/M2) native support - new Linux support - code signed Grab Clip Deidentifier v0.8 here:

Jailyn/Ben

12,459 просмотров • 3 лет назад

We’re amazed: AMD now beats Nvidia in inference for Image-to-Video generation. 20% faster and 35% cheaper when using our Higgsfield DoP model. AMD MI300X also has a very easy setup. Details and tests are below. 🧩 1/8
0:15

We’re amazed: AMD now beats Nvidia in inference for Image-to-Video generation. 20% faster and 35% cheaper when using our Higgsfield DoP model. AMD MI300X also has a very easy setup. Details and tests are below. 🧩 1/8

Higgsfield AI 🧩

842,034 просмотров • 1 год назад

Grok-1 support in gpt-fast at faster(?) than anyone else has reported so far. 75 tok/s for a 300B+ parameter model on an 8xA100 node. If I understand correctly, ColossalAI reported 15 seconds to generate 100 tokens. gpt-fast takes 4.2 seconds to generate *400* tokens.
0:10

Grok-1 support in gpt-fast at faster(?) than anyone else has reported so far. 75 tok/s for a 300B+ parameter model on an 8xA100 node. If I understand correctly, ColossalAI reported 15 seconds to generate 100 tokens. gpt-fast takes 4.2 seconds to generate *400* tokens.

Horace He

54,137 просмотров • 2 лет назад

we sped up distributed inference by up to 5x with decentralized speculative decoding. many don't realize that AI models normally generate text one single word at a time, waiting for the network after every word. speculative decoding changes this by using a "guess & confirm" system, similar to autocomplete. how it's done: 1. draft locally (the guess) instead of waiting for the network, a tiny, fast model on your device guesses the next few words instantly, without waiting for the network. 2. confirm remotely (the check) the massive remote model doesn't generate from scratch; it just checks the draft. it looks at the guesses in a batch and says "yes, yes, no." you get multiple words in the time it usually takes to get one. 3. adaptive logic dsd is smart. if the topic is creative, it lets the draft flow loose. if the topic is math or code, it checks more strictly. it balances speed and precision automatically so your inference almost feel instant. find out more: paper: blog:
0:18

we sped up distributed inference by up to 5x with decentralized speculative decoding. many don't realize that AI models normally generate text one single word at a time, waiting for the network after every word. speculative decoding changes this by using a "guess & confirm" system, similar to autocomplete. how it's done: 1. draft locally (the guess) instead of waiting for the network, a tiny, fast model on your device guesses the next few words instantly, without waiting for the network. 2. confirm remotely (the check) the massive remote model doesn't generate from scratch; it just checks the draft. it looks at the guesses in a batch and says "yes, yes, no." you get multiple words in the time it usually takes to get one. 3. adaptive logic dsd is smart. if the topic is creative, it lets the draft flow loose. if the topic is math or code, it checks more strictly. it balances speed and precision automatically so your inference almost feel instant. find out more: paper: blog:

Parallax

45,584 просмотров • 7 месяцев назад

We (Claude and I) made a new package: voxmlx It's an MLX implementation of Mistral's Voxtral mini realtime speech recognition model. It supports streaming audio and runs pretty fast on a laptop. To use it, simply: `uvx voxmlx` Also I did not write a single line of code for this package. Every line of code was written by Claude Code. More on that in thread.
0:16

We (Claude and I) made a new package: voxmlx It's an MLX implementation of Mistral's Voxtral mini realtime speech recognition model. It supports streaming audio and runs pretty fast on a laptop. To use it, simply: `uvx voxmlx` Also I did not write a single line of code for this package. Every line of code was written by Claude Code. More on that in thread.

Awni Hannun

33,896 просмотров • 6 месяцев назад

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses
0:13

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses

Hao AI Lab

78,660 просмотров • 1 год назад

A Good Q2 for Cerebras. Cloud revenue nearly quadrupled. Core revenue more than doubled. Beat guidance on every core business metric - revenue, gross margins, operating margins. Raised full-year guidance across the board. $25.4 billion in remaining backlog. And 600MW+ of data center capacity now up or under contract for delivery by Q4 2027 to serve it. Behind the numbers: Manufacturing capacity scaling more than 10x in 2026. We partnered with OpenAI and are serving their frontier model, GPT-5.6 Sol. Partnerships with AMD and Amazon Web Services establish Cerebras as the leader in disaggregated inference. Fast inference is winning across many markets: coding, agentic flows and increasingly cybersecurity.
0:18

A Good Q2 for Cerebras. Cloud revenue nearly quadrupled. Core revenue more than doubled. Beat guidance on every core business metric - revenue, gross margins, operating margins. Raised full-year guidance across the board. $25.4 billion in remaining backlog. And 600MW+ of data center capacity now up or under contract for delivery by Q4 2027 to serve it. Behind the numbers: Manufacturing capacity scaling more than 10x in 2026. We partnered with OpenAI and are serving their frontier model, GPT-5.6 Sol. Partnerships with AMD and Amazon Web Services establish Cerebras as the leader in disaggregated inference. Fast inference is winning across many markets: coding, agentic flows and increasingly cybersecurity.

Andrew Feldman

22,729 просмотров • 17 дней назад

🔱 Introducing Manta Pacific: the EVM-native modular L2 for ZK applications Now, developing and deploying ZK applications is possible for Solidity developers with just a few lines of code. Learn more:
0:17

🔱 Introducing Manta Pacific: the EVM-native modular L2 for ZK applications Now, developing and deploying ZK applications is possible for Solidity developers with just a few lines of code. Learn more:

Manta Network (🔱,🔱)

1,461,983 просмотров • 3 лет назад

Expanded chain support. Now easier than ever to bring your assets in. Startale App introduces support for deposits across Ethereum Optimism Solana Base Arbitrum BNB Chain and more. Seamless, fast, and secure auto-bridging to Soneium 💿, powered by Rhinestone.
0:13

Expanded chain support. Now easier than ever to bring your assets in. Startale App introduces support for deposits across Ethereum Optimism Solana Base Arbitrum BNB Chain and more. Seamless, fast, and secure auto-bridging to Soneium 💿, powered by Rhinestone.

Startale App

34,598 просмотров • 3 месяцев назад

Introducing react-native-nano-icons! 💎🔬 The ultimate way to handle custom icons in React Native: 🛠️ Integrated SVG-to-font build process ⚡ Blazing fast native components 🎨 Multi-color font icon support 📐 Use icons inline within RN components 🚀 Expo Go & Web support out of the box Check out our GitHub 👇 or learn more in the 🧵
0:18

Introducing react-native-nano-icons! 💎🔬 The ultimate way to handle custom icons in React Native: 🛠️ Integrated SVG-to-font build process ⚡ Blazing fast native components 🎨 Multi-color font icon support 📐 Use icons inline within RN components 🚀 Expo Go & Web support out of the box Check out our GitHub 👇 or learn more in the 🧵

Software Mansion

66,552 просмотров • 4 месяцев назад

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference
0:15

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference

ani

17,269 просмотров • 3 месяцев назад

made something I wanted for a while: a high performance custom tab/splits library in swift for native Mac apps. with support for drag/drop, CoreAnimation accelerated transitions, and more. spent a lot of time on the polish of this thing to feel snappy, but fun.
0:18

made something I wanted for a while: a high performance custom tab/splits library in swift for native Mac apps. with support for drag/drop, CoreAnimation accelerated transitions, and more. spent a lot of time on the polish of this thing to feel snappy, but fun.

almonk

25,165 просмотров • 7 месяцев назад

A lot of work over the weekend: LYGIA: cmake/python bundle script for C++ VERA v1.1.0: - support for gsplats - LYGIA bundle to stream line default shaders - better camera interactions GlslViewer v3.5.1: support for gsplats and native lygia support
0:15

A lot of work over the weekend: LYGIA: cmake/python bundle script for C++ VERA v1.1.0: - support for gsplats - LYGIA bundle to stream line default shaders - better camera interactions GlslViewer v3.5.1: support for gsplats and native lygia support

patricio gonzalez vivo

15,097 просмотров • 6 месяцев назад

Happy #InternationalWomensDay! We celebrate women’s achievements in all aspects of life and support a more inclusive, diverse, and connected world! 🫶 #IWD #IWD23
0:12

Happy #InternationalWomensDay! We celebrate women’s achievements in all aspects of life and support a more inclusive, diverse, and connected world! 🫶 #IWD #IWD23

Huawei

65,163 просмотров • 3 лет назад