Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Happy to OSS gpt-fast, a fast and hackable implementation of transformer inference in <1000 lines of native PyTorch with support for quantization, speculative decoding, TP, Nvidia/AMD support, and more! Code: Blog: (1/12)

cHHillee's profile picture

Horace He

50,898 subscribers

476,897 görüntüleme • 2 yıl önce •via X (Twitter)

Bilim & TeknolojiHaberler & PolitikaEğitim
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Just released a MuJoCo implementation of end-effector control with differential inverse kinematics. It's blazing fast, super readable, has 0 external dependencies and best of all, it's just a few lines of code:
0:15

Just released a MuJoCo implementation of end-effector control with differential inverse kinematics. It's blazing fast, super readable, has 0 external dependencies and best of all, it's just a few lines of code:

Kevin Zakka

26,427 görüntüleme • 2 yıl önce

gpt-oss-20b is lightning fast on a NVIDIA RTX 5090. I get around 160-180 tokens per second.
0:11

gpt-oss-20b is lightning fast on a NVIDIA RTX 5090. I get around 160-180 tokens per second.

Lewis Menelaws

127,103 görüntüleme • 11 ay önce

New version of Clip Deidentifier available! - much faster, up to twice as fast - better interface for selecting what you want - Apple Silicon (M1/M2) native support - new Linux support - code signed Grab Clip Deidentifier v0.8 here:
0:12

New version of Clip Deidentifier available! - much faster, up to twice as fast - better interface for selecting what you want - Apple Silicon (M1/M2) native support - new Linux support - code signed Grab Clip Deidentifier v0.8 here:

Jailyn/Ben

12,438 görüntüleme • 3 yıl önce

The blog post about exposing SwiftUI views to React Native is out! ✨ It shows you how to build a SwiftUI paywall in React Native with just a few lines of SwiftUI code! Check it out on Callstack Engineers blog 🎉
0:14

The blog post about exposing SwiftUI views to React Native is out! ✨ It shows you how to build a SwiftUI paywall in React Native with just a few lines of SwiftUI code! Check it out on Callstack Engineers blog 🎉

Oskar Kwaśniewski

20,794 görüntüleme • 1 yıl önce

Exposing SwiftUI views to React Native is a powerful feature! 👀 Here is an example of a SwiftUI paywall in React Native with just a few lines of SwiftUI code! A blog post on how to create native views using SwiftUI is coming soon! ✨
0:18

Exposing SwiftUI views to React Native is a powerful feature! 👀 Here is an example of a SwiftUI paywall in React Native with just a few lines of SwiftUI code! A blog post on how to create native views using SwiftUI is coming soon! ✨

Oskar Kwaśniewski

45,371 görüntüleme • 1 yıl önce

Want to run Deepseek R1 ? Text-generation-inference v3.1.0 is out and supports it out of the box. Both on AMD and Nvidia !
0:14

Want to run Deepseek R1 ? Text-generation-inference v3.1.0 is out and supports it out of the box. Both on AMD and Nvidia !

Nicolas Patry

28,859 görüntüleme • 1 yıl önce

We’re amazed: AMD now beats Nvidia in inference for Image-to-Video generation. 20% faster and 35% cheaper when using our Higgsfield DoP model. AMD MI300X also has a very easy setup. Details and tests are below. 🧩 1/8
0:15

We’re amazed: AMD now beats Nvidia in inference for Image-to-Video generation. 20% faster and 35% cheaper when using our Higgsfield DoP model. AMD MI300X also has a very easy setup. Details and tests are below. 🧩 1/8

Higgsfield AI 🧩

841,890 görüntüleme • 1 yıl önce

Grok-1 support in gpt-fast at faster(?) than anyone else has reported so far. 75 tok/s for a 300B+ parameter model on an 8xA100 node. If I understand correctly, ColossalAI reported 15 seconds to generate 100 tokens. gpt-fast takes 4.2 seconds to generate *400* tokens.
0:10

Grok-1 support in gpt-fast at faster(?) than anyone else has reported so far. 75 tok/s for a 300B+ parameter model on an 8xA100 node. If I understand correctly, ColossalAI reported 15 seconds to generate 100 tokens. gpt-fast takes 4.2 seconds to generate *400* tokens.

Horace He

54,137 görüntüleme • 2 yıl önce

we sped up distributed inference by up to 5x with decentralized speculative decoding. many don't realize that AI models normally generate text one single word at a time, waiting for the network after every word. speculative decoding changes this by using a "guess & confirm" system, similar to autocomplete. how it's done: 1. draft locally (the guess) instead of waiting for the network, a tiny, fast model on your device guesses the next few words instantly, without waiting for the network. 2. confirm remotely (the check) the massive remote model doesn't generate from scratch; it just checks the draft. it looks at the guesses in a batch and says "yes, yes, no." you get multiple words in the time it usually takes to get one. 3. adaptive logic dsd is smart. if the topic is creative, it lets the draft flow loose. if the topic is math or code, it checks more strictly. it balances speed and precision automatically so your inference almost feel instant. find out more: paper: blog:
0:18

we sped up distributed inference by up to 5x with decentralized speculative decoding. many don't realize that AI models normally generate text one single word at a time, waiting for the network after every word. speculative decoding changes this by using a "guess & confirm" system, similar to autocomplete. how it's done: 1. draft locally (the guess) instead of waiting for the network, a tiny, fast model on your device guesses the next few words instantly, without waiting for the network. 2. confirm remotely (the check) the massive remote model doesn't generate from scratch; it just checks the draft. it looks at the guesses in a batch and says "yes, yes, no." you get multiple words in the time it usually takes to get one. 3. adaptive logic dsd is smart. if the topic is creative, it lets the draft flow loose. if the topic is math or code, it checks more strictly. it balances speed and precision automatically so your inference almost feel instant. find out more: paper: blog:

Parallax

45,425 görüntüleme • 6 ay önce

We (Claude and I) made a new package: voxmlx It's an MLX implementation of Mistral's Voxtral mini realtime speech recognition model. It supports streaming audio and runs pretty fast on a laptop. To use it, simply: `uvx voxmlx` Also I did not write a single line of code for this package. Every line of code was written by Claude Code. More on that in thread.
0:16

We (Claude and I) made a new package: voxmlx It's an MLX implementation of Mistral's Voxtral mini realtime speech recognition model. It supports streaming audio and runs pretty fast on a laptop. To use it, simply: `uvx voxmlx` Also I did not write a single line of code for this package. Every line of code was written by Claude Code. More on that in thread.

Awni Hannun

33,891 görüntüleme • 4 ay önce

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses
0:13

(1/n) 🚀 With FastVideo, you can now generate a 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licenses

Hao AI Lab

78,660 görüntüleme • 11 ay önce

🔱 Introducing Manta Pacific: the EVM-native modular L2 for ZK applications Now, developing and deploying ZK applications is possible for Solidity developers with just a few lines of code. Learn more:
0:17

🔱 Introducing Manta Pacific: the EVM-native modular L2 for ZK applications Now, developing and deploying ZK applications is possible for Solidity developers with just a few lines of code. Learn more:

Manta Network (🔱,🔱)

1,461,798 görüntüleme • 3 yıl önce

The Momentum Support Gem replaces the Onslaught Support Gem and gives you a momentary speed boost that's great for going fast in the early acts.
0:14

The Momentum Support Gem replaces the Onslaught Support Gem and gives you a momentary speed boost that's great for going fast in the early acts.

Path of Exile

152,713 görüntüleme • 3 yıl önce

Expanded chain support. Now easier than ever to bring your assets in. Startale App introduces support for deposits across Ethereum Optimism Solana Base Arbitrum BNB Chain and more. Seamless, fast, and secure auto-bridging to Soneium 💿, powered by Rhinestone.
0:13

Expanded chain support. Now easier than ever to bring your assets in. Startale App introduces support for deposits across Ethereum Optimism Solana Base Arbitrum BNB Chain and more. Seamless, fast, and secure auto-bridging to Soneium 💿, powered by Rhinestone.

Startale App

33,736 görüntüleme • 1 ay önce

xAI's Grok continues to dominate on OpenRouter: • 1.07T tokens on Grok Code Fast 1 (#1) • 1.02T tokens on Grok 4 Fast (#2) • Grok Code Fast 1 alone commands 46%+ share in programming. • xAI leads with 37.8% market share, well ahead of Google, Anthropic, OpenAI & others.
0:14

xAI's Grok continues to dominate on OpenRouter: • 1.07T tokens on Grok Code Fast 1 (#1) • 1.02T tokens on Grok 4 Fast (#2) • Grok Code Fast 1 alone commands 46%+ share in programming. • xAI leads with 37.8% market share, well ahead of Google, Anthropic, OpenAI & others.

DogeDesigner

143,443 görüntüleme • 9 ay önce

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference
0:15

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference

ani

16,583 görüntüleme • 1 ay önce

Introducing react-native-nano-icons! 💎🔬 The ultimate way to handle custom icons in React Native: 🛠️ Integrated SVG-to-font build process ⚡ Blazing fast native components 🎨 Multi-color font icon support 📐 Use icons inline within RN components 🚀 Expo Go & Web support out of the box Check out our GitHub 👇 or learn more in the 🧵
0:18

Introducing react-native-nano-icons! 💎🔬 The ultimate way to handle custom icons in React Native: 🛠️ Integrated SVG-to-font build process ⚡ Blazing fast native components 🎨 Multi-color font icon support 📐 Use icons inline within RN components 🚀 Expo Go & Web support out of the box Check out our GitHub 👇 or learn more in the 🧵

Software Mansion

65,876 görüntüleme • 2 ay önce