Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Your Mac is about to run inference like a datacenter. Coming soon to MLX-Swift: Continuous batching: the fastest way to handle multiple inference streams locally. It starts with regular inference and seamlessly upgrades to batched mode when new requests arrive. The best of both worlds. Based on the work... show more

Ronald Mannak

5,767 subscribers

60,570 görüntüleme • 8 ay önce •via X (Twitter)

Eğitim Bilim & Teknoloji Finans

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

We're teaming up with Cerebras to build the fastest possible inference. Coming soon to Amazon Bedrock, we’re delivering inference performance an order of magnitude faster than what’s available today by connecting AWS Trainium3 for compute-intensive prefill with Cerebras CS-3 to power decode. Learn more about the partnership.

We're teaming up with Cerebras to build the fastest possible inference. Coming soon to Amazon Bedrock, we’re delivering inference performance an order of magnitude faster than what’s available today by connecting AWS Trainium3 for compute-intensive prefill with Cerebras CS-3 to power decode. Learn more about the partnership.

Amazon Web Services

205,173 görüntüleme • 4 ay önce

Free, instant access to 1M inference tok/day is excited to launch Cerebras Supernova! Developers get 1M free inference toks/day, and if you're a startup, get up to 3 months FREE of increased context windows, rate limits, and 245M tok/day. All on the world's fastest inference provider. This is the most generous free tier that exists. It really is land of the free...(credits)🇺🇸

Free, instant access to 1M inference tok/day is excited to launch Cerebras Supernova! Developers get 1M free inference toks/day, and if you're a startup, get up to 3 months FREE of increased context windows, rate limits, and 245M tok/day. All on the world's fastest inference provider. This is the most generous free tier that exists. It really is land of the free...(credits)🇺🇸

Sarah Chieng

122,158 görüntüleme • 1 yıl önce

IN NEWS: Baseten raises a $150M series D round. Tuhin Srivastava (Founder & CEO, Baseten) on the future of inference: “I think the token price goes down and inference should get cheaper over time. And that really just means there is going to be more inference.” “Every time we lower prices or optimize models to make it cheaper, four months later customers are spending more anyway.” “Inference prices will go down, but if the world is run by AI in 10 years, there is going to be a lot of inference. It better be cheap.”

IN NEWS: Baseten raises a $150M series D round. Tuhin Srivastava (Founder & CEO, Baseten) on the future of inference: “I think the token price goes down and inference should get cheaper over time. And that really just means there is going to be more inference.” “Every time we lower prices or optimize models to make it cheaper, four months later customers are spending more anyway.” “Inference prices will go down, but if the world is run by AI in 10 years, there is going to be a lot of inference. It better be cheap.”

TBPN

17,057 görüntüleme • 10 ay önce

Every AI product running in production depends on an inference system someone had to engineer, optimize and more. Scheduling, batching, routing, cost per token. This is a craft. The Inference Frontier Program spotlights the builders behind that work. 💡 Watch the video and nominate a team:

Every AI product running in production depends on an inference system someone had to engineer, optimize and more. Scheduling, batching, routing, cost per token. This is a craft. The Inference Frontier Program spotlights the builders behind that work. 💡 Watch the video and nominate a team:

Nebius

54,224 görüntüleme • 4 ay önce

Jesse Pollak on the ecosystem forming around tokenized inference "There's this really cool tokenized inference market that Venice has innovated, where they've taken inference and turned it into a token called DIEM where you can buy it and you get $1 of inference per day" "That innovation of tokenizing inference has unlocked a ton of innovation around it, where people are using that tokenized inference to do things" "There's other projects that let people sell their tokenized inference at a discount so other people can get cheaper inference and the whole system works more efficiently" "I feel like that's the first time we're really seeing the intersection of AI plus markets plus crypto and it's thanks to Venice and it's happening on Base"

Jesse Pollak on the ecosystem forming around tokenized inference "There's this really cool tokenized inference market that Venice has innovated, where they've taken inference and turned it into a token called DIEM where you can buy it and you get $1 of inference per day" "That innovation of tokenizing inference has unlocked a ton of innovation around it, where people are using that tokenized inference to do things" "There's other projects that let people sell their tokenized inference at a discount so other people can get cheaper inference and the whole system works more efficiently" "I feel like that's the first time we're really seeing the intersection of AI plus markets plus crypto and it's thanks to Venice and it's happening on Base"

Market Bubble

25,780 görüntüleme • 8 gün önce

Dear Visitors, A new case is brought to Mr. Inference, while the shadow of serial murder looms over the stars, can he, or can he not, stop it from happening? New Truth & Inference Event [Enlightenment Under the Stars] Coming Soon. #IdentityV #TruthandInference

Dear Visitors, A new case is brought to Mr. Inference, while the shadow of serial murder looms over the stars, can he, or can he not, stop it from happening? New Truth & Inference Event [Enlightenment Under the Stars] Coming Soon. #IdentityV #TruthandInference

Identity V

64,836 görüntüleme • 1 yıl önce

Ready to experience the future of #AI? #Inferium's MVP is coming. Be among the first to: 🧠 Evaluate cutting-edge AI models ⚡ Run #inference on AI models 💰 Earn rewards & top the leaderboard Join waitlist now to secure early access and exclusive perks: #Inferium #MVP #AI #Inference

Ready to experience the future of #AI? #Inferium's MVP is coming. Be among the first to: 🧠 Evaluate cutting-edge AI models ⚡ Run #inference on AI models 💰 Earn rewards & top the leaderboard Join waitlist now to secure early access and exclusive perks: #Inferium #MVP #AI #Inference

Inferium

37,344 görüntüleme • 2 yıl önce

mlx distributed uses mpi for machine to machine communication, and this is how you can handle uneven distribution of model layers. Thanks Awni Hannun for the help

mlx distributed uses mpi for machine to machine communication, and this is how you can handle uneven distribution of model layers. Thanks Awni Hannun for the help

Alex Ziskind

31,265 görüntüleme • 1 yıl önce

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

AI inference is already live on UOMI Router. Access frontier open-source models through a decentralized inference network with: • Low-cost inference • OpenAI-compatible APIs • Verifiable compute • On-chain settlement Start building: And if you have idle GPUs, there's now a second way to participate. The UOMI Provider whitelist is officially open. Instead of letting your hardware sit unused, you can contribute GPU power to the UOMI Inference Network and earn from real AI demand. Join the Provider whitelist:

Uomi

18,816 görüntüleme • 1 ay önce

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 görüntüleme • 4 ay önce

Free, instant access to 1M inference tok/day is now waitlist-free! Building the new onboarding flow and removing our waitlist were so fun to ship & all API signups automatically get > Free 1,000,000 toks/day > SOTA models on the world's fastest inference provider

Free, instant access to 1M inference tok/day is now waitlist-free! Building the new onboarding flow and removing our waitlist were so fun to ship & all API signups automatically get > Free 1,000,000 toks/day > SOTA models on the world's fastest inference provider

Sarah Chieng

120,480 görüntüleme • 1 yıl önce

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 görüntüleme • 2 yıl önce

"The inflection point for inference has arrived." — Jensen Huang, Founder & CEO of NVIDIA We’ve officially crossed a new milestone in the inference era — where widespread adoption of AI shifts from learning to doing. The breakthrough: extreme codesign across hardware and software driving down cost per token. Lower cost → more inference More inference → more users and applications More users and applications → exponential AI revenue growth

"The inflection point for inference has arrived." — Jensen Huang, Founder & CEO of NVIDIA We’ve officially crossed a new milestone in the inference era — where widespread adoption of AI shifts from learning to doing. The breakthrough: extreme codesign across hardware and software driving down cost per token. Lower cost → more inference More inference → more users and applications More users and applications → exponential AI revenue growth

NVIDIA

81,834 görüntüleme • 3 ay önce

Demo of Tether's QVAC running local inference on a mobile device at incredible speed, via llama.cpp + LLAMA 3.2 1B parameters. QVAC is a generalized inference and fine-tuning runtime able to adapt to any device, from smartphones, to laptops and servers. Lots of models being supported already. More to come. No limits. Infinite intelligence. Coming soon

Demo of Tether's QVAC running local inference on a mobile device at incredible speed, via llama.cpp + LLAMA 3.2 1B parameters. QVAC is a generalized inference and fine-tuning runtime able to adapt to any device, from smartphones, to laptops and servers. Lots of models being supported already. More to come. No limits. Infinite intelligence. Coming soon

Paolo Ardoino 🤖

42,074 görüntüleme • 1 yıl önce

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

oMLX is working really well as single machine inference engine for coding agents! Caching is managed perfectly (it can use a ton of disk space, be aware!) and oQ quantization delivers great results. Behind the scenes it uses the standard MLX building blocks (75% created by Prince Canuma 🙏): - mlx-lm - mlx-vlm - mlx-embeddings - mlx-audio I tested Qwen3.6-35B-A3B-oQ6 on M5 Max with two pi instances and it was fast and furious and leveraging cache like crazy as you can see in the video. Let me try to create some oQ versions (2,4,6?) of MiniMax M2.7 now and then I'll pass to distributed inference. I must win! 💪

Ivan Fioravanti ᯅ

19,384 görüntüleme • 3 ay önce

Before Fable 5 was shut down, it pushed Gemma 4 to 255 tok/s on WebGPU. Some didn't believe it was real. Today we're releasing the demo and kernels it wrote for you to see yourself. Run it locally in your browser. Agentic kernel optimization is the future of on-device inference

Before Fable 5 was shut down, it pushed Gemma 4 to 255 tok/s on WebGPU. Some didn't believe it was real. Today we're releasing the demo and kernels it wrote for you to see yourself. Run it locally in your browser. Agentic kernel optimization is the future of on-device inference

Xenova

482,761 görüntüleme • 1 ay önce

Releasing moondream-zig! It is a fast, implementation of moondream2 inference on the CPU written from-scratch in Zig :) moondream-zig provides 1.5-2x faster inference compared to huggingface on the same device. moondream vik

Releasing moondream-zig! It is a fast, implementation of moondream2 inference on the CPU written from-scratch in Zig :) moondream-zig provides 1.5-2x faster inference compared to huggingface on the same device. moondream vik

snow

29,245 görüntüleme • 1 yıl önce

GEMMA 4 IS STARTING TO KILL A LOT OF AI SUBSCRIPTIONS. Local inference, multimodal support, strong coding, and no API bill is making “just run it on your own machine” feel way more real.

GEMMA 4 IS STARTING TO KILL A LOT OF AI SUBSCRIPTIONS. Local inference, multimodal support, strong coding, and no API bill is making “just run it on your own machine” feel way more real.

0xMarioNawfal

65,679 görüntüleme • 3 ay önce