Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

📣 We are bringing WebGPU to React Native! This update enables: 🚀 Modern GPU APIs (Vulkan & Metal) and enables general purpose GPU computation for AI/ML 🐎 Seamless Reanimated integration 🧊 Bringing stunning 3d experiences to React Native using ThreeJS and React Three Fiber

mustafa01ali's profile picture

Mustafa Ali

8,788 subscribers

262,567 görüntüleme • 1 yıl önce •via X (Twitter)

OyunBilim & TeknolojiEğitim
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

react-native-streamdown v0.2.0 is out! 🚀 Powered by the new react-native-enriched-markdown v0.6.0, markdown now streams more smoothly with the full GitHub Flavored Markdown (GFM) support, including seamless real-time rendering for tables and rich formatting. If you’re building LLM or AI-chat experiences in React Native, this update is for you. Check it out! 👇
0:14

react-native-streamdown v0.2.0 is out! 🚀 Powered by the new react-native-enriched-markdown v0.6.0, markdown now streams more smoothly with the full GitHub Flavored Markdown (GFM) support, including seamless real-time rendering for tables and rich formatting. If you’re building LLM or AI-chat experiences in React Native, this update is for you. Check it out! 👇

Software Mansion

11,740 görüntüleme • 3 ay önce

Finally got something to show - we're working on a new 3D library for React Native! 👀🧊 👉 Powered by native Graphics APIs (Metal/Vulkan) 👉 Full control over rendering in JS 👉 120 FPS rendering on a Worklet Thread (no lags!) 👉 Supports hotswapping .glb/.gltf models
0:12

Finally got something to show - we're working on a new 3D library for React Native! 👀🧊 👉 Powered by native Graphics APIs (Metal/Vulkan) 👉 Full control over rendering in JS 👉 120 FPS rendering on a Worklet Thread (no lags!) 👉 Supports hotswapping .glb/.gltf models

Marc

86,230 görüntüleme • 2 yıl önce

Just updated Luna - AI Chat Template. 🚀 - Refreshed UI - Speech-to-text - Streaming responses - OpenAI, Gemini and Claude integration built-in Check it out here: Built with React Native Expo Nativewind and Software Mansion reanimated
0:12

Just updated Luna - AI Chat Template. 🚀 - Refreshed UI - Speech-to-text - Streaming responses - OpenAI, Gemini and Claude integration built-in Check it out here: Built with React Native Expo Nativewind and Software Mansion reanimated

Thomino

21,334 görüntüleme • 7 ay önce

I'm 9 years old and I wrote a blog post about the hook I use to build this scroll direction based animation from Gmail app ✦ Built for React Native using Expo ✦ Powered by Software Mansion reanimated ✦ Blog post at
0:12

I'm 9 years old and I wrote a blog post about the hook I use to build this scroll direction based animation from Gmail app ✦ Built for React Native using Expo ✦ Powered by Software Mansion reanimated ✦ Blog post at

Volodymyr

24,579 görüntüleme • 1 yıl önce

i'm writing my own gui framework in c++ and rewriting my image editor using it, which is going good so far, i wanted to move from swiftui, qt, tauri, electron, react native, and flutter so i can use metal and directx gpu apis on macos and windows as much as possible,so i think going full metal/directx is the way to go i know i have a very hard and long way to go though as there are many many flaws,bugs in this very very early version and doesnt utilize the gpu as much as i want to but i hope we will get there soon ,i also need to make the ui look somewhat decent instead of whatever mess it is right now
0:17

i'm writing my own gui framework in c++ and rewriting my image editor using it, which is going good so far, i wanted to move from swiftui, qt, tauri, electron, react native, and flutter so i can use metal and directx gpu apis on macos and windows as much as possible,so i think going full metal/directx is the way to go i know i have a very hard and long way to go though as there are many many flaws,bugs in this very very early version and doesnt utilize the gpu as much as i want to but i hope we will get there soon ,i also need to make the ui look somewhat decent instead of whatever mess it is right now

Ruben Veidt

85,053 görüntüleme • 1 yıl önce

We are alive! 🚀 Surge is live on Sui — and AI project applications are now OPEN at The first AI Agent Launchpad native to Sui, powered by Cetus🐳, offering seamless integration for efficient trading and growth. Surf the AI wave on Sui! 🏄‍♂️ Dive into docs: #SurgeOnSui #AILaunchpad #SUI #CetusProtocol
0:17

We are alive! 🚀 Surge is live on Sui — and AI project applications are now OPEN at The first AI Agent Launchpad native to Sui, powered by Cetus🐳, offering seamless integration for efficient trading and growth. Surf the AI wave on Sui! 🏄‍♂️ Dive into docs: #SurgeOnSui #AILaunchpad #SUI #CetusProtocol

Surge

58,740 görüntüleme • 1 yıl önce

We're excited to announce react-native-executorch v0.3.0 packed with powerful AI features 🎉 What's New: 💬Speech to Text transcription using Whisper and Moonshine 📙OCR for extracting text from images 🚀 MPS Support ⚡️ Hookless API 🧠 LLM Message History ⬇️ Background Model Downloads 📊 Benchmarks 🧪 Experimental OCR for extracting vertical text from images
0:17

We're excited to announce react-native-executorch v0.3.0 packed with powerful AI features 🎉 What's New: 💬Speech to Text transcription using Whisper and Moonshine 📙OCR for extracting text from images 🚀 MPS Support ⚡️ Hookless API 🧠 LLM Message History ⬇️ Background Model Downloads 📊 Benchmarks 🧪 Experimental OCR for extracting vertical text from images

Software Mansion

29,905 görüntüleme • 1 yıl önce

Ritual natively enables AI computation onchain. That includes LLM inference, ML, and any arbitrary logic. While this fundamentally changes what smart contracts can do, the learning curve is real, which is why we one curated entry point to cut through the noise ⏬⏬ Welcome to Ritual Tools, everything a net-new builder needs and nothing you don’t. ❖ Quickstart Guides ❖ Core Protocol Docs ❖ Example Contracts & dApps ❖ Community Channels Our goal is to transform builders into passive learners into active members who can quickly and easily adopt our native agentic tools in an intuitive and seamless manner within 15 minutes or less.
0:14

Ritual natively enables AI computation onchain. That includes LLM inference, ML, and any arbitrary logic. While this fundamentally changes what smart contracts can do, the learning curve is real, which is why we one curated entry point to cut through the noise ⏬⏬ Welcome to Ritual Tools, everything a net-new builder needs and nothing you don’t. ❖ Quickstart Guides ❖ Core Protocol Docs ❖ Example Contracts & dApps ❖ Community Channels Our goal is to transform builders into passive learners into active members who can quickly and easily adopt our native agentic tools in an intuitive and seamless manner within 15 minutes or less.

Ritual Foundation

34,756 görüntüleme • 2 ay önce

HTML enters 3D! Or vice versa? With the new HTML in Canvas by WICG, we can finally put native DOM elements directly into WebGL/WebGPU scenes. It is experimental for now, but the possibilities for 3D interfaces and special effects are huge. This demo was built using Three.js and Omma AI (tool by Spline ) It’s a fun new way to explore what the web can do! Are you interested in seeing the demo?
0:10

HTML enters 3D! Or vice versa? With the new HTML in Canvas by WICG, we can finally put native DOM elements directly into WebGL/WebGPU scenes. It is experimental for now, but the possibilities for 3D interfaces and special effects are huge. This demo was built using Three.js and Omma AI (tool by Spline ) It’s a fun new way to explore what the web can do! Are you interested in seeing the demo?

Gábor Pribék

176,535 görüntüleme • 4 ay önce

⬛️ We are currently accelerating the incubation of GPU Nodes into the infraX Network, with 12 H100’s currently available for operation. Despite the incubation of such immense GPU power, the infraX Platform is optimally designed to run on the least amount of computational power possible, meaning a lot of our available GPU nodes are currently sitting idle. Currently, we're utilising a single gigantic NVIDIA H100 server with 80GB of VRAM and over 220GB of RAM to run our Platform. To put that in perspective, it rivals the computational power of an adult human brain. This setup enables us to handle immense computational load and deliver high-quality AI content to our users, however we have much more in store. Our remaining, immense network of GPU units is currently being prepared for rental operations as we look to transform the corporate GPU lending sphere through our corporate GPU lending protocol. We already have many high tier Web3 Players ready for technical integration, with more approaching us daily. Through our V3 DApp we look to make these integrations publicly viewable with real time usage graphs integrated directly into our Platform, allowing for exceedingly unique viewing opportunities. $INFRA
0:11

⬛️ We are currently accelerating the incubation of GPU Nodes into the infraX Network, with 12 H100’s currently available for operation. Despite the incubation of such immense GPU power, the infraX Platform is optimally designed to run on the least amount of computational power possible, meaning a lot of our available GPU nodes are currently sitting idle. Currently, we're utilising a single gigantic NVIDIA H100 server with 80GB of VRAM and over 220GB of RAM to run our Platform. To put that in perspective, it rivals the computational power of an adult human brain. This setup enables us to handle immense computational load and deliver high-quality AI content to our users, however we have much more in store. Our remaining, immense network of GPU units is currently being prepared for rental operations as we look to transform the corporate GPU lending sphere through our corporate GPU lending protocol. We already have many high tier Web3 Players ready for technical integration, with more approaching us daily. Through our V3 DApp we look to make these integrations publicly viewable with real time usage graphs integrated directly into our Platform, allowing for exceedingly unique viewing opportunities. $INFRA

infraX | $INFRA

42,843 görüntüleme • 1 yıl önce

Native USDC and CCTP are now live on X Layer by OKX! PSPs, fintechs, AI agents, and DeFi apps and protocols on X Layer can now access the world’s largest regulated dollar stablecoin for a range of use cases: → DeFi activity: Use USDC as collateral to enable onchain lending, borrowing, and spot and perps trading with 24/7 settlement → Crosschain money movement: Move USDC seamlessly across chains with CCTP for robust liquidity access to power swaps, purchases, and treasury management → AI-powered workflows: Support agentic payments on X Layer’s x402 ecosystem, enabling automated AI agent payments for services, APIs, and more → Institutional-grade settlement: Seamless USDC issuance and redemption on X Layer via Circle Mint for qualified businesses Day 1 apps: OKX, OKX DEX Bridge With this integration, USDC is supported natively on 36 blockchains. CCTP is now available on 26 blockchains and enables secure USDC crosschain transfers.
0:14

Native USDC and CCTP are now live on X Layer by OKX! PSPs, fintechs, AI agents, and DeFi apps and protocols on X Layer can now access the world’s largest regulated dollar stablecoin for a range of use cases: → DeFi activity: Use USDC as collateral to enable onchain lending, borrowing, and spot and perps trading with 24/7 settlement → Crosschain money movement: Move USDC seamlessly across chains with CCTP for robust liquidity access to power swaps, purchases, and treasury management → AI-powered workflows: Support agentic payments on X Layer’s x402 ecosystem, enabling automated AI agent payments for services, APIs, and more → Institutional-grade settlement: Seamless USDC issuance and redemption on X Layer via Circle Mint for qualified businesses Day 1 apps: OKX, OKX DEX Bridge With this integration, USDC is supported natively on 36 blockchains. CCTP is now available on 26 blockchains and enables secure USDC crosschain transfers.

Circle

89,783 görüntüleme • 29 gün önce

Here's what The Browser Company's AI eng & ML teams are working on for Dia right now: (This is a pitch to come work for us; info at end) 🤖 COMPUTER USE – we've built our own bespoke APIs on top of Chromium to optimize latency, accuracy, and cost of computer-using agents. Demo attached. Big breakthroughs here in recent weeks. 🛡️ ON-DEVICE MODELS – we've built our own custom infra to run everything from encoder-only models to full LLMs on device. It's cross-platform, supports LoRa adapters, and optimized for the GPU. This system preserves privacy and enables fast inference times. 🧠 MEMORY – with your permission, Dia automatically tailors your AI experiences to you, personally, based on the tabs you open while browsing normally every day. We're also bringing vertical memory to specific features. ♻️ DATA FLYWHEELS – our Fall/Winter P0 is to double-down on training custom models based on implicit signals from daily use of Dia. Dia should get smarter and more useful the more people use it. Whether via RL, auto-generated prompts, or otherwise. If this work sounds interesting to you please visit our jobs page or email careers@thebrowser.company. Hiring nearly every related role -- from ML engineers to people prototyping with AI and context/prompt writers -- everyone encouraged to apply!!
0:14

Here's what The Browser Company's AI eng & ML teams are working on for Dia right now: (This is a pitch to come work for us; info at end) 🤖 COMPUTER USE – we've built our own bespoke APIs on top of Chromium to optimize latency, accuracy, and cost of computer-using agents. Demo attached. Big breakthroughs here in recent weeks. 🛡️ ON-DEVICE MODELS – we've built our own custom infra to run everything from encoder-only models to full LLMs on device. It's cross-platform, supports LoRa adapters, and optimized for the GPU. This system preserves privacy and enables fast inference times. 🧠 MEMORY – with your permission, Dia automatically tailors your AI experiences to you, personally, based on the tabs you open while browsing normally every day. We're also bringing vertical memory to specific features. ♻️ DATA FLYWHEELS – our Fall/Winter P0 is to double-down on training custom models based on implicit signals from daily use of Dia. Dia should get smarter and more useful the more people use it. Whether via RL, auto-generated prompts, or otherwise. If this work sounds interesting to you please visit our jobs page or email [email protected]. Hiring nearly every related role -- from ML engineers to people prototyping with AI and context/prompt writers -- everyone encouraged to apply!!

Josh Miller

68,130 görüntüleme • 1 yıl önce

i'm too stubborn to simply give up on the concept of Benji after almost 10 years(!!) of iterating on it with different stacks... 2018: graphql backend + react SPA 2019: made a react native app for 6 months 2020: gave up on it and wrote 3 years of using other bs tools and being grumpy about them 2023: rewrote it in blitz js (JUGE mistake) 2024: iterate on react native + PWA + RN pwa wrapper 2025: time for a rewrite to Zero To Shipped ⛴️ stack (but next js sucks for this kind of app, add api, mcp, cross platform app, and many other things 2026: i want it all actions INSTANT AND FAST AF and next js is the wrong tech choice for a mostly client app, so i'm starting a rewrite with TANSTACK + Convex + Better Auth it feels so snappy 😭 btw 99% of the customers abandoned it, there are like 3 ride or die fans but i know i can turn it around!!!! let /goal cook.
0:18

i'm too stubborn to simply give up on the concept of Benji after almost 10 years(!!) of iterating on it with different stacks... 2018: graphql backend + react SPA 2019: made a react native app for 6 months 2020: gave up on it and wrote 3 years of using other bs tools and being grumpy about them 2023: rewrote it in blitz js (JUGE mistake) 2024: iterate on react native + PWA + RN pwa wrapper 2025: time for a rewrite to Zero To Shipped ⛴️ stack (but next js sucks for this kind of app, add api, mcp, cross platform app, and many other things 2026: i want it all actions INSTANT AND FAST AF and next js is the wrong tech choice for a mostly client app, so i'm starting a rewrite with TANSTACK + Convex + Better Auth it feels so snappy 😭 btw 99% of the customers abandoned it, there are like 3 ride or die fans but i know i can turn it around!!!! let /goal cook.

kitze · supermac.io 🐦‍🔥

16,580 görüntüleme • 3 ay önce

Native USDC, EURC, and CCTP are now live on Cronos, powering Cronos and the wider ecosystem. Developers, PSPs, and DeFi teams on Cronos can now access the world’s largest regulated dollar and euro stablecoins for a range of use cases: → DeFi activity: Use USDC and EURC for lending, trading, and settlement → Crosschain money movement: Move USDC seamlessly across chains with CCTP for robust liquidity access → Payments and treasury: Manage payouts, settlement, and treasury operations → AI-powered workflows: Support agentic payments and emerging agent-to-agent applications with programmable stablecoins Day 1 apps: Crypto.com, LI.FI, Relay, VVS-Finance, WolfSWAP | SWAP & WIN With this integration, USDC and EURC are supported natively on 35 and 7 blockchains, respectively. CCTP is now available on 25 blockchains and enables smooth USDC crosschain transfers without third-party bridges. Get started today:
0:13

Native USDC, EURC, and CCTP are now live on Cronos, powering Cronos and the wider ecosystem. Developers, PSPs, and DeFi teams on Cronos can now access the world’s largest regulated dollar and euro stablecoins for a range of use cases: → DeFi activity: Use USDC and EURC for lending, trading, and settlement → Crosschain money movement: Move USDC seamlessly across chains with CCTP for robust liquidity access → Payments and treasury: Manage payouts, settlement, and treasury operations → AI-powered workflows: Support agentic payments and emerging agent-to-agent applications with programmable stablecoins Day 1 apps: Crypto.com, LI.FI, Relay, VVS-Finance, WolfSWAP | SWAP & WIN With this integration, USDC and EURC are supported natively on 35 and 7 blockchains, respectively. CCTP is now available on 25 blockchains and enables smooth USDC crosschain transfers without third-party bridges. Get started today:

Circle

58,863 görüntüleme • 2 ay önce

Release: LichtFeld Studio v0.5.3 is out! With 316 commits merged into master, this release is a huge step forward for LichtFeld Studio. What's new in v0.5.3 • Vulkan viewer/rendering migration: New Vulkan viewport pipeline, pass graph, VkSplat renderer, Vulkan point-cloud renderer, 3DGUT/VkSplat support, improved alpha/depth composition, tighter CUDA/Vulkan interoperability, and device matching on multi-GPU systems. • RAD + LOD workflow: Added RAD file export/import, RAD LOD viewer, Spark-style GPU LOD selection, GPU-driven page prefetching, a bounded VRAM pool, out-of-core PLY-to-RAD LOD conversion, and RAD import/export speedups of approximately 3–5×. • HiGS / macro-tile inference: Added a macro-tile inference path for the Vulkan viewer, including macro sorting, batched rasterization, composition, and capacity management. • Asset Manager: Added and significantly enhanced the Asset Manager with thumbnails, SH information, faster synchronization, import-from-URL support, docked mode, data-loading popup integration, and general UI cleanup. • Viewport export: Integrated viewport export directly into the application as a toolbar/overlay tool, added fast render_view_u8-style readback paths, fixed high-resolution clipping issues, improved orthographic export parity, resolved 32K image/video export problems, and added post-export GPU resource cleanup. • Selection and tooling: Added and reworked selection toolbar controls, the Select menu, ring selection, color eyedropper, distance-from-center selection, faster point-cloud and zoomed-out selection paths, Vulkan measurement tool fixes, and drag-and-drop scene graph improvements. • UI/RmlUi platform work: Major RmlUi redesign efforts, hot reloading for RML/RCSS/Python UI files, reactive UI/store integration, viewport toolbar flyouts, improved histogram interactions, input settings enhancements, custom TRS gizmos, and numerous panel, tooltip, and localization fixes. • Windowing and UX: Added borderless window support, title bar drag/maximize/restore behavior, work-area-aware maximize functionality, resize responsiveness and performance improvements, and DPI/UI scaling fixes. • Training and data features: Added adaptive depth loss and depth gradients for the EWA rasterizer, mask loading/application fixes, a new combined Ignore+Segment mask mode, --add-splat, --freeze, improved checkpoint and training state handling, and training speed and VRAM optimizations. • COLMAP/equirectangular support: Added SPHERICAL/equirectangular camera model support and canonical EQUIRECTANGULAR handling, along with fixes for undistortion and camera export. This release will be available to all supporters as a Windows binary via approximately in about an hour. At the same time, LichtFeld Studio remains committed to being free and open source under GPLv3 and can also be built directly from source. Please consider supporting the ongoing development of LichtFeld Studio through a donation via the portal or the supporters page. Thank you to everyone who supports this project financially, contributes code, reports bugs, provides datasets, helps with the website, and contributes in countless other ways. A special thank you to our foundational sponsor Core11 and our Gold Sponsor Volinga, whose support has helped make the current state of the software possible. Thank you as well to every donor and to all of our new Bronze Sponsors. Looking ahead to v0.6 For the next major release, work will focus primarily on stability and user experience. This includes improved cleanup workflows and the ability to modify training parameters while training is in progress. I would also like to introduce a native .licht project format that allows users to save and restore their complete editor state. You can find links to our main sponsors below. Please also visit our website to discover all our Bronze Sponsors. Hint: We do not yet have a Silver Sponsor or Platinum 😉
0:17

Release: LichtFeld Studio v0.5.3 is out! With 316 commits merged into master, this release is a huge step forward for LichtFeld Studio. What's new in v0.5.3 • Vulkan viewer/rendering migration: New Vulkan viewport pipeline, pass graph, VkSplat renderer, Vulkan point-cloud renderer, 3DGUT/VkSplat support, improved alpha/depth composition, tighter CUDA/Vulkan interoperability, and device matching on multi-GPU systems. • RAD + LOD workflow: Added RAD file export/import, RAD LOD viewer, Spark-style GPU LOD selection, GPU-driven page prefetching, a bounded VRAM pool, out-of-core PLY-to-RAD LOD conversion, and RAD import/export speedups of approximately 3–5×. • HiGS / macro-tile inference: Added a macro-tile inference path for the Vulkan viewer, including macro sorting, batched rasterization, composition, and capacity management. • Asset Manager: Added and significantly enhanced the Asset Manager with thumbnails, SH information, faster synchronization, import-from-URL support, docked mode, data-loading popup integration, and general UI cleanup. • Viewport export: Integrated viewport export directly into the application as a toolbar/overlay tool, added fast render_view_u8-style readback paths, fixed high-resolution clipping issues, improved orthographic export parity, resolved 32K image/video export problems, and added post-export GPU resource cleanup. • Selection and tooling: Added and reworked selection toolbar controls, the Select menu, ring selection, color eyedropper, distance-from-center selection, faster point-cloud and zoomed-out selection paths, Vulkan measurement tool fixes, and drag-and-drop scene graph improvements. • UI/RmlUi platform work: Major RmlUi redesign efforts, hot reloading for RML/RCSS/Python UI files, reactive UI/store integration, viewport toolbar flyouts, improved histogram interactions, input settings enhancements, custom TRS gizmos, and numerous panel, tooltip, and localization fixes. • Windowing and UX: Added borderless window support, title bar drag/maximize/restore behavior, work-area-aware maximize functionality, resize responsiveness and performance improvements, and DPI/UI scaling fixes. • Training and data features: Added adaptive depth loss and depth gradients for the EWA rasterizer, mask loading/application fixes, a new combined Ignore+Segment mask mode, --add-splat, --freeze, improved checkpoint and training state handling, and training speed and VRAM optimizations. • COLMAP/equirectangular support: Added SPHERICAL/equirectangular camera model support and canonical EQUIRECTANGULAR handling, along with fixes for undistortion and camera export. This release will be available to all supporters as a Windows binary via approximately in about an hour. At the same time, LichtFeld Studio remains committed to being free and open source under GPLv3 and can also be built directly from source. Please consider supporting the ongoing development of LichtFeld Studio through a donation via the portal or the supporters page. Thank you to everyone who supports this project financially, contributes code, reports bugs, provides datasets, helps with the website, and contributes in countless other ways. A special thank you to our foundational sponsor Core11 and our Gold Sponsor Volinga, whose support has helped make the current state of the software possible. Thank you as well to every donor and to all of our new Bronze Sponsors. Looking ahead to v0.6 For the next major release, work will focus primarily on stability and user experience. This includes improved cleanup workflows and the ability to modify training parameters while training is in progress. I would also like to introduce a native .licht project format that allows users to save and restore their complete editor state. You can find links to our main sponsors below. Please also visit our website to discover all our Bronze Sponsors. Hint: We do not yet have a Silver Sponsor or Platinum 😉

MrNeRF

26,219 görüntüleme • 2 ay önce

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.
0:14

Day 11/90 of Inference Engineering How does vLLM work and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.

max fu

70,797 görüntüleme • 1 ay önce

Free NVIDIA GPU with 16 GB VRAM GPU for Running Local LLMs! If you want to master local LLMs but you're waiting until you can afford a $1,500 GPU, you're honestly not going to make it. The open source AI ecosystem is moving way too fast for you to wait on your budget to catch up. Especially when you can build a bleeding edge inference engine from scratch right now, completely for free. You don't need a heavy local rig to start. Google is literally letting you use an enterprise grade NVIDIA Tesla T4 GPU for $0/hour. At standard cloud computing rates (~$0.20/hr), Google Colab’s 4 hour daily free tier hands you roughly $24 worth of data center tier GPU compute every single month. And most people just waste it. Let’s talk about the hardware you get access to for free. The NVIDIA Tesla T4 is an absolute workhorse: - Architecture: NVIDIA Turing (TU104) - VRAM: 16GB GDDR6 (320 GB/s bandwidth) - Compute: 320 Tensor Cores | 2560 CUDA Cores - Performance: 130 TOPS INT8 | 8.1 TFLOPS FP32 - Power: Sipping energy at a max 70W TDP This is the exact same hardware I used to run DeepMind's Gemma 4 26B A4B QAT MoE at a 250,000 context window without a single Out Of Memory (OOM) crash. If you have a web browser and 10 minutes, you have everything you need. I’ve put together a fully documented, cell by cell Google Colab notebook that teaches you exactly how to do this. Here is what the notebook actually teaches you: - How to provision an Ubuntu Linux environment with CUDA 13.0 and verify your driver stack. - How to pull the source code and compile the latest llama.cpp C++ binaries from scratch, specifically optimizing the build for your exact GPU using the -DCMAKE_CUDA_ARCHITECTURES=native flag. - How to directly download quantized local LLMs (GGUF format) straight from HuggingFace using the CLI. - How to manage 16GB VRAM limits, offload neural network layers to the GPU, and push massive context windows. Compile raw llama.cpp, ollama run a model, or spin up the LM Studio CLI. Pick whatever stack you are comfortable with. just start building. No hardware. No credit card. No excuses. Bookmark this post right now so you don't lose the tutorial. Even if you don't have time to run it today, you are going to want this workflow in your engineering toolkit. The link to the free Colab Notebook is in the comments below. Lemme know if you need more tutorials like this.
0:11

Free NVIDIA GPU with 16 GB VRAM GPU for Running Local LLMs! If you want to master local LLMs but you're waiting until you can afford a $1,500 GPU, you're honestly not going to make it. The open source AI ecosystem is moving way too fast for you to wait on your budget to catch up. Especially when you can build a bleeding edge inference engine from scratch right now, completely for free. You don't need a heavy local rig to start. Google is literally letting you use an enterprise grade NVIDIA Tesla T4 GPU for $0/hour. At standard cloud computing rates (~$0.20/hr), Google Colab’s 4 hour daily free tier hands you roughly $24 worth of data center tier GPU compute every single month. And most people just waste it. Let’s talk about the hardware you get access to for free. The NVIDIA Tesla T4 is an absolute workhorse: - Architecture: NVIDIA Turing (TU104) - VRAM: 16GB GDDR6 (320 GB/s bandwidth) - Compute: 320 Tensor Cores | 2560 CUDA Cores - Performance: 130 TOPS INT8 | 8.1 TFLOPS FP32 - Power: Sipping energy at a max 70W TDP This is the exact same hardware I used to run DeepMind's Gemma 4 26B A4B QAT MoE at a 250,000 context window without a single Out Of Memory (OOM) crash. If you have a web browser and 10 minutes, you have everything you need. I’ve put together a fully documented, cell by cell Google Colab notebook that teaches you exactly how to do this. Here is what the notebook actually teaches you: - How to provision an Ubuntu Linux environment with CUDA 13.0 and verify your driver stack. - How to pull the source code and compile the latest llama.cpp C++ binaries from scratch, specifically optimizing the build for your exact GPU using the -DCMAKE_CUDA_ARCHITECTURES=native flag. - How to directly download quantized local LLMs (GGUF format) straight from HuggingFace using the CLI. - How to manage 16GB VRAM limits, offload neural network layers to the GPU, and push massive context windows. Compile raw llama.cpp, ollama run a model, or spin up the LM Studio CLI. Pick whatever stack you are comfortable with. just start building. No hardware. No credit card. No excuses. Bookmark this post right now so you don't lose the tutorial. Even if you don't have time to run it today, you are going to want this workflow in your engineering toolkit. The link to the free Colab Notebook is in the comments below. Lemme know if you need more tutorials like this.

Alok

178,744 görüntüleme • 2 ay önce

claude fable 5 is live. spawn 5.0 was built with it: 1,687 prompts, 102 sessions, my job shifted from architecture to judging taste. what we built, each of which would've been at least a month with a whole team on opus: — a from-scratch physics engine (mantle) that rivals rapier, testable in a day as a side thread — clustered froxel lighting: 8 lights to 1,000+ fully dynamic ("it's stupid fast" — creator of threejs) — realtime diffuse GI on webgpu, on your phone (landing in the next update) — million-particle gpu vfx with a shader architecture beyond what unreal and unity ship — mmo-scale netcode plus, for good measure: an (alpha) native ios app. all of it in about a week. all of it running on mobile. and the list doesn't include half of what we shipped — full changelog in the thread below. and the part the benchmarks won't show you: this model has a wonderful personality. it's genuinely funny. i laughed so hard i cried multiple times, mid-physics-rewrite. the genius and the character aren't separate features.
0:17

claude fable 5 is live. spawn 5.0 was built with it: 1,687 prompts, 102 sessions, my job shifted from architecture to judging taste. what we built, each of which would've been at least a month with a whole team on opus: — a from-scratch physics engine (mantle) that rivals rapier, testable in a day as a side thread — clustered froxel lighting: 8 lights to 1,000+ fully dynamic ("it's stupid fast" — creator of threejs) — realtime diffuse GI on webgpu, on your phone (landing in the next update) — million-particle gpu vfx with a shader architecture beyond what unreal and unity ship — mmo-scale netcode plus, for good measure: an (alpha) native ios app. all of it in about a week. all of it running on mobile. and the list doesn't include half of what we shipped — full changelog in the thread below. and the part the benchmarks won't show you: this model has a wonderful personality. it's genuinely funny. i laughed so hard i cried multiple times, mid-physics-rewrite. the genius and the character aren't separate features.

jacob

55,515 görüntüleme • 2 ay önce