Загрузка видео...

Не удалось загрузить видео

На главную

Now available: Qwen 3.8-2.4T-A95B from Alibaba Cloud on DigitalOcean Serverless Inference via NVIDIA HGX™ B300 GPUs. 🤖 1M context, built for long-horizon coding. 🔗 One API, usage-based pricing, no infra to manage.

digitalocean's profile picture

DigitalOcean

222,128 subscribers

450,372 просмотров • 1 месяц назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

There's a new fish in the sea. 🐟☁️ Efficiently scale your #AI workloads with AMD Instinct MI300X GPUs with ROCm Software, now available on #DigitalOcean in single-tenant Bare Metal configurations. 🔗
0:10

There's a new fish in the sea. 🐟☁️ Efficiently scale your #AI workloads with AMD Instinct MI300X GPUs with ROCm Software, now available on #DigitalOcean in single-tenant Bare Metal configurations. 🔗

DigitalOcean

121,910 просмотров • 1 год назад

#Developers, looking to scale your business? ✨ #DigitalOcean’s got the tools you need—powerful GPUs + expert guides to level up your #AI & #MachineLearning. 🚀☁️ Join us at Deploy in Austin, TX to learn how to run AI/ML workloads & scale effortlessly with DigitalOcean. 🔗
0:20

#Developers, looking to scale your business? ✨ #DigitalOcean’s got the tools you need—powerful GPUs + expert guides to level up your #AI & #MachineLearning. 🚀☁️ Join us at Deploy in Austin, TX to learn how to run AI/ML workloads & scale effortlessly with DigitalOcean. 🔗

DigitalOcean

33,472 просмотров • 1 год назад

Introducing the Open Intelligence Summit. We're bringing together AI founders, builders, and researchers to discuss everything open: weights, data, agents, infra, and tooling. The intelligence layer is the product now. Own it. October 12-13. San Francisco. Register here. 🔗
0:15

Introducing the Open Intelligence Summit. We're bringing together AI founders, builders, and researchers to discuss everything open: weights, data, agents, infra, and tooling. The intelligence layer is the product now. Own it. October 12-13. San Francisco. Register here. 🔗

DigitalOcean

117,048 просмотров • 18 дней назад

#DigitalOcean has everything #AI #Developers need and Deploy is the perfect place to see what we mean. ☁️👩‍💻👨‍💻💯 Register now to attend in-person or virtually. 🔗
0:20

#DigitalOcean has everything #AI #Developers need and Deploy is the perfect place to see what we mean. ☁️👩‍💻👨‍💻💯 Register now to attend in-person or virtually. 🔗

DigitalOcean

558,345 просмотров • 1 год назад

75% cost savings? Sign us up. ☁️💰✋ By switching from AWS & Heroku to #DigitalOcean, Picap experienced significant cost savings, improved performance, & unparalleled support. 🔗
0:14

75% cost savings? Sign us up. ☁️💰✋ By switching from AWS & Heroku to #DigitalOcean, Picap experienced significant cost savings, improved performance, & unparalleled support. 🔗

DigitalOcean

108,830 просмотров • 1 год назад

Congrats Laravel on launching Laravel VPS, powered by #DigitalOcean. ✨ #Laravel has leveraged DigitalOcean’s infrastructure to achieve enterprise-scale while maintaining a cost effective solution and simplifying the end-user experience. 🔗
0:10

Congrats Laravel on launching Laravel VPS, powered by #DigitalOcean. ✨ #Laravel has leveraged DigitalOcean’s infrastructure to achieve enterprise-scale while maintaining a cost effective solution and simplifying the end-user experience. 🔗

DigitalOcean

37,892 просмотров • 1 год назад

Introducing Singularity AI Machines. One-click deployable GPUs with built-in LLMs. Deploy in one click, and within minutes you get an OpenAI-compatible API running high-speed inference on compute that stays under your control. You can literally deploy a machine and: 1. Apply to OpenRouter as a provider 2. Start your own AI business 3. Join Singularity Grid in one click without owning any hardware 4. Let AI agents launch and manage your machines for you Until now, becoming an AI provider felt like something only infra companies could do. We are making it accessible to literally everyone. Singularity AI Machines are live.
0:16

Introducing Singularity AI Machines. One-click deployable GPUs with built-in LLMs. Deploy in one click, and within minutes you get an OpenAI-compatible API running high-speed inference on compute that stays under your control. You can literally deploy a machine and: 1. Apply to OpenRouter as a provider 2. Start your own AI business 3. Join Singularity Grid in one click without owning any hardware 4. Let AI agents launch and manage your machines for you Until now, becoming an AI provider felt like something only infra companies could do. We are making it accessible to literally everyone. Singularity AI Machines are live.

The Singularity Layer

13,891 просмотров • 2 месяцев назад

We're proud to collaborate with #AI leaders & innovators to give developers access to powerful cutting-edge technology through #DigitalOcean's AI Ecosystem. 🌩️🤝✨
0:14

We're proud to collaborate with #AI leaders & innovators to give developers access to powerful cutting-edge technology through #DigitalOcean's AI Ecosystem. 🌩️🤝✨

DigitalOcean

38,459 просмотров • 1 год назад

This guy built a mini AI farm out of 4 Nvidia boxes It does not look like a data center. It looks like a stack of small machines sitting next to a laptop. But each box is a DGX Spark with Grace Blackwell inside, 128GB unified memory, and enough room to run models normal gaming GPUs cannot even open. Using the launch price from the article, 4 of them is almost $12,000 of local AI compute on one desk. That sounds expensive until you compare it to cloud GPUs. A serious AI builder can burn $1,500 to $3,000 a month renting A100s and H100s for client work, fine-tunes, agents and 70B models. He basically moved that bill from the cloud into hardware he owns. 4 Nvidia boxes. 512GB unified memory. No hourly meter running in the background. No rented GPUs eating the margin every time an agent runs too long. The funny part is most people still think local AI means a slow laptop running a toy model. Meanwhile guys like this are stacking compute at home. Save this, local AI is turning into the new mining farm.
0:11

This guy built a mini AI farm out of 4 Nvidia boxes It does not look like a data center. It looks like a stack of small machines sitting next to a laptop. But each box is a DGX Spark with Grace Blackwell inside, 128GB unified memory, and enough room to run models normal gaming GPUs cannot even open. Using the launch price from the article, 4 of them is almost $12,000 of local AI compute on one desk. That sounds expensive until you compare it to cloud GPUs. A serious AI builder can burn $1,500 to $3,000 a month renting A100s and H100s for client work, fine-tunes, agents and 70B models. He basically moved that bill from the cloud into hardware he owns. 4 Nvidia boxes. 512GB unified memory. No hourly meter running in the background. No rented GPUs eating the margin every time an agent runs too long. The funny part is most people still think local AI means a slow laptop running a toy model. Meanwhile guys like this are stacking compute at home. Save this, local AI is turning into the new mining farm.

Gipp 🦅

591,405 просмотров • 4 месяцев назад

Here's what The Browser Company's AI eng & ML teams are working on for Dia right now: (This is a pitch to come work for us; info at end) 🤖 COMPUTER USE – we've built our own bespoke APIs on top of Chromium to optimize latency, accuracy, and cost of computer-using agents. Demo attached. Big breakthroughs here in recent weeks. 🛡️ ON-DEVICE MODELS – we've built our own custom infra to run everything from encoder-only models to full LLMs on device. It's cross-platform, supports LoRa adapters, and optimized for the GPU. This system preserves privacy and enables fast inference times. 🧠 MEMORY – with your permission, Dia automatically tailors your AI experiences to you, personally, based on the tabs you open while browsing normally every day. We're also bringing vertical memory to specific features. ♻️ DATA FLYWHEELS – our Fall/Winter P0 is to double-down on training custom models based on implicit signals from daily use of Dia. Dia should get smarter and more useful the more people use it. Whether via RL, auto-generated prompts, or otherwise. If this work sounds interesting to you please visit our jobs page or email careers@thebrowser.company. Hiring nearly every related role -- from ML engineers to people prototyping with AI and context/prompt writers -- everyone encouraged to apply!!
0:14

Here's what The Browser Company's AI eng & ML teams are working on for Dia right now: (This is a pitch to come work for us; info at end) 🤖 COMPUTER USE – we've built our own bespoke APIs on top of Chromium to optimize latency, accuracy, and cost of computer-using agents. Demo attached. Big breakthroughs here in recent weeks. 🛡️ ON-DEVICE MODELS – we've built our own custom infra to run everything from encoder-only models to full LLMs on device. It's cross-platform, supports LoRa adapters, and optimized for the GPU. This system preserves privacy and enables fast inference times. 🧠 MEMORY – with your permission, Dia automatically tailors your AI experiences to you, personally, based on the tabs you open while browsing normally every day. We're also bringing vertical memory to specific features. ♻️ DATA FLYWHEELS – our Fall/Winter P0 is to double-down on training custom models based on implicit signals from daily use of Dia. Dia should get smarter and more useful the more people use it. Whether via RL, auto-generated prompts, or otherwise. If this work sounds interesting to you please visit our jobs page or email [email protected]. Hiring nearly every related role -- from ML engineers to people prototyping with AI and context/prompt writers -- everyone encouraged to apply!!

Josh Miller

68,130 просмотров • 1 год назад

Introducing Pods Hyperspace Pods lets a small group of people - a family, a startup, a few friends, to pool their laptops and desktops into one AI cluster. Everyone installs the CLI, someone creates a pod, shares an invite link, and the machines form a mesh. Models like Qwen 3.5 32B or GLM-5 Turbo that need more memory than any single laptop has get automatically sharded across the group's devices - layers split proportionally, inference pipelined through the ring. From the outside it looks like one OpenAI-compatible API endpoint with a pk_* key that drops straight into your AI tools and products. No configuration beyond pasting the key and changing the base URL. A team of five paying for cloud AI burns $500–2,000 a month on API calls. The same team's existing machines can serve Qwen 3.5 (competitive on SWE-bench) and GLM-5 Turbo (#1 on BrowseComp for tool-calling and web research) for free - the hardware is already on their desks. When a query genuinely needs a frontier model nobody has locally, the pod falls back to cloud at wholesale rates from a shared treasury. But for the daily work - code reviews, refactors, research, drafting - local models handle it and nobody gets billed. And when it is idle, you can rent out your pod on the compute marketplace, with fine-grained permissions for access management. There's no central server involved in inference. Prompts go from your machine to your pod members' machines and back: all of this enabled by the fully peer-to-peer Hyperspace network. Pod state - who's a member, which API keys are valid, how much treasury is left - is replicated across members with consensus, so the whole thing works on a local network. Members behind home routers don't need port forwarding either. The practical setup for most pods is three models covering different jobs: Qwen 3.5 32B for code and reasoning, GLM-5 Turbo for browsing and research, Gemma 4 for fast lightweight tasks. All running on hardware you already own. Pods ship today in Hyperspace v5.19. Model sharding, API keys, treasury, and Raft coordinator are all live. What Makes This Different - No middleman. Your prompts travel from your IDE to your pod members' hardware and back. There is no server in between reading your data. - No vendor lock-in. Pod membership, API keys, and treasury are replicated across your own machines using Raft consensus. If the internet goes down, your local network keeps working. There is no database in someone else's cloud that your pod depends on. - Automatic sharding. You don't configure layer ranges or calculate VRAM budgets. Tell the pod which model you want. It figures out how to split it across whatever hardware is online. - Real NAT traversal. Your friend behind a home router with a dynamic IP? Works. No VPN, no Tailscale, no port forwarding. The nodes handle it. - Free when local. This is the part that matters most. Cloud AI bills scale with usage. Pod inference on local hardware scales with nothing. The marginal cost of your 10,000th prompt is the electricity your laptop was already using. Coming soon: - Pod federation: pods form alliances with other pods. - Marketplace: pods with spare capacity can sell inference to other pods.
0:19

Introducing Pods Hyperspace Pods lets a small group of people - a family, a startup, a few friends, to pool their laptops and desktops into one AI cluster. Everyone installs the CLI, someone creates a pod, shares an invite link, and the machines form a mesh. Models like Qwen 3.5 32B or GLM-5 Turbo that need more memory than any single laptop has get automatically sharded across the group's devices - layers split proportionally, inference pipelined through the ring. From the outside it looks like one OpenAI-compatible API endpoint with a pk_* key that drops straight into your AI tools and products. No configuration beyond pasting the key and changing the base URL. A team of five paying for cloud AI burns $500–2,000 a month on API calls. The same team's existing machines can serve Qwen 3.5 (competitive on SWE-bench) and GLM-5 Turbo (#1 on BrowseComp for tool-calling and web research) for free - the hardware is already on their desks. When a query genuinely needs a frontier model nobody has locally, the pod falls back to cloud at wholesale rates from a shared treasury. But for the daily work - code reviews, refactors, research, drafting - local models handle it and nobody gets billed. And when it is idle, you can rent out your pod on the compute marketplace, with fine-grained permissions for access management. There's no central server involved in inference. Prompts go from your machine to your pod members' machines and back: all of this enabled by the fully peer-to-peer Hyperspace network. Pod state - who's a member, which API keys are valid, how much treasury is left - is replicated across members with consensus, so the whole thing works on a local network. Members behind home routers don't need port forwarding either. The practical setup for most pods is three models covering different jobs: Qwen 3.5 32B for code and reasoning, GLM-5 Turbo for browsing and research, Gemma 4 for fast lightweight tasks. All running on hardware you already own. Pods ship today in Hyperspace v5.19. Model sharding, API keys, treasury, and Raft coordinator are all live. What Makes This Different - No middleman. Your prompts travel from your IDE to your pod members' hardware and back. There is no server in between reading your data. - No vendor lock-in. Pod membership, API keys, and treasury are replicated across your own machines using Raft consensus. If the internet goes down, your local network keeps working. There is no database in someone else's cloud that your pod depends on. - Automatic sharding. You don't configure layer ranges or calculate VRAM budgets. Tell the pod which model you want. It figures out how to split it across whatever hardware is online. - Real NAT traversal. Your friend behind a home router with a dynamic IP? Works. No VPN, no Tailscale, no port forwarding. The nodes handle it. - Free when local. This is the part that matters most. Cloud AI bills scale with usage. Pod inference on local hardware scales with nothing. The marginal cost of your 10,000th prompt is the electricity your laptop was already using. Coming soon: - Pod federation: pods form alliances with other pods. - Marketplace: pods with spare capacity can sell inference to other pods.

Varun

309,792 просмотров • 5 месяцев назад

The "I don't have enough VRAM" excuse just died. I’m running Meta’s new 30B Muse Glimmer Q6_K_XL with a massive 130k context window on just 26GB VRAM FREE compute on Kaggle. Kaggle provides you free 2x Nvidia T4 GPUs. 30 hours usage each week! Yesterday, I showed you the violent throughput of Muse Glimmer on a single RTX 4090. Today, we are securing a Dual NVIDIA T4 GPU cluster with 32GB of total VRAM for exactly $0 and dropping the massive 24.5GB Q6_K_XL GGUF onto it. Here is the exact Kaggle workflow and benchmarking breakdown: # 1. The Storage Bypass & Setup I built a clean cell by cell script in the file. We dynamically fetch the CUDA accelerated llama.cpp binaries and use wget to stream the model directly into Kaggle's /kaggle/tmp scratch storage, which cleanly bypasses their 19.5GB output directory limit. # 2. The Multi GPU Performance With the -ngl 99 flag offloading all model layers across both T4 GPUs (32GB VRAM combined), we pushed a massive 131,072 token context window (-c 131072). The benchmark numbers: Prefill: 265.9 t/s Decode: 9.0 t/s VRAM Total: 26.5 GB # 3. The Architecture Insight The Q6_K_XL model itself is 24.5 GB. Because of Muse Glimmer's aggressive 16:1 GQA, the unquantized KV cache for a massive 130k context window only takes up 2 GB of memory. No heavily degraded Q4 KV quantization required. It just works. No compiling from source. No credit card. No OOM crashes. Zero excuses. If you’re running a single RTX 3090, 4090, or 5090, you need to experience this hyper efficient KV cache right now before the upcoming Qwen 3.8 27B drop completely steals your VRAM tomorrow. pick the Q4 or Q5 quants for 24 GB VRAM rigs. I'm dropping the Unsloth huggingface GGUF links and the free Kaggle notebook link in the replies. spin up your own instance, and show me your multi GPU benchmarks.
0:18

The "I don't have enough VRAM" excuse just died. I’m running Meta’s new 30B Muse Glimmer Q6_K_XL with a massive 130k context window on just 26GB VRAM FREE compute on Kaggle. Kaggle provides you free 2x Nvidia T4 GPUs. 30 hours usage each week! Yesterday, I showed you the violent throughput of Muse Glimmer on a single RTX 4090. Today, we are securing a Dual NVIDIA T4 GPU cluster with 32GB of total VRAM for exactly $0 and dropping the massive 24.5GB Q6_K_XL GGUF onto it. Here is the exact Kaggle workflow and benchmarking breakdown: # 1. The Storage Bypass & Setup I built a clean cell by cell script in the file. We dynamically fetch the CUDA accelerated llama.cpp binaries and use wget to stream the model directly into Kaggle's /kaggle/tmp scratch storage, which cleanly bypasses their 19.5GB output directory limit. # 2. The Multi GPU Performance With the -ngl 99 flag offloading all model layers across both T4 GPUs (32GB VRAM combined), we pushed a massive 131,072 token context window (-c 131072). The benchmark numbers: Prefill: 265.9 t/s Decode: 9.0 t/s VRAM Total: 26.5 GB # 3. The Architecture Insight The Q6_K_XL model itself is 24.5 GB. Because of Muse Glimmer's aggressive 16:1 GQA, the unquantized KV cache for a massive 130k context window only takes up 2 GB of memory. No heavily degraded Q4 KV quantization required. It just works. No compiling from source. No credit card. No OOM crashes. Zero excuses. If you’re running a single RTX 3090, 4090, or 5090, you need to experience this hyper efficient KV cache right now before the upcoming Qwen 3.8 27B drop completely steals your VRAM tomorrow. pick the Q4 or Q5 quants for 24 GB VRAM rigs. I'm dropping the Unsloth huggingface GGUF links and the free Kaggle notebook link in the replies. spin up your own instance, and show me your multi GPU benchmarks.

Alok

19,370 просмотров • 1 месяц назад

Introducing fx, a tiny, open, native coding agent from Vercel Labs. Originally an internal tool, fx is a harness and CLI written in Zig, optimized for research and embedding in larger systems. Today, we're open sourcing it. fx is built on three principles: 1. Fast. A single native binary, no runtime to install. It cold starts in 10µs and does no unnecessary work or I/O before accepting input. fx is the answer to "how fast can a coding agent be?" 2. Light. The 6.3MiB binary uses single-digit megabytes of memory at baseline, made for instant installation and embedding in resource-constrained environments and agent sandboxes. 3. Open. Apache-2.0, model and provider agnostic, suitable for local and cloud inference. Its small core extends through skills, plugins, and MCP. Minimalism is an obsession throughout the entire harness: system prompt, tools, features, binary. The goal was to keep context usage and time to first token low, and make fx optimal for model benchmarking, sandboxing, evals, and gyms. You can use fx directly or embed it as infrastructure. The CLI feels more like a Unix shell than an IDE in the terminal: it preserves scroll history, produces minimal output, and uses complex TUI rendering very, very sparingly. Programmatically, 𝚏𝚡 𝚊𝚜𝚔 --𝚓𝚜𝚘𝚗 gives structured output, 𝚏𝚡 𝚊𝚌𝚙 connects to editors and other clients, and WebAssembly can even run the whole thing inside the browser (see: Privacy is a design constraint: no product telemetry, sessions and usage stay local, and no source code or prompts are shared with any endpoint other than inference. With local inference and auto-updates off, fx is fully hermetic. fx is experimental. Use at your own risk and expect frequent changes. Chat with us on X ( or file issues ( 𝚌𝚞𝚛𝚕 -𝚏𝚜𝚂𝙻 𝚏𝚡.𝚜𝚑/𝚜𝚎𝚝𝚞𝚙.𝚜𝚑 | 𝚋𝚊𝚜𝚑
0:11

Introducing fx, a tiny, open, native coding agent from Vercel Labs. Originally an internal tool, fx is a harness and CLI written in Zig, optimized for research and embedding in larger systems. Today, we're open sourcing it. fx is built on three principles: 1. Fast. A single native binary, no runtime to install. It cold starts in 10µs and does no unnecessary work or I/O before accepting input. fx is the answer to "how fast can a coding agent be?" 2. Light. The 6.3MiB binary uses single-digit megabytes of memory at baseline, made for instant installation and embedding in resource-constrained environments and agent sandboxes. 3. Open. Apache-2.0, model and provider agnostic, suitable for local and cloud inference. Its small core extends through skills, plugins, and MCP. Minimalism is an obsession throughout the entire harness: system prompt, tools, features, binary. The goal was to keep context usage and time to first token low, and make fx optimal for model benchmarking, sandboxing, evals, and gyms. You can use fx directly or embed it as infrastructure. The CLI feels more like a Unix shell than an IDE in the terminal: it preserves scroll history, produces minimal output, and uses complex TUI rendering very, very sparingly. Programmatically, 𝚏𝚡 𝚊𝚜𝚔 --𝚓𝚜𝚘𝚗 gives structured output, 𝚏𝚡 𝚊𝚌𝚙 connects to editors and other clients, and WebAssembly can even run the whole thing inside the browser (see: Privacy is a design constraint: no product telemetry, sessions and usage stay local, and no source code or prompts are shared with any endpoint other than inference. With local inference and auto-updates off, fx is fully hermetic. fx is experimental. Use at your own risk and expect frequent changes. Chat with us on X ( or file issues ( 𝚌𝚞𝚛𝚕 -𝚏𝚜𝚂𝙻 𝚏𝚡.𝚜𝚑/𝚜𝚎𝚝𝚞𝚙.𝚜𝚑 | 𝚋𝚊𝚜𝚑

Vercel Developers

962,552 просмотров • 1 месяц назад

single RTX 3090. 24 GB VRAM. Qwen3.5-35B-A3B. 4-bit quant, 113 tokens per second at full 262K context harnessing Claude Code locally with no API, no subscription, no proxy. told it what it is. 30 Mamba2 layers, 10 attention, 256 experts, 8 active per token. said "build something that shows off what you can do." it visualized its own architecture. interactive. tokens flowing through layers. 256 experts lighting up on routing. served in the browser from the same GPU running inference. single prompt. then i said level up. 3D. Three.js. separate files. flythrough camera. clickable layers. it planned first, scaffolded 6 files, hit one API bug, fixed it itself, then optimized for smooth framerate. two iterations to a working 3D neural network explorer. llama.cpp just merged a native Anthropic endpoint. Claude Code points at localhost. the whole setup is two commands. no LiteLLM. no proxy config. the open source models coming out of china right now are genuinely changing what's possible on consumer hardware. respect to the Qwen team. this is acceleration.
0:07

single RTX 3090. 24 GB VRAM. Qwen3.5-35B-A3B. 4-bit quant, 113 tokens per second at full 262K context harnessing Claude Code locally with no API, no subscription, no proxy. told it what it is. 30 Mamba2 layers, 10 attention, 256 experts, 8 active per token. said "build something that shows off what you can do." it visualized its own architecture. interactive. tokens flowing through layers. 256 experts lighting up on routing. served in the browser from the same GPU running inference. single prompt. then i said level up. 3D. Three.js. separate files. flythrough camera. clickable layers. it planned first, scaffolded 6 files, hit one API bug, fixed it itself, then optimized for smooth framerate. two iterations to a working 3D neural network explorer. llama.cpp just merged a native Anthropic endpoint. Claude Code points at localhost. the whole setup is two commands. no LiteLLM. no proxy config. the open source models coming out of china right now are genuinely changing what's possible on consumer hardware. respect to the Qwen team. this is acceleration.

Sudo su

110,464 просмотров • 7 месяцев назад

This Chinese developer launched Llama 70B locally on a MacBook on a plane and for a full 11 hours without internet ran client projects. He was sitting by the window on a transatlantic flight with a MacBook Pro M4 with 64 GB of memory. WiFi on board cost $25 for the flight. He declined. No cloud API, no connection to Anthropic or OpenAI servers, no internet at all. Just a local Llama 3.3 70B on bf16 and his own orchestrator script. The model runs through llama.cpp. Generation speed, 71 tokens per second. Context around 60,000 tokens. Memory usage, 48.6 GiB out of 64. Battery at takeoff, 3 hours 21 minutes. And he gave the orchestrator this system prompt before takeoff: "You are an offline orchestrator running on a single MacBook. There is no network. The only resources you have are local files in /Users/dev/work, the Llama 70B inference server at localhost:8080, and a battery budget of 3 hours 21 minutes. Process the queue at /Users/dev/work/queue.jsonl (one client task per line). For each task: draft → run local evals → save artefact to /Users/dev/work/done/. Save context checkpoints every 12 tasks so you can resume after a battery swap. Stop only on empty queue or when battery drops below 5%." So the system knows exactly what resources it is running on. It knows it has no connection to the outside world for the next 11 hours. It knows it has finite memory and a finite battery. It knows the human will not intervene until the plane lands. The system runs in 1 loop. Takes a task from the queue, runs it through inference, saves the artifact, writes a checkpoint. Task after task, just like that. And only when the battery drops below 5% does the orchestrator automatically pause, waits for the laptop to switch to the backup power bank, and continues from the last checkpoint. Here is what the system actually writes in his log during the flight: "saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)" "restored context checkpoint (pos_min = 488, pos_max = 50118)" "prompt processing progress: n_tokens = 50 / 60 818" "task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md" Outside the window, clouds, blue sky, and no WiFi. On the tray, 1 MacBook, an open terminal on 2 screens, and an inference server on localhost. From what I have observed, this is the cleanest offline AI workflow I have seen in the past year: 11 hours of flight, $0 for WiFi, and the entire client queue closed before landing.
0:10

This Chinese developer launched Llama 70B locally on a MacBook on a plane and for a full 11 hours without internet ran client projects. He was sitting by the window on a transatlantic flight with a MacBook Pro M4 with 64 GB of memory. WiFi on board cost $25 for the flight. He declined. No cloud API, no connection to Anthropic or OpenAI servers, no internet at all. Just a local Llama 3.3 70B on bf16 and his own orchestrator script. The model runs through llama.cpp. Generation speed, 71 tokens per second. Context around 60,000 tokens. Memory usage, 48.6 GiB out of 64. Battery at takeoff, 3 hours 21 minutes. And he gave the orchestrator this system prompt before takeoff: "You are an offline orchestrator running on a single MacBook. There is no network. The only resources you have are local files in /Users/dev/work, the Llama 70B inference server at localhost:8080, and a battery budget of 3 hours 21 minutes. Process the queue at /Users/dev/work/queue.jsonl (one client task per line). For each task: draft → run local evals → save artefact to /Users/dev/work/done/. Save context checkpoints every 12 tasks so you can resume after a battery swap. Stop only on empty queue or when battery drops below 5%." So the system knows exactly what resources it is running on. It knows it has no connection to the outside world for the next 11 hours. It knows it has finite memory and a finite battery. It knows the human will not intervene until the plane lands. The system runs in 1 loop. Takes a task from the queue, runs it through inference, saves the artifact, writes a checkpoint. Task after task, just like that. And only when the battery drops below 5% does the orchestrator automatically pause, waits for the laptop to switch to the backup power bank, and continues from the last checkpoint. Here is what the system actually writes in his log during the flight: "saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)" "restored context checkpoint (pos_min = 488, pos_max = 50118)" "prompt processing progress: n_tokens = 50 / 60 818" "task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md" Outside the window, clouds, blue sky, and no WiFi. On the tray, 1 MacBook, an open terminal on 2 screens, and an inference server on localhost. From what I have observed, this is the cleanest offline AI workflow I have seen in the past year: 11 hours of flight, $0 for WiFi, and the entire client queue closed before landing.

Blaze

1,843,280 просмотров • 5 месяцев назад

so I wired Nous Research Hermes Agent up so it can live as a Grok Bot GROKBOT inside the grokbot app (private net / Tailscale). two ways, pick based on how your Hermes host runs: Hermes SSH Relay — Desktop-primary, no API gateway. SSH ask-relay, simplest “talk to hermes from inside grokbot app”. Hermes API Fleet — gateway + native API on :8642. better when you want parallel fleet asks / less hop overhead. same repo ⬆️ (docs for both) Grokbot Bot Templates: Hermes SSH Relay: Hermes API Fleet : what it unlocks: chat your hermes agent from grokbot UI, run herdr-style fleet ops from one place, keep hermes brains on the Mac while grok handles the front door. top 3 uses we care about: • identity bots per host so you always know who you’re talking to • easy orchestration across the Hermes fleet without jumping SSH by hand • bridge + connector pattern for hermes ↔ grokbot comms without stuffing secrets in the template Ps, imho: Dont point grok at writing into hermes memory plugins unless you mean to. (I let mine READ hermes Hindsight memory but not write to it) With this I now have a single CHIEF OF STAFF I talk to as a grokbot who orchestrates across 7 hermes machines, and 20+ grokbots, all running various coding harnesses, AI pipelines, GTM, AEO/SEO/GEO, UGC, seamlessly. I'm even having grokbot tell my chief Hermes Agent to run long horizon kanban goals. I'm also using a dedicated bot to keep my entire hermes fleet updated, maintained, pruned, local LLM optimized etc. I'll drop some of those templates over the next few days... follow with notis. Oh and shit a token usage window maximizer I'll share soon...burning it in this week, so far, have been blasting at 10x the rate across a dozen harnesses/subs. This is greatest unlock EVER in 7 months of harness jockeying. I am finally ONLY a HUMAN making decisions, ideating, all via a single grok bot Chief of Staff. Valhalla!
0:10

so I wired Nous Research Hermes Agent up so it can live as a Grok Bot GROKBOT inside the grokbot app (private net / Tailscale). two ways, pick based on how your Hermes host runs: Hermes SSH Relay — Desktop-primary, no API gateway. SSH ask-relay, simplest “talk to hermes from inside grokbot app”. Hermes API Fleet — gateway + native API on :8642. better when you want parallel fleet asks / less hop overhead. same repo ⬆️ (docs for both) Grokbot Bot Templates: Hermes SSH Relay: Hermes API Fleet : what it unlocks: chat your hermes agent from grokbot UI, run herdr-style fleet ops from one place, keep hermes brains on the Mac while grok handles the front door. top 3 uses we care about: • identity bots per host so you always know who you’re talking to • easy orchestration across the Hermes fleet without jumping SSH by hand • bridge + connector pattern for hermes ↔ grokbot comms without stuffing secrets in the template Ps, imho: Dont point grok at writing into hermes memory plugins unless you mean to. (I let mine READ hermes Hindsight memory but not write to it) With this I now have a single CHIEF OF STAFF I talk to as a grokbot who orchestrates across 7 hermes machines, and 20+ grokbots, all running various coding harnesses, AI pipelines, GTM, AEO/SEO/GEO, UGC, seamlessly. I'm even having grokbot tell my chief Hermes Agent to run long horizon kanban goals. I'm also using a dedicated bot to keep my entire hermes fleet updated, maintained, pruned, local LLM optimized etc. I'll drop some of those templates over the next few days... follow with notis. Oh and shit a token usage window maximizer I'll share soon...burning it in this week, so far, have been blasting at 10x the rate across a dozen harnesses/subs. This is greatest unlock EVER in 7 months of harness jockeying. I am finally ONLY a HUMAN making decisions, ideating, all via a single grok bot Chief of Staff. Valhalla!

PixelRainbow

151,643 просмотров • 13 дней назад

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?
0:14

50% more context unlocked for Qwen 3.8 27b Q4_K_XL dflash 2 on a single RTX 4090 (24 GB VRAM) I found a hidden VRAM tax in llama.cpp. By combining my custom 2 bit DFlash 2 drafter with one overlooked server flag, I just unlocked another +80,000 tokens of context. Qwen3.8-27B is now running a massive 250,000 context at 75 tokens/s on a single RTX 4090. Here is the secret: By default, `llama-server` reserves massive chunks of your VRAM to handle multiple concurrent users (batching). If you are running a single user session, you are bleeding memory for features you aren't using. By passing the `--parallel 1` flag, you force the engine to dedicate 100% of your 24GB VRAM buffer to a single user. When we combine the VRAM saved by our Q2_K 2-bit drafter with the VRAM saved by `--parallel 1`, the context ceilings absolutely explode: Note: all benchmarks carried out with a massive 28k prompt. Ubuntu 22. ### THE NEW 24GB PHYSICAL LIMITS (Single RTX 4090): # 1. The "Repo Swallower" (Q4 KV Cache): - Context: 250,000 tokens (Up from 170k!) - Speed: 73.66 t/s decode | 1,608 t/s prefill - Peak VRAM: 23.8 GB # 2. The "High-Precision SWE" (Q8 KV Cache): - Context: 150,000 tokens (Up from 100k!) - Speed: 75.01 t/s decode | 1,667 t/s prefill - Peak VRAM: 23.9 GB # 3. The "Pristine Attention" (Unquantized FP16 KV): - Context: 90,000 tokens - Speed: 80.58 t/s decode | 1,699 t/s prefill - Peak VRAM: 23.92 GB ### HOW TO RUN THE 250K GOD STACK TODAY: (Requires PR #27342 + my Q2_K Hugging Face drafter) llama.cpp flags: ./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -md Qwen3.8-27B-DFlash2-Q2_K.gguf --spec-type draft-dflash --spec-draft-n-max 3 -c 250000 -ngl 99 --parallel 1 --port 8080 -ctv q4_0 -ctk q4_0 We are pushing a quarter million tokens of context with speculative DFlash 2 decoding at 73 tokens/second on a single consumer gaming GPU. I dropped my custom 2 bit Hugging Face GGUF links, visual performance graphs, and the PR #27342 build instructions in the replies below. If you own a single RTX 3090 or 4090, it is officially time to cancel your API subscriptions and let local silicon eat the cloud. how much monthly API spend does an optimized 4090 rig like this actually replace for you?

Alok

39,189 просмотров • 1 месяц назад