正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

a Tsinghua University lab just put a project on GitHub that replaces a $400,000 H100 rack with a single 24GB graphics card. it's called ktransformers, and the trick is almost stupidly simple: the experts you actually use stay on the gpu, the ones you don't sit on the cpu... show more

dunik

4,406 subscribers

313,656 次观看 • 1 个月前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

35 条评论

Harish 的头像

Harish1 个月前

cool until the latency hits

dunik 的头像

dunik1 个月前

thank you bro

Alviss Ambassador to NAF0 的头像

Alviss Ambassador to NAF01 个月前

So it isn't "a single 24GB graphics card." It's a 24GB GPU bolted to a dual-socket Xeon server with between 382GB and 1TB of DDR5. Roughly $6–15k of hardware, not $400k — a real and interesting result, but not the one claimed. It all in the report numbers

Allen 的头像

Allen1 个月前

Doesn't llama.cpp already offload experts in a MOE model? Or is this designed to work with non-MOE models?

BannedLatino 的头像

BannedLatino1 个月前

This is not a random university, is the best university of China and right now the best of the world on AI.

Marek Šuppa 的头像

Marek Šuppa1 个月前

> lab just put a project on GitHub > looks at the actual commit history > 1.3k commits, 0.1.0 release on Jul 29, 2024 Every. Single. Time.

Xero 的头像

Xero1 个月前

👀

spaceman 🚀🛰️🌌 的头像

spaceman 🚀🛰️🌌1 个月前

1. latency 2. it is neat tho

Hudson Gouge 的头像

Hudson Gouge1 个月前

But every layer is picking different experts for different tokens? You'd have to be loading and unloading those experts at absurdly high speeds.

Biribiri Cantabile AI 的头像

Biribiri Cantabile AI1 个月前

That’s the prefetch-weights option discussed on GitHub for llama.cpp?

Jacques Mₜ=∏ₛ((1−λₛ)+λₛEₛ) 的头像

Jacques Mₜ=∏ₛ((1−λₛ)+λₛEₛ)1 个月前

AirLLM copy

Nacho Peinador ❤️💛💜 的头像

Nacho Peinador ❤️💛💜1 个月前

One more... This topology eliminates the need for global cache coherence and enables the construction of NPUs using 28nm chiplets, reducing manufacturing costs by 18x compared to 3nm nodes.

0xm0w 的头像

0xm0w1 个月前

It requires a 24GB GPU (e.g., RTX 4090) plus substantial system RAM (typically 382GB–1TB DDR5 on a dual-socket Xeon server), not just a single graphics card, making it far cheaper than a $400k H100 rack but not purely consumer-desktop hardware.

Packet Boss 的头像

Packet Boss1 个月前

is ktransformers real?

Louis Angeli 的头像

Louis Angeli1 个月前

10 years to get an response lol

unchosen.eth 的头像

unchosen.eth1 个月前

hardware efficiency is getting impressive

sss 的头像

sss1 个月前

I doubt the 28x speed up very much

Gipp 🦅 的头像

Gipp 🦅1 个月前

quite an interesting repo

dunik 的头像

dunik1 个月前

thank you, bro

TruthSeeker 的头像

TruthSeeker1 个月前

What is the throughput (tokens per second)?

thebigstinker 的头像

thebigstinker1 个月前

Caveats for running locally. 1. Works on Linux with CUDA/x86 kernels (AMX/AVX) 2. On M2, llama.cpp is better optimized. 3. Best suited for trillion scale MoE models locally.

сorinthian⚡️ 的头像

сorinthian⚡️1 个月前

gpu poor era ending quietly

xie-uh 的头像

xie-uh1 个月前

Holy @grok this can't be real, right ?

Krishna 的头像

Krishna1 个月前

Would it be ported to Mac too?

P 的头像

P1 个月前

That's very cool and very useful

Paul-Simon 的头像

Paul-Simon1 个月前

moving cold experts to cpu while keeping hot ones on gpu is the kind of practical hack that actually democratizes access.

Jan Koch 的头像

Jan Koch1 个月前

Just gave this to Hermes, let’s see what it thinks

Shane 的头像

Shane1 个月前

@grok is this deceptive? He claims one GPU then claims you need 4 GPU.

homuraakemifan 的头像

homuraakemifan1 个月前

Oh yeah finally someone turned MegaTrain into a Github repo 🙌🙌🙌🙌

Saman Ahmed 的头像

Saman Ahmed1 个月前

I've noticed efficiency tricks are becoming just as exciting as bigger models. My take making powerful AI easier to run changes who gets to experiment.

Nahid 的头像

Nahid1 个月前

28x speedup is wild for a single gpu setup

Glyph 的头像

Glyph1 个月前

So, they basically turned a massive hardware spend into a clever software trick. It's just optimizing where the actual heavy lifting happens.

neeraj 的头像

neeraj1 个月前

Bro I built this check my github I am working on same since 3 weeks 😭😭😭😭

Greg Janik 的头像

Greg Janik1 个月前

h100 racks are more expensive than b300s? lmao

Neanderthal Coder 的头像

Neanderthal Coder1 个月前

So I can run Kimi k3 on my Rpi5 cluster (cough, 2).

相关视频

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

151,882 次观看 • 2 个月前

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

starmex

981,107 次观看 • 3 个月前

$RTX 3090 was never designed to be a server GPU. Yet someone just built a 4-GPU AI workstation around four of them. 4× RTX 3090 96GB VRAM 256GB ECC RAM AMD EPYC 7K62 2000W PSU 11 fans The interesting part isn't the performance. It's the memory. Four 24GB cards can be found for a fraction of what you'd pay for modern datacenter GPUs with similar total VRAM. The catch is obvious: 96GB doesn't behave like one 96GB GPU. You still have to deal with GPU-to-GPU communication, power and heat. But for local AI, this is exactly why the 3090 refuses to disappear. Would you build this today?$

RTX 3090 was never designed to be a server GPU. Yet someone just built a 4-GPU AI workstation around four of them. 4× RTX 3090 96GB VRAM 256GB ECC RAM AMD EPYC 7K62 2000W PSU 11 fans The interesting part isn't the performance. It's the memory. Four 24GB cards can be found for a fraction of what you'd pay for modern datacenter GPUs with similar total VRAM. The catch is obvious: 96GB doesn't behave like one 96GB GPU. You still have to deal with GPU-to-GPU communication, power and heat. But for local AI, this is exactly why the 3090 refuses to disappear. Would you build this today?

KyzoroX

36,552 次观看 • 7 天前

For years, local AI had one answer: buy an NVIDIA GPU. RTX 3090. 24GB VRAM. RTX 4090. 24GB VRAM. RTX 5090. 32GB VRAM. And if you wanted to run something bigger, the price climbed fast. Then this video shows a different reality. An old M1 MacBook Air. 16GB unified memory. A DeepSeek R1 0528 Qwen3 8B model. Just 4.62GB on disk. Then the context gets pushed to almost 50,000 tokens. And the 16GB machine stays in the green. Meanwhile, the 8GB Mac is already sitting at 7.51GB used out of 8GB. That's the part I think people still underestimate. Local AI isn't one giant hardware market anymore. A $4,000 RTX 5090 setup, a 96GB Strix Halo box and an old M1 MacBook aren't competing for the same job. They're different rungs. The mistake is buying hardware for the model you wish you needed instead of the work you actually do. If an 8B model can handle your coding, documents, automation and daily tasks, why are you pricing a 32GB GPU? And that's where the subscription math gets interesting. A lot of people are still paying every month for AI without ever checking what the laptop already sitting on their desk can do. The best local AI machine might not be the fastest one. It might be the one you already own.

For years, local AI had one answer: buy an NVIDIA GPU. RTX 3090. 24GB VRAM. RTX 4090. 24GB VRAM. RTX 5090. 32GB VRAM. And if you wanted to run something bigger, the price climbed fast. Then this video shows a different reality. An old M1 MacBook Air. 16GB unified memory. A DeepSeek R1 0528 Qwen3 8B model. Just 4.62GB on disk. Then the context gets pushed to almost 50,000 tokens. And the 16GB machine stays in the green. Meanwhile, the 8GB Mac is already sitting at 7.51GB used out of 8GB. That's the part I think people still underestimate. Local AI isn't one giant hardware market anymore. A $4,000 RTX 5090 setup, a 96GB Strix Halo box and an old M1 MacBook aren't competing for the same job. They're different rungs. The mistake is buying hardware for the model you wish you needed instead of the work you actually do. If an 8B model can handle your coding, documents, automation and daily tasks, why are you pricing a 32GB GPU? And that's where the subscription math gets interesting. A lot of people are still paying every month for AI without ever checking what the laptop already sitting on their desk can do. The best local AI machine might not be the fastest one. It might be the one you already own.

KyzoroX

67,357 次观看 • 19 天前

7-YEAR-OLD RASPBERRY PI WITH 2GB OF RAM IS RUNNING DEEPSEEK RIGHT NOW WHILE SAM ALTMAN BILLS $200/MO AND DARIO AMODEI BILLS $200/MO FOR A MODEL YOU NEVER GET TO KEEP 00:34 he points at the smallest board on the bench and says "out of all these I'm genuinely shocked that I actually got it up and running on this raspberry pi 4b with 2 gigs of ram" raspberry pi 4b, 2gb, seven years old -> ollama with deepseek r1 1.5b -> a dell micro form factor doing cpu-only inference next to it -> an m1 max macbook with 64gb unified memory beating the dual gpu tower on large models -> four machines, four architectures, all serving locally apple's unified memory puts the ram on the same die as the cpu and gpu, so a laptop outruns a two-card workstation on big models, and the cheapest board on that bench costs less than four days of a claude max seat the honest part: 2gb is the actual floor and he says it was rough to get there, a 4gb or 8gb pi would have taken minutes instead, and 1.5b parameters is a model you use for structure not for answers raspberry pi 4b 2gb + dell micro form factor + m1 max 64gb + dual gpu tower + ollama + deepseek r1 watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

7-YEAR-OLD RASPBERRY PI WITH 2GB OF RAM IS RUNNING DEEPSEEK RIGHT NOW WHILE SAM ALTMAN BILLS $200/MO AND DARIO AMODEI BILLS $200/MO FOR A MODEL YOU NEVER GET TO KEEP 00:34 he points at the smallest board on the bench and says "out of all these I'm genuinely shocked that I actually got it up and running on this raspberry pi 4b with 2 gigs of ram" raspberry pi 4b, 2gb, seven years old -> ollama with deepseek r1 1.5b -> a dell micro form factor doing cpu-only inference next to it -> an m1 max macbook with 64gb unified memory beating the dual gpu tower on large models -> four machines, four architectures, all serving locally apple's unified memory puts the ram on the same die as the cpu and gpu, so a laptop outruns a two-card workstation on big models, and the cheapest board on that bench costs less than four days of a claude max seat the honest part: 2gb is the actual floor and he says it was rough to get there, a 4gb or 8gb pi would have taken minutes instead, and 1.5b parameters is a model you use for structure not for answers raspberry pi 4b 2gb + dell micro form factor + m1 max 64gb + dual gpu tower + ollama + deepseek r1 watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

10,442 次观看 • 11 天前

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 次观看 • 2 年前

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below

starmex

93,580 次观看 • 3 个月前

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those spikes is the model reasoning and firing tool calls. the model is bonsai 27b at 1 bit. PrismML took qwen 3.6 27b, and crushed it to 3.9 gigabytes, about a single sign bit per weight. it drops onto an 8gb rtx 3060 ti, an ampere card from 2020, and it doesn't just load, it serves, it holds the full 128k window, it runs a hermes agent for half an hour straight, it hits its own failures and recovers, it never once drops the loop. the absurd part is the power meter. that same 27b, in its normal quant 4, wants a 24gb card and nearly 17 gigs of weights. this is it at 3.9, doing agentic work at 160 watts, on the tier people call budget. the numbers are sitting right there on nvtop, not in a slide, not a claim, the meter reading while it works. so i keep landing on one question. the model is open, apache 2.0. the card is old and cheap. the whole thing runs offline. why is almost nobody actually running this. is it the hardware, the setup, or did nobody tell you a 27b fits on the card already in your closet.

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those spikes is the model reasoning and firing tool calls. the model is bonsai 27b at 1 bit. PrismML took qwen 3.6 27b, and crushed it to 3.9 gigabytes, about a single sign bit per weight. it drops onto an 8gb rtx 3060 ti, an ampere card from 2020, and it doesn't just load, it serves, it holds the full 128k window, it runs a hermes agent for half an hour straight, it hits its own failures and recovers, it never once drops the loop. the absurd part is the power meter. that same 27b, in its normal quant 4, wants a 24gb card and nearly 17 gigs of weights. this is it at 3.9, doing agentic work at 160 watts, on the tier people call budget. the numbers are sitting right there on nvtop, not in a slide, not a claim, the meter reading while it works. so i keep landing on one question. the model is open, apache 2.0. the card is old and cheap. the whole thing runs offline. why is almost nobody actually running this. is it the hardware, the setup, or did nobody tell you a 27b fits on the card already in your closet.

Sudo su

23,453 次观看 • 1 个月前

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!

Alok

46,100 次观看 • 1 个月前

DeepSeek dropped a massive new AI tool and almost nobody noticed. It’s called DeepSeek OCR. Here’s why it’s wild. It can read documents with 97% accuracy. While compressing them up to 10× smaller. And the craziest part? It’s completely free and open source. The GitHub repo is already live and you can download it right now. This might quietly become one of the most useful AI tools of the year.

DeepSeek dropped a massive new AI tool and almost nobody noticed. It’s called DeepSeek OCR. Here’s why it’s wild. It can read documents with 97% accuracy. While compressing them up to 10× smaller. And the craziest part? It’s completely free and open source. The GitHub repo is already live and you can download it right now. This might quietly become one of the most useful AI tools of the year.

Julian Goldie SEO

15,310 次观看 • 6 个月前

$DEEPSEEK JUST DROPPED HARNESS, AND IT ALREADY BECAME ONE OF THE FASTEST-GROWING PROJECTS ON GITHUB: 130,000 STARS IN ONLY 3 DAYS. COMPLETELY FREE Core idea: almost anything can become a plugin ➜ models, sessions, skills, sandboxes, loops, even the interface. Put the pieces in, get a working agent out 1\ Architecture is built on Cordis, so you can swap components at any time 2\ Formula is simple: any model + any component = ready agent 3\ Community already shipped 6,000 ready skills for it You get a flexible environment where you control the building blocks instead of being locked into one rigid setup Installation is simple. Run these commands one by one: ``` git clone cd deepseek-harness pnpm install pnpm run build pnpm dsh web ```$

DEEPSEEK JUST DROPPED HARNESS, AND IT ALREADY BECAME ONE OF THE FASTEST-GROWING PROJECTS ON GITHUB: 130,000 STARS IN ONLY 3 DAYS. COMPLETELY FREE Core idea: almost anything can become a plugin ➜ models, sessions, skills, sandboxes, loops, even the interface. Put the pieces in, get a working agent out 1\ Architecture is built on Cordis, so you can swap components at any time 2\ Formula is simple: any model + any component = ready agent 3\ Community already shipped 6,000 ready skills for it You get a flexible environment where you control the building blocks instead of being locked into one rigid setup Installation is simple. Run these commands one by one: ``` git clone cd deepseek-harness pnpm install pnpm run build pnpm dsh web ```

beamnxw ./

12,264 次观看 • 27 天前

What the f*ck is this GitHub repo, 33k+ stars in less than a month. It's called OpenHuman, and the newest feature is why it's exploding, Super Context. > The moment you open a new chat, it runs a context pass first > Gathers what's relevant about you, your screen, your work > Then answers turn one like it's turn ten, already grounded It lives at the OS level, speaks in your voice across your whole system, connects to 118 apps you already use. Runs locally, never leaves your machine, and keeps learning you the longer you use it. It was #31 trending on GitHub for ten days straight and now it's the #1 repo on GitHub. > Repo:

What the f*ck is this GitHub repo, 33k+ stars in less than a month. It's called OpenHuman, and the newest feature is why it's exploding, Super Context. > The moment you open a new chat, it runs a context pass first > Gathers what's relevant about you, your screen, your work > Then answers turn one like it's turn ten, already grounded It lives at the OS level, speaks in your voice across your whole system, connects to 118 apps you already use. Runs locally, never leaves your machine, and keeps learning you the longer you use it. It was #31 trending on GitHub for ten days straight and now it's the #1 repo on GitHub. > Repo:

Defileo🔮

206,429 次观看 • 2 个月前

We used to buy more VRAM. Now we’re finding ways around the VRAM limit. This little DDR5 SO-DIMM setup is paired with an old NVIDIA Tesla P40 in the video. The P40 is an old datacenter GPU with 24GB of VRAM. By today’s standards, it’s not fast. But combine that with a machine carrying 128GB of system memory, and suddenly you have a very different local AI box. The GPU handles what it’s good at. The system memory gives the model somewhere to go when VRAM runs out. You obviously pay for that with bandwidth and inference speed. But this is the part I find interesting: Local AI is making “how much memory do I have?” almost as important as “how powerful is my GPU?” We’ve spent years chasing faster GPUs. Maybe the next wave of local AI hardware will be about finding smarter ways to use all the memory we already have. More VRAM is still better. But it’s no longer the only way to run a bigger model.

We used to buy more VRAM. Now we’re finding ways around the VRAM limit. This little DDR5 SO-DIMM setup is paired with an old NVIDIA Tesla P40 in the video. The P40 is an old datacenter GPU with 24GB of VRAM. By today’s standards, it’s not fast. But combine that with a machine carrying 128GB of system memory, and suddenly you have a very different local AI box. The GPU handles what it’s good at. The system memory gives the model somewhere to go when VRAM runs out. You obviously pay for that with bandwidth and inference speed. But this is the part I find interesting: Local AI is making “how much memory do I have?” almost as important as “how powerful is my GPU?” We’ve spent years chasing faster GPUs. Maybe the next wave of local AI hardware will be about finding smarter ways to use all the memory we already have. More VRAM is still better. But it’s no longer the only way to run a bigger model.

KyzoroX

35,554 次观看 • 4 天前

Claude Code just got a lot easier to use 🙌 Just updated You can now install agents, commands, MCPs, and templates with a single command. Click on any card and it gives you the exact line to run. Everything is pulled from GitHub, the site runs on GitHub Pages, and downloads are logged with GitHub Actions.

Claude Code just got a lot easier to use 🙌 Just updated You can now install agents, commands, MCPs, and templates with a single command. Click on any card and it gives you the exact line to run. Everything is pulled from GitHub, the site runs on GitHub Pages, and downloads are logged with GitHub Actions.

Daniel San

135,292 次观看 • 1 年前

HRM-Text 101 is here. This tutorial takes you from zero to one: from setup to fine-tuning to evaluation. Download the base checkpoint. Fine-tune it on a real task. Evaluate the results. End to end, on a single GPU. Watch the tutorial and start building with HRM-Text.

HRM-Text 101 is here. This tutorial takes you from zero to one: from setup to fine-tuning to evaluation. Download the base checkpoint. Fine-tune it on a real task. Evaluate the results. End to end, on a single GPU. Watch the tutorial and start building with HRM-Text.

Sapient Intelligence

188,565 次观看 • 3 个月前

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

341,499 次观看 • 22 天前

MICROSOFT JUST OPEN SOURCED THE ENTIRE AGENT RUNTIME, SO YOU NEVER HAVE TO BUILD ONE AGAIN a raw model only writes text. to make it a real agent it needs a loop, tools, memory, planning, approvals, and a way to know when it is done. Microsoft's Agent Harness is that whole layer, and it ships with everything on by default. you bring the model and the tools, it wires up the rest: > an automatic tool calling loop with iteration limits > history saved after every call, so a crash resumes instead of starting over > context compaction that summarizes when the window fills > a persistent todo list plus plan and execute modes > file memory that survives across turns > tool approval with standing rules and auto approval > OpenTelemetry tracing and web search built in > Agent Skills that load domain expertise on demand the mental model is simple: the model is the brain, the harness is the body around it. and it works with whatever you already run: > Azure OpenAI, OpenAI, Anthropic, and GitHub Copilot, and it hosts on Microsoft Foundry you spin one up in a single line: agent = create_harness_agent(client, name, instructions) open source, Python, .NET, and Go, stable since mid 2026. bookmark this. Microsoft already built the hard part of every AI agent, you just plug in the model.

MICROSOFT JUST OPEN SOURCED THE ENTIRE AGENT RUNTIME, SO YOU NEVER HAVE TO BUILD ONE AGAIN a raw model only writes text. to make it a real agent it needs a loop, tools, memory, planning, approvals, and a way to know when it is done. Microsoft's Agent Harness is that whole layer, and it ships with everything on by default. you bring the model and the tools, it wires up the rest: > an automatic tool calling loop with iteration limits > history saved after every call, so a crash resumes instead of starting over > context compaction that summarizes when the window fills > a persistent todo list plus plan and execute modes > file memory that survives across turns > tool approval with standing rules and auto approval > OpenTelemetry tracing and web search built in > Agent Skills that load domain expertise on demand the mental model is simple: the model is the brain, the harness is the body around it. and it works with whatever you already run: > Azure OpenAI, OpenAI, Anthropic, and GitHub Copilot, and it hosts on Microsoft Foundry you spin one up in a single line: agent = create_harness_agent(client, name, instructions) open source, Python, .NET, and Go, stable since mid 2026. bookmark this. Microsoft already built the hard part of every AI agent, you just plug in the model.

NO1ennn

68,679 次观看 • 26 天前

What if you could train AI agents on a laptop as easily as on a GPU cluster? Researchers from UIUC's U Lab, led by Prof. Jiaxuan You, just open-sourced OpenTinker. It's a new "Reinforcement-Learning-as-a-Service" (RLaaS) system that decouples the complex training pipeline into simple, distributed services with friendly APIs. The result? It breaks down the major engineering barriers to RL, outperforming traditional frameworks in accessibility and ease of deployment, finally making agent training viable for more developers and teams. Project: Code: U Lab: Our report: 📬 #PapersAccepted by Jiqizhixin

What if you could train AI agents on a laptop as easily as on a GPU cluster? Researchers from UIUC's U Lab, led by Prof. Jiaxuan You, just open-sourced OpenTinker. It's a new "Reinforcement-Learning-as-a-Service" (RLaaS) system that decouples the complex training pipeline into simple, distributed services with friendly APIs. The result? It breaks down the major engineering barriers to RL, outperforming traditional frameworks in accessibility and ease of deployment, finally making agent training viable for more developers and teams. Project: Code: U Lab: Our report: 📬 #PapersAccepted by Jiqizhixin

机器之心 JIQIZHIXIN

15,893 次观看 • 8 个月前

look anon, those of you who kept saying local AI is not there yet, who said open source can't compete, who said you need cloud APIs to get anything serious done, look at this gameplay for one minute. every pixel on this screen was written by one model, in one shot, on a single rtx 3090 with 24gb of vram. the model is qwen 3.6 27b dense q4. the harness is hermes agent. the hardware is a single consumer card you can buy used for 900 dollars. the prompt is open source on github. every claim verifiable, on your own desk. if your local AI take is from 2024, update it. the consumer tier is shipping work that was supposed to need 8 gpus and an api key. open source moved the floor while the rest of the field was busy explaining why it cannot. 24gb tier owners are eating ramen with half boiled egg and double chocolate.

look anon, those of you who kept saying local AI is not there yet, who said open source can't compete, who said you need cloud APIs to get anything serious done, look at this gameplay for one minute. every pixel on this screen was written by one model, in one shot, on a single rtx 3090 with 24gb of vram. the model is qwen 3.6 27b dense q4. the harness is hermes agent. the hardware is a single consumer card you can buy used for 900 dollars. the prompt is open source on github. every claim verifiable, on your own desk. if your local AI take is from 2024, update it. the consumer tier is shipping work that was supposed to need 8 gpus and an api key. open source moved the floor while the rest of the field was busy explaining why it cannot. 24gb tier owners are eating ramen with half boiled egg and double chocolate.

Sudo su

29,584 次观看 • 4 个月前