正在加载视频...

视频加载失败

Introducing 0GM-1.0-35B-A3B. Our first proprietary AI model. Mixture of Experts (MoE), 35B parameters, 3B active per token. Trained on our own decentralized GPU network. Open source under Apache 2.0.

48,838 次观看 • 5 个月前 •via X (Twitter)

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Someone is running Qwen3.8-35B-A3B on a single RTX 3060 with just 12GB of VRAM and 16GB of system RAM. And the reported numbers are surprisingly good. ~50 tok/s decode ~550 tok/s prefill 262K context window 12GB VRAM 16GB system RAM No multi-GPU setup. Just a consumer graphics card and a carefully configured local inference engine. The model is particularly interesting because it’s Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. That makes it worth investigating beyond raw inference speed. The developer currently uses Ornith-1.5 as the engine behind their Hermes and Pi agents. Now the question is whether this new model can deliver enough quality and reliability to replace it. The next step is testing it as an agent, not just a chatbot. The planned comparison includes four models: • Qwen3.8-35B-A3B • Ornith-1.5-35B • Qwen3.8-27B • Base Qwen3.6-35B-A3B An agentic coding benchmark should help reveal whether the distilled model can follow instructions, modify code, solve problems, and complete multi-step tasks effectively. That distinction matters because high token throughput doesn’t automatically translate into a better coding agent. A model can generate code quickly and still struggle with planning, debugging, tool calls, or completing a task without intervention. The hardware configuration is also worth examining. The developer is using llama-server with the Q4_K_M GGUF quantization, Flash Attention enabled, and 26 MoE layers assigned to CPU processing. The configuration also enables a 262,144-token context window, Q8 KV-cache quantization, and reasoning mode. This is how the setup is configured: llama-server \ -m Qwen3.8-35B-A3B-Q4_K_M.gguf \ -ngl 99 \ --n-cpu-moe 26 \ -c 262144 \ -fa on \ --jinja \ -np 1 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --reasoning on One important detail: the model’s 35B parameter count doesn’t mean all 35 billion parameters are activated for every token. The A3B designation indicates roughly 3 billion active parameters per token. That sparse MoE architecture helps explain how a model of this size can deliver high generation speeds on modest hardware. CPU-based expert processing also helps work around the GPU’s limited memory, although the exact trade-offs depend on the workload. The 262K context claim is equally interesting. Supporting a context window that large doesn’t mean every request at that length will maintain the same speed. Prompt length, KV-cache memory, and CPU/GPU placement can all affect performance. The developer’s next test is particularly useful: graphing decode speed against context length. That should help show how performance changes as the prompt grows, rather than presenting one speed figure without the conditions behind it. If the coding benchmark also holds up, this could become a serious option for people who want capable local agents without investing in expensive GPUs. For now, the reported inference numbers are promising. The real test is whether Qwen3.8-35B-A3B can match or outperform Ornith-1.5-35B on actual agentic coding tasks. Fast local inference gets your attention. Reliable autonomous task completion is what earns a model a place in the stack.

FHILY👑

19,254 次观看 • 2 天前

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

344,809 次观看 • 1 个月前