正在加载视频...

视频加载失败

How can we use small LLMs to shift more AI workloads onto our laptops and phones? In our paper and open-source code, we pair on-device LLMs (ollama) with frontier LLMs in the cloud (OpenAI, Together), to solve token-intensive workloads on your 💻 at 17.5% of the cloud cost while...

193,452 次观看 • 1 年前 •via X (Twitter)

11 条评论

Dan Biderman 的头像
Dan Biderman1 年前

With co-first authors @Avanika15, @EyubogluSabri at @hazyresearch. Paper: Blog: Ollama Blog: GitHub: NotebookLM Podcast:

Dan Biderman 的头像
Dan Biderman1 年前

Frontier language models (LMs) are powerful but pricey. Small LMs are fast and cheap but still catching up. Why not get the best of both? We investigate the tradeoffs arising when a frontier LM manages small on-device LMs with access to data.

Dan Biderman 的头像
Dan Biderman1 年前

We explore two communication protocols: “Minion” and “Minions” (plural). In Minion, the cloud model simply chats with a single local model until the answer is found, achieving 87% of the cloud-only accuracy at 3.3% of the cloud costs.

Dan Biderman 的头像
Dan Biderman1 年前

We show that Minion drops 13% of cloud-only accuracy due to two limitations of current small LMs: 1⃣ They struggle with long contexts. 2⃣They do not reliably follow multi-step instructions. Moreover, on local workstations with NVIDIA GPUs, Minion achieves low utilization because the small LM generates only one message at a time (no batching).

Dan Biderman 的头像
Dan Biderman1 年前

In Minions (plural) we handle these issues by chunking the context and having the small LM perform single-step tasks on each chunk, in parallel. Because the cloud LM does not see the context, we have it generate code for task decomposition, context chunking, and results aggregation. Minion achieves 97.9% of cloud-only accuracy with 17.5% of the cost. It also enables parallelism on GPU workstations. The frontier model is free to choose the tasks and chunking granularity without any imposed heuristics.

Dan Biderman 的头像
Dan Biderman1 年前

The Minions paradigm is tailored for scaling inference-time compute: we can increase the granularity of the chunking, add tasks, or perform many generations, to improve accuracy in the cost of extra compute, locally and remotely.

Dan Biderman 的头像
Dan Biderman1 年前

We envision a future where local agents economically communicate with the cloud to continuously polish your writing or refactor your code. We take a step in this direction, and believe that the more people look into this problem, the better and cheaper AI will be for all of us. Let’s build and understand it together 🚀🌕

Dan Biderman 的头像
Dan Biderman1 年前

Thanks to @scott_linderman, @james_y_zou, and @hazyresearch for co-advising, and @avnermay at @togethercompute for collaborating! Huge thanks to the @ollama crew for their support in this release! Thank you @StanfordAILab and @StanfordCRFM.

Reown 的头像
Reown1 年前

AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.

Siyan Sylvia Li 🦋 的头像
Siyan Sylvia Li 🦋1 年前

@ollama @OpenAI @Together This is incredible work! Congratulations!! And thank you for acknowledging our PAPILLON work!! It is truly amazing what adopting the local- and frontier-LLM collaboration paradigm could achieve!!

Daniel Han 的头像
Daniel Han1 年前

@ollama @OpenAI @Together This is so so cool Dan!! Also love the cute branding and everything! 😀

相关视频