How can we use small LLMs to shift more AI workloads onto our laptops and phones? In our paper and open-source code, we pair on-device LLMs (ollama) with frontier LLMs in the cloud (OpenAI, Together), to solve token-intensive workloads on your 💻 at 17.5% of the cloud cost while... maintaining 97.9% of the accuracy. See Gru and the Minions in action below, 🔉on please (h/t )!show more

With co-first authors @Avanika15, @EyubogluSabri at @hazyresearch. Paper: Blog: Ollama Blog: GitHub: NotebookLM Podcast:

Frontier language models (LMs) are powerful but pricey. Small LMs are fast and cheap but still catching up. Why not get the best of both? We investigate the tradeoffs arising when a frontier LM manages small on-device LMs with access to data.

We explore two communication protocols: “Minion” and “Minions” (plural). In Minion, the cloud model simply chats with a single local model until the answer is found, achieving 87% of the cloud-only accuracy at 3.3% of the cloud costs.

We show that Minion drops 13% of cloud-only accuracy due to two limitations of current small LMs: 1⃣ They struggle with long contexts. 2⃣They do not reliably follow multi-step instructions. Moreover, on local workstations with NVIDIA GPUs, Minion achieves low utilization because the small LM generates only one message at a time (no batching).

In Minions (plural) we handle these issues by chunking the context and having the small LM perform single-step tasks on each chunk, in parallel. Because the cloud LM does not see the context, we have it generate code for task decomposition, context chunking, and results aggregation. Minion achieves 97.9% of cloud-only accuracy with 17.5% of the cost. It also enables parallelism on GPU workstations. The frontier model is free to choose the tasks and chunking granularity without any imposed heuristics.

The Minions paradigm is tailored for scaling inference-time compute: we can increase the granularity of the chunking, add tasks, or perform many generations, to improve accuracy in the cost of extra compute, locally and remotely.

We envision a future where local agents economically communicate with the cloud to continuously polish your writing or refactor your code. We take a step in this direction, and believe that the more people look into this problem, the better and cheaper AI will be for all of us. Let’s build and understand it together 🚀🌕

Thanks to @scott_linderman, @james_y_zou, and @hazyresearch for co-advising, and @avnermay at @togethercompute for collaborating! Huge thanks to the @ollama crew for their support in this release! Thank you @StanfordAILab and @StanfordCRFM.

AppKit is the full-stack toolkit to build onchain app UX 🪄 ✅ Social, Email, and Wallet Login ✅ Embedded Wallets ✅ Crypto Swaps ✅ On-ramp Integrate with just 20 lines of code across 10+ languages for all EVM chains and Solana. Onboard millions of users for free today.

@ollama @OpenAI @Together This is incredible work! Congratulations!! And thank you for acknowledging our PAPILLON work!! It is truly amazing what adopting the local- and frontier-LLM collaboration paradigm could achieve!!

@ollama @OpenAI @Together This is so so cool Dan!! Also love the cute branding and everything! 😀