Загрузка видео...

Не удалось загрузить видео

На главную

local LLMs on Mac are getting good. but the workflow around models is still messy. you have models in: •⁠ ⁠LM Studio •⁠ ⁠Ollama •⁠ ⁠Hugging Face cache •⁠ ⁠random folders •⁠ ⁠manual downloads and every runtime expects something slightly different. we built ModelHub to clean up this layer....

sabeshbharathi's profile picture

sabesh 📟

2,269 subscribers

31,497 просмотров • 4 месяцев назад •via X (Twitter)

Новости и политикаОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

You need to know 4 things to run local AI on your own machine: 1) The model is the brain file. Gemma, Llama, Mistral, Qwen, Phi. 2) The warehouse is where you find the model. Hugging Face. 3) The software is what runs the model. LM Studio for the first try, Ollama when you build apps. 4) The workflow is the product you make from the other three. The size of the model decides the hardware: - 2B to 4B for phones and small tasks - 12B for the middle ground - 26B to 31B for a workstation Quantization compresses the model until it fits on a normal machine: - Q4 is easier to run - Q8 keeps more quality, but it needs more memory Q4 costs you a little quality but it saves you a $5,000 workstation. Start on the spare laptop from 2021. Open Hugging Face and read one model card slowly.
7:20

You need to know 4 things to run local AI on your own machine: 1) The model is the brain file. Gemma, Llama, Mistral, Qwen, Phi. 2) The warehouse is where you find the model. Hugging Face. 3) The software is what runs the model. LM Studio for the first try, Ollama when you build apps. 4) The workflow is the product you make from the other three. The size of the model decides the hardware: - 2B to 4B for phones and small tasks - 12B for the middle ground - 26B to 31B for a workstation Quantization compresses the model until it fits on a normal machine: - Q4 is easier to run - Q8 keeps more quality, but it needs more memory Q4 costs you a little quality but it saves you a $5,000 workstation. Start on the spare laptop from 2021. Open Hugging Face and read one model card slowly.

The Startup Ideas Podcast (SIP) 🧃

10,547 просмотров • 29 дней назад

Hermes just got a local model nobody is talking about. And it runs your entire workflow for free. You do not need to pay for smart agents anymore. A developer just dropped a free brain called Karnis on Hugging Face. It is built perfectly for Hermes. It runs right on your computer using LM Studio. It watches real agents work and copies their exact steps. It edits files and runs commands for you. Just download LM Studio, search for Karnis, and connect it to Hermes. Your private agent is ready to work.
10:00

Hermes just got a local model nobody is talking about. And it runs your entire workflow for free. You do not need to pay for smart agents anymore. A developer just dropped a free brain called Karnis on Hugging Face. It is built perfectly for Hermes. It runs right on your computer using LM Studio. It watches real agents work and copies their exact steps. It edits files and runs commands for you. Just download LM Studio, search for Karnis, and connect it to Hermes. Your private agent is ready to work.

Julian Goldie SEO

22,365 просмотров • 6 месяцев назад

Create your own local and free AI agent using an open source model You can combine: - IBM's Granite 3.3 8B AI model - LM Studio to run it on a laptop - Smolagents to build your agent Small AI models are now powerful enough to run an autonomous agent. Thanks to IBM France for sponsoring this content!
8:44

Create your own local and free AI agent using an open source model You can combine: - IBM's Granite 3.3 8B AI model - LM Studio to run it on a laptop - Smolagents to build your agent Small AI models are now powerful enough to run an autonomous agent. Thanks to IBM France for sponsoring this content!

Paul Couvert

18,890 просмотров • 1 год назад

i tested a fully local ai coding setup: gemma4 from Google DeepMind running through LM Studio, paired with pi which is a minimal open source coding harness this is part 1 of a two part series. here we cover the why and the complete hands-on setup 0:00 - Running a coding agent fully local 0:47 - Why Pi as the coding agent harness 1:39 - Why Gemma 4 and which variant to pick 2:44 - Architecture: Pi, LM Studio and Gemma 4 3:45 - LM Studio setup and model download 5:01 - Starting the local server 5:19 - API sanity check with curl 6:08 - Installing the Pi coding agent 6:20 - Configuring models.json for local models 8:20 - First run: selecting the model in Pi 8:58 - Fixing the context length error 10:16 - Testing the working setup 10:50 - What is coming in Part 2
11:32

i tested a fully local ai coding setup: gemma4 from Google DeepMind running through LM Studio, paired with pi which is a minimal open source coding harness this is part 1 of a two part series. here we cover the why and the complete hands-on setup 0:00 - Running a coding agent fully local 0:47 - Why Pi as the coding agent harness 1:39 - Why Gemma 4 and which variant to pick 2:44 - Architecture: Pi, LM Studio and Gemma 4 3:45 - LM Studio setup and model download 5:01 - Starting the local server 5:19 - API sanity check with curl 6:08 - Installing the Pi coding agent 6:20 - Configuring models.json for local models 8:20 - First run: selecting the model in Pi 8:58 - Fixing the context length error 10:16 - Testing the working setup 10:50 - What is coming in Part 2

Nidhi Singh

16,289 просмотров • 3 месяцев назад

Which models can your machine actually run? Magnitude is a 100% free, open source desktop app that: - profiles your hardware and runs sample calculations - predicts tok/s for every model before you download - recommends the best models, from fast to smart Pick your models and it handles the rest: - downloads and tunes the models for your hardware - connects agents like pi, opencode, hermes in one click - runs models on demand as your agent works Works on whatever hardware you already own: MacBook (Intel or Apple Silicon), Mac Mini, Mac Studio, DGX Spark, Strix Halo, any NVIDIA/AMD GPU, or just a CPU Download it on macOS, Windows, or Linux: Open source:
0:32

Which models can your machine actually run? Magnitude is a 100% free, open source desktop app that: - profiles your hardware and runs sample calculations - predicts tok/s for every model before you download - recommends the best models, from fast to smart Pick your models and it handles the rest: - downloads and tunes the models for your hardware - connects agents like pi, opencode, hermes in one click - runs models on demand as your agent works Works on whatever hardware you already own: MacBook (Intel or Apple Silicon), Mac Mini, Mac Studio, DGX Spark, Strix Halo, any NVIDIA/AMD GPU, or just a CPU Download it on macOS, Windows, or Linux: Open source:

Tom Greenwald

48,528 просмотров • 19 дней назад

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀
0:21

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

🚨 AI News | TestingCatalog

41,281 просмотров • 1 месяц назад

STOP DOWNLOADING LLMS YOUR MACHINE WAS NEVER GOING TO RUN LLMFIT is a new open-source tool that scans your hardware first, then tells you exactly which models will actually run well on your setup. It inspects: → RAM → CPU → GPU(s) → VRAM / unified memory Then it scores every model across four dimensions: → quality, based on parameter count and quantization → speed, estimating tokens per second for your exact backend → fit, matching memory usage to your hardware → context window support for your use case Each model then gets a clear label: Perfect, Good, Marginal, or Too Tight. It also picks the best quantization automatically, stepping down until something fits instead of leaving you to guess GGUF sizes by hand. Coverage is broad too! 100's of models from Meta, Mistral, Qwen, DeepSeek, and more. It works with Ollama, llama.cpp, MLX, and LM Studio out of the box. And that’s not even it! There’s also a lightweight TUI, a web dashboard, and REST API endpoints if you want to plug it into orchestrators or deployment pipelines. This should be step zero before every local model download :) 100% free and open-source. macOS, Linux, and Windows. Repo in 🧵↓
0:46

STOP DOWNLOADING LLMS YOUR MACHINE WAS NEVER GOING TO RUN LLMFIT is a new open-source tool that scans your hardware first, then tells you exactly which models will actually run well on your setup. It inspects: → RAM → CPU → GPU(s) → VRAM / unified memory Then it scores every model across four dimensions: → quality, based on parameter count and quantization → speed, estimating tokens per second for your exact backend → fit, matching memory usage to your hardware → context window support for your use case Each model then gets a clear label: Perfect, Good, Marginal, or Too Tight. It also picks the best quantization automatically, stepping down until something fits instead of leaving you to guess GGUF sizes by hand. Coverage is broad too! 100's of models from Meta, Mistral, Qwen, DeepSeek, and more. It works with Ollama, llama.cpp, MLX, and LM Studio out of the box. And that’s not even it! There’s also a lightweight TUI, a web dashboard, and REST API endpoints if you want to plug it into orchestrators or deployment pipelines. This should be step zero before every local model download :) 100% free and open-source. macOS, Linux, and Windows. Repo in 🧵↓

Charly Wargnier ♨️

17,975 просмотров • 5 дней назад

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.
1:52

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.

Alex Cheema

30,854 просмотров • 3 месяцев назад

THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running on 127.0.0.1:8080 plugged it into VS Code, connected it to OpenClaw, 53 tok/s on an M1 Max with 262K context zero wrappers, zero timeout errors, zero API fees bookmark & like this before your next timeout error hits​​​​​​​​​​​​​​​​ full breakdown of my raw llama.cpp setup ↓
21:03

THIS AMERICAN DEVELOPER SPENT WEEKS DEBUGGING TIMEOUT ERRORS IN OLLAMA. THEN HE LOOKED UNDER THE HOOD LM Studio is just llama.cpp Ollama is just llama.cpp so he cloned llama.cpp from source, pulled Qwen 3.6 35B off Hugging Face, set up asymmetric KV quantization and got a local server running on 127.0.0.1:8080 plugged it into VS Code, connected it to OpenClaw, 53 tok/s on an M1 Max with 262K context zero wrappers, zero timeout errors, zero API fees bookmark & like this before your next timeout error hits​​​​​​​​​​​​​​​​ full breakdown of my raw llama.cpp setup ↓

leopardracer

241,145 просмотров • 4 месяцев назад

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.
0:26

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Harper Carroll

30,392 просмотров • 2 месяцев назад

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.
0:39

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.

Poolside

31,242 просмотров • 2 месяцев назад

🐆 Rapid-MLX v0.12 is here. We’ve officially evolved from a simple chat app into a full-fledged, on-device AI studio for Apple Silicon! 🖥️✨ We didn't just push the MLX inference engine to its limits and expand support for a massive lineup of local open-source models—we are alpha-launching the highly anticipated Desktop Version. (A huge shoutout to the IoTeX community for grinding through the closed beta with us. Your feedback was incredible and helped shape this beast.) Here are the game-changing features you can run on your Mac right now, 100% free and 100% offline 👇 🚀 Blazing Fast Local LLMs Run anything from 4B up to Qwen3.5-122B completely offline. No guessing games—we recommend models matched perfectly to your Mac's actual RAM. Rich chat includes syntax highlighting, markdown tables, and honest tok/s metrics. 🎨 Local Image Generation A brand new Images tab to render directly on your machine. Pick a model (FLUX.2-klein, Z-Image-Turbo), prompt, and refine. Everything lands in a visual filmstrip. 👁️ Vision & Live Web Tools Attach an image and chat about it with local vision. Need real-time data? Our built-in web tools (weather, search, page-fetch) run mid-answer with strict, transparent privacy controls. 🤖 Plug-and-Play Coding Agents Wire up Claude Code, Codex, Cline, or Continue in seconds. One copy-paste from the Launch tab spins up a local OpenAI/Anthropic-compatible endpoint. 🔒 Private by Design Everything runs on-device. Signed, notarized, and entirely local. Your data stays yours. Turn your Mac into an AI powerhouse today. ⚡️
0:37

🐆 Rapid-MLX v0.12 is here. We’ve officially evolved from a simple chat app into a full-fledged, on-device AI studio for Apple Silicon! 🖥️✨ We didn't just push the MLX inference engine to its limits and expand support for a massive lineup of local open-source models—we are alpha-launching the highly anticipated Desktop Version. (A huge shoutout to the IoTeX community for grinding through the closed beta with us. Your feedback was incredible and helped shape this beast.) Here are the game-changing features you can run on your Mac right now, 100% free and 100% offline 👇 🚀 Blazing Fast Local LLMs Run anything from 4B up to Qwen3.5-122B completely offline. No guessing games—we recommend models matched perfectly to your Mac's actual RAM. Rich chat includes syntax highlighting, markdown tables, and honest tok/s metrics. 🎨 Local Image Generation A brand new Images tab to render directly on your machine. Pick a model (FLUX.2-klein, Z-Image-Turbo), prompt, and refine. Everything lands in a visual filmstrip. 👁️ Vision & Live Web Tools Attach an image and chat about it with local vision. Need real-time data? Our built-in web tools (weather, search, page-fetch) run mid-answer with strict, transparent privacy controls. 🤖 Plug-and-Play Coding Agents Wire up Claude Code, Codex, Cline, or Continue in seconds. One copy-paste from the Launch tab spins up a local OpenAI/Anthropic-compatible endpoint. 🔒 Private by Design Everything runs on-device. Signed, notarized, and entirely local. Your data stays yours. Turn your Mac into an AI powerhouse today. ⚡️

raullen

34,194 просмотров • 1 месяц назад

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource
0:24

The #1 problem with local AI is now solved. There’s a free tool called that checks your hardware and tells you which models will actually run well before you download anything. So instead of guessing and hitting out-of-memory errors…it grades every model against your machine. What it does (right in your browser, no install): → detects your setup (RAM / CPU / GPU / VRAM) → scores each model for fit, speed, and context length → grades every quantization level (Q4_K_M, Q6_K, Q8_0, etc.) → labels what runs great vs okay vs too heavy It covers most of the open-weight stack, Llama, Qwen, Gemma, Mistral, DeepSeek, Phi and more, pulling requirements from llama.cpp, Ollama, and LM Studio. And it’s fully opensource

Alvaro Cintas

195,200 просмотров • 1 месяц назад

🦙 ollama is used by 9 million developers and 85% of the Fortune 500, giving co-founder and CEO Jeffrey Morgan (Jeffrey Morgan) a unique view into which AI models people are actually using and how that’s changing. Right now, the biggest shift he sees is toward open models, driven by coding agents, falling costs, and capabilities that are rapidly catching up to the frontier labs. On Ollama Cloud, that shift has driven a 150x increase in token usage since the start of the year. In this episode of Lightcone Podcast, Jeff joins Garry Tan, Jared Friedman, Diana, and Harj Taggar to talk about the future of open models and the story behind Ollama, from two years of searching for the right idea to building one of the most widely used AI developer tools in the world. 00:43 — The Shift to Open Models 03:03 — How AI Agents Are Driving Token Usage 05:31 — Are Open Models Catching Up? 08:26 — What Happens When a New Model Launches 11:31 — Ollama as an Operating System for AI 14:05 — The New Opportunities Above the Model Layer 18:19 — Why 80–90% of Enterprise Tokens Could Be Open 20:57 — The Future Is Local and Cloud 26:40 — Why AI Is Coming Back to Your Computer 28:56 — The Coming Era of Unlimited Tokens 32:30 — Do We Still Need a “God Model”? 33:41 — Open Models and Geopolitics 36:14 — The Origins of Ollama 40:36 — Two Years Lost in the Wilderness 42:39 — The Pivot That Changed Everything 47:02 — How Ollama Found a Business Model 49:43 — Why Second-Time Founders Did YC
57:15

🦙 ollama is used by 9 million developers and 85% of the Fortune 500, giving co-founder and CEO Jeffrey Morgan (Jeffrey Morgan) a unique view into which AI models people are actually using and how that’s changing. Right now, the biggest shift he sees is toward open models, driven by coding agents, falling costs, and capabilities that are rapidly catching up to the frontier labs. On Ollama Cloud, that shift has driven a 150x increase in token usage since the start of the year. In this episode of Lightcone Podcast, Jeff joins Garry Tan, Jared Friedman, Diana, and Harj Taggar to talk about the future of open models and the story behind Ollama, from two years of searching for the right idea to building one of the most widely used AI developer tools in the world. 00:43 — The Shift to Open Models 03:03 — How AI Agents Are Driving Token Usage 05:31 — Are Open Models Catching Up? 08:26 — What Happens When a New Model Launches 11:31 — Ollama as an Operating System for AI 14:05 — The New Opportunities Above the Model Layer 18:19 — Why 80–90% of Enterprise Tokens Could Be Open 20:57 — The Future Is Local and Cloud 26:40 — Why AI Is Coming Back to Your Computer 28:56 — The Coming Era of Unlimited Tokens 32:30 — Do We Still Need a “God Model”? 33:41 — Open Models and Geopolitics 36:14 — The Origins of Ollama 40:36 — Two Years Lost in the Wilderness 42:39 — The Pivot That Changed Everything 47:02 — How Ollama Found a Business Model 49:43 — Why Second-Time Founders Did YC

Y Combinator

313,654 просмотров • 1 месяц назад

Local AI 101: open models, Hugging Face, and businesses to build (38 min masterclass) I still think cloud AI is the default for most things, and honestly it should be, the frontier models are the strongest and easiest to use. But something shifted in the last 4-5 months. You can now run genuinely good open models directly on your own laptop, or even your phone. And once you actually try it, it changes how you think about what AI is even for. LOCAL AI, CLEARLY EXPLAINED: 1. The model is the brain doing the thinking. Gemma, Llama, Mistral, and Qwen are the main families, and each is better at different things, some at reasoning, some at coding, some small enough to run on a phone. 2. Hugging Face is the warehouse where you find them. You go there to see what each model is good at, check the license, and grab the compressed versions that run on a normal computer. 3. The software is what runs the model on your machine. Start with LM Studio if you're not technical, it feels like a normal app where you search, download, and start chatting. Ollama is the one you reach for when you want to plug a model into your own apps. 4. The workflow is the actual product you build on top of it all. That's what I'm ideating around for some businesses to create. I think local AI just made a specific kind of business way easier to start. Find an industry that: 1. Sits on sensitive data they'd never paste into ChatGPT 2. Does the same review over and over 3. Runs on software from 2003 Then build a local AI tool that does that review on their own machine, so the data never leaves the building! Take home health agencies. Nurses write visit notes all day, and if a note is missing a detail, the billing gets denied or the audit flags it. Here's how I'd start: 1. Find 5 small agencies. Offer to review a batch of their notes for them. 2. Run the notes through Gemma locally (free, private, no cloud). Read every output yourself. 3. Write down the 20 issues that keep showing up: missing vitals, vague med changes, notes that don't support the billed level. 4. That list of 20 is your checklist. The checklist is the product. 5. Turn it into a local desktop app that flags those 20 things before a note gets submitted. You just went from a service anyone could offer to a product nobody else has, and you learned exactly what to build by doing the work by hand first. Same recipe works for restoration contractors (draft the damage report on-site before the tech leaves) and wealth advisors (catch the compliance landmine in a client email before it sends). Basically the framework is sensitive data, repeated review, ancient software. I think there are tons of businesses like this! Almost none of it clicked for me until I actually started using local AI. So if you take one thing from this, go run a model on your own machine once. Also a fun thing to try with your friends. Feel free to send this to a friend. The episode is live for free on The Startup Ideas Podcast (SIP) 🧃 (thanks to Google for sponsoring today's episode and supporting local AI) I feel like local AI one of those things you need to try for it to really click. Run one model on your own machine and you'll see what I mean! I go way deeper in the full 38 minute masterclass, the models, the setup, and the businesses to build. Link below. LINK TO WATCH: OR WATCH BELOW ON X What do you think of local AI?
38:45

Local AI 101: open models, Hugging Face, and businesses to build (38 min masterclass) I still think cloud AI is the default for most things, and honestly it should be, the frontier models are the strongest and easiest to use. But something shifted in the last 4-5 months. You can now run genuinely good open models directly on your own laptop, or even your phone. And once you actually try it, it changes how you think about what AI is even for. LOCAL AI, CLEARLY EXPLAINED: 1. The model is the brain doing the thinking. Gemma, Llama, Mistral, and Qwen are the main families, and each is better at different things, some at reasoning, some at coding, some small enough to run on a phone. 2. Hugging Face is the warehouse where you find them. You go there to see what each model is good at, check the license, and grab the compressed versions that run on a normal computer. 3. The software is what runs the model on your machine. Start with LM Studio if you're not technical, it feels like a normal app where you search, download, and start chatting. Ollama is the one you reach for when you want to plug a model into your own apps. 4. The workflow is the actual product you build on top of it all. That's what I'm ideating around for some businesses to create. I think local AI just made a specific kind of business way easier to start. Find an industry that: 1. Sits on sensitive data they'd never paste into ChatGPT 2. Does the same review over and over 3. Runs on software from 2003 Then build a local AI tool that does that review on their own machine, so the data never leaves the building! Take home health agencies. Nurses write visit notes all day, and if a note is missing a detail, the billing gets denied or the audit flags it. Here's how I'd start: 1. Find 5 small agencies. Offer to review a batch of their notes for them. 2. Run the notes through Gemma locally (free, private, no cloud). Read every output yourself. 3. Write down the 20 issues that keep showing up: missing vitals, vague med changes, notes that don't support the billed level. 4. That list of 20 is your checklist. The checklist is the product. 5. Turn it into a local desktop app that flags those 20 things before a note gets submitted. You just went from a service anyone could offer to a product nobody else has, and you learned exactly what to build by doing the work by hand first. Same recipe works for restoration contractors (draft the damage report on-site before the tech leaves) and wealth advisors (catch the compliance landmine in a client email before it sends). Basically the framework is sensitive data, repeated review, ancient software. I think there are tons of businesses like this! Almost none of it clicked for me until I actually started using local AI. So if you take one thing from this, go run a model on your own machine once. Also a fun thing to try with your friends. Feel free to send this to a friend. The episode is live for free on The Startup Ideas Podcast (SIP) 🧃 (thanks to Google for sponsoring today's episode and supporting local AI) I feel like local AI one of those things you need to try for it to really click. Run one model on your own machine and you'll see what I mean! I go way deeper in the full 38 minute masterclass, the models, the setup, and the businesses to build. Link below. LINK TO WATCH: OR WATCH BELOW ON X What do you think of local AI?

GREG ISENBERG

33,547 просмотров • 29 дней назад

Today, we’re launching Gemma 4, our most intelligent open models to date. Built with the same breakthrough technology as Gemini 3, Gemma 4 brings advanced reasoning to your personal hardware and devices. Here’s what Gemma 4 unlocks for developers: — Intelligence-per-parameter: Our 31B (Dense) and 26B (MoE) models deliver state-of-the-art performance for their size, outcompeting models 20x their size on Arena.ai — Commercial flexibility: Released under a permissive Apache 2.0 license for complete developer flexibility and digital sovereignty — Agentic workflows: Native support for function-calling and structured JSON output allows you to build reliable, autonomous agents — Multimodal edge AI: The E2B and E4B models bring native vision, audio, and low latency to mobile and IoT devices — Long-context reasoning: Up to 256K context windows allow you to process entire repositories or large documents in a single prompt Whether you're building global applications in 140+ languages or local-first AI code assistants, Gemma 4 is built to be your foundation. Explore in Google AI Studio or download the weights on Hugging Face, Kaggle, and ollama.
2:51

Today, we’re launching Gemma 4, our most intelligent open models to date. Built with the same breakthrough technology as Gemini 3, Gemma 4 brings advanced reasoning to your personal hardware and devices. Here’s what Gemma 4 unlocks for developers: — Intelligence-per-parameter: Our 31B (Dense) and 26B (MoE) models deliver state-of-the-art performance for their size, outcompeting models 20x their size on Arena.ai — Commercial flexibility: Released under a permissive Apache 2.0 license for complete developer flexibility and digital sovereignty — Agentic workflows: Native support for function-calling and structured JSON output allows you to build reliable, autonomous agents — Multimodal edge AI: The E2B and E4B models bring native vision, audio, and low latency to mobile and IoT devices — Long-context reasoning: Up to 256K context windows allow you to process entire repositories or large documents in a single prompt Whether you're building global applications in 140+ languages or local-first AI code assistants, Gemma 4 is built to be your foundation. Explore in Google AI Studio or download the weights on Hugging Face, Kaggle, and ollama.

Google AI

1,670,991 просмотров • 6 месяцев назад

✨ Personal AI is the next computing platform. AI is shifting from something you access to something you build with, locally, at the edge, and across systems. We’re unlocking new possibilities for developers: • AMD Ryzen AI Halo, a local-first developer system, preorder starting in June, develop AI without limits on your desk • Gorgon Halo with up to 192GB unified memory, supporting 300B+ parameter models locally, run massive models locally We’re excited to partner with clem 🤗🤗, Co-founder and CEO of Hugging Face, to advance open-source AI for Ryzen AI. Our focus is seamless AI, from model to deployment. Cloud, edge, device. One continuum. Multi-agent systems, local inference at scale, open models as infrastructure. This is the next computing era 🚀
4:49

✨ Personal AI is the next computing platform. AI is shifting from something you access to something you build with, locally, at the edge, and across systems. We’re unlocking new possibilities for developers: • AMD Ryzen AI Halo, a local-first developer system, preorder starting in June, develop AI without limits on your desk • Gorgon Halo with up to 192GB unified memory, supporting 300B+ parameter models locally, run massive models locally We’re excited to partner with clem 🤗🤗, Co-founder and CEO of Hugging Face, to advance open-source AI for Ryzen AI. Our focus is seamless AI, from model to deployment. Cloud, edge, device. One continuum. Multi-agent systems, local inference at scale, open models as infrastructure. This is the next computing era 🚀

Jack Huynh

50,015 просмотров • 4 месяцев назад

$15,000 for a Mac sounds insane. Until you realize what Apple is actually selling. 256GB of unified memory. Not a huge GPU farm. Not an NVIDIA rack. Not 4 different machines. Just one box that can keep multiple large LLMs in memory and run them locally. That changes the equation for local AI. NVIDIA wins when you need raw GPU compute. Apple gets interesting when the bottleneck is memory capacity. Because once your model doesn't fit into 24GB, 32GB or 48GB of VRAM, buying a faster GPU doesn't solve the problem. You need more memory. And that's where machines like the Mac Studio start looking less crazy. The interesting question isn't: “Is a $15K Mac faster than an NVIDIA GPU?” It isn't. The real question is: “How much are you willing to pay to make huge local models actually fit?” That’s where unified memory gets interesting.
0:44

$15,000 for a Mac sounds insane. Until you realize what Apple is actually selling. 256GB of unified memory. Not a huge GPU farm. Not an NVIDIA rack. Not 4 different machines. Just one box that can keep multiple large LLMs in memory and run them locally. That changes the equation for local AI. NVIDIA wins when you need raw GPU compute. Apple gets interesting when the bottleneck is memory capacity. Because once your model doesn't fit into 24GB, 32GB or 48GB of VRAM, buying a faster GPU doesn't solve the problem. You need more memory. And that's where machines like the Mac Studio start looking less crazy. The interesting question isn't: “Is a $15K Mac faster than an NVIDIA GPU?” It isn't. The real question is: “How much are you willing to pay to make huge local models actually fit?” That’s where unified memory gets interesting.

KyzoroX

19,536 просмотров • 1 месяц назад

MiniMax-M2.1 running fully local in AWQ-4Bit with full context window (170 GB VRAM w full context) - 1000~ to 16,000~ tps prefill - 100~ tps generation speeds - Opencode It’s doing real work, updating my blog with little steering or specificity. The problem with local LLMs is that they require too much steering, this means baby sitting which I don’t have the time to do MiniMax cracked the cost, intelligence, and speed challenge, I would say this is a top tier model. I run frontier models like Gemini and it just fails to call tools, in this year lol… ——————— I think glm-4.?-air is needed still. We need a viable model at each hardware entry point, a Mac M1 Ultra 192GB? is relatively cheap 5k to be able to run this model at 40 tps is a huge societal unlock. Smaller models can be good but size matters :p
3:01

MiniMax-M2.1 running fully local in AWQ-4Bit with full context window (170 GB VRAM w full context) - 1000~ to 16,000~ tps prefill - 100~ tps generation speeds - Opencode It’s doing real work, updating my blog with little steering or specificity. The problem with local LLMs is that they require too much steering, this means baby sitting which I don’t have the time to do MiniMax cracked the cost, intelligence, and speed challenge, I would say this is a top tier model. I run frontier models like Gemini and it just fails to call tools, in this year lol… ——————— I think glm-4.?-air is needed still. We need a viable model at each hardware entry point, a Mac M1 Ultra 192GB? is relatively cheap 5k to be able to run this model at 40 tps is a huge societal unlock. Smaller models can be good but size matters :p

0xSero

23,991 просмотров • 9 месяцев назад