Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

You can now run SQLCoder with a GUI on Apple Silicon or any NVIDIA GPU-enabled device! On Apple Silicon, just run CMAKE_ARGS="-DLLAMA_METAL=on" pip install "sqlcoder[llama-cpp]" sqlcoder launch The Apple Silicon version is not super accurate, but works great for simple questions – and everything happens (fairly fast) on just... show more

Rishabh Srivastava

12,553 subscribers

10,405 views • 2 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 Comments

Cris Lenta2 years ago

This looks great! Have you tried also running it with .cpp models? Curious of the performance

Rishabh Srivastava2 years ago

Thank you! The demo above is for a llama.cpp model (with q5 km quantization). Haven't done a rigorous eval on those yet though – will do in a few hours!

Nerdynav2 years ago

hey great work, what's the license for this? Are code llama fine tunes allowed in their Meta's licensing?

Gautam Padiyar2 years ago

GPU with 20gb memory not required?

Rishabh Srivastava2 years ago

7B model, so you only need ~16GB or so of memory to make it run without beam search

Harsh Patel2 years ago

On Apple silicon how do I make it use the 7b-2 model? It is downloading the v1 I guess

Related Videos

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:

Awni Hannun

42,761 views • 1 year ago

Weekend plans? How about diving into LLMs on Apple devices? 🕷️💻 Just dropped my first tutorial: Run LLMs on your Mac with MLX! Perfect for AI beginners with Apple Silicon. More tech adventures coming soon! Please watch & subscribe:

Weekend plans? How about diving into LLMs on Apple devices? 🕷️💻 Just dropped my first tutorial: Run LLMs on your Mac with MLX! Perfect for AI beginners with Apple Silicon. More tech adventures coming soon! Please watch & subscribe:

qnguyen3

27,144 views • 2 years ago

Gemma 4 now works on-device using React Native! You can now run Gemma 4 fully offline in your cross-platform apps with local hardware acceleration: ⚡ Vulkan delegate on Android ⚡ MLX delegate on Apple Silicon See Gemma 4's vision and tool-use capabilities in action, instantly reading a flyer and scheduling a calendar event, 100% on-device.

Gemma 4 now works on-device using React Native! You can now run Gemma 4 fully offline in your cross-platform apps with local hardware acceleration: ⚡ Vulkan delegate on Android ⚡ MLX delegate on Apple Silicon See Gemma 4's vision and tool-use capabilities in action, instantly reading a flyer and scheduling a calendar event, 100% on-device.

Google Gemma

115,299 views • 19 days ago

Alibaba’s Qwen 3.5 is now running fully on-device on the iPhone 17 Pro. It outperforms models 4x its size, delivers strong visual understanding, and lets you switch reasoning on or off. This demo uses the 2B 6-bit version, optimized with MLX for Apple Silicon.

Alibaba’s Qwen 3.5 is now running fully on-device on the iPhone 17 Pro. It outperforms models 4x its size, delivers strong visual understanding, and lets you switch reasoning on or off. This demo uses the 2B 6-bit version, optimized with MLX for Apple Silicon.

Hugging Models

202,781 views • 4 months ago

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Prince Canuma

123,480 views • 1 year ago

Claude Code just got a lot easier to use 🙌 Just updated You can now install agents, commands, MCPs, and templates with a single command. Click on any card and it gives you the exact line to run. Everything is pulled from GitHub, the site runs on GitHub Pages, and downloads are logged with GitHub Actions.

Claude Code just got a lot easier to use 🙌 Just updated You can now install agents, commands, MCPs, and templates with a single command. Click on any card and it gives you the exact line to run. Everything is pulled from GitHub, the site runs on GitHub Pages, and downloads are logged with GitHub Actions.

Daniel San

135,254 views • 1 year ago

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

Julian Goldie SEO

11,234 views • 23 days ago

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

36,711 views • 8 days ago

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨

Zhijian Liu

49,358 views • 4 months ago

gpt4o in ChatGPT vs Apple Intelligence (on device Foundation model) prompt: "what are the top 3 coffee shops in lower manhattan?" not saying one is better than the other but fun to test. reply with your prompts and ill run a side by side for you.

gpt4o in ChatGPT vs Apple Intelligence (on device Foundation model) prompt: "what are the top 3 coffee shops in lower manhattan?" not saying one is better than the other but fun to test. reply with your prompts and ill run a side by side for you.

jpa

248,720 views • 1 year ago

Apple just upgraded the Formula 1 experience! 🏎️ Apple just made Formula One a lot easier to follow in the U.S. One place for the full race weekend. Practice, qualifying, sprint sessions, and every Grand Prix, live or on demand. The gifted package was cool, too. Apple TV 4K, pins, and a seriously clean F1 jacket. But the bigger story is what this means for fans. No more bouncing between apps or trying to figure out where the next session is airing. Race day just got way more convenient. Are you watching F1 on Apple TV this season? #formula1 #apple #appletv #gifted

Apple just upgraded the Formula 1 experience! 🏎️ Apple just made Formula One a lot easier to follow in the U.S. One place for the full race weekend. Practice, qualifying, sprint sessions, and every Grand Prix, live or on demand. The gifted package was cool, too. Apple TV 4K, pins, and a seriously clean F1 jacket. But the bigger story is what this means for fans. No more bouncing between apps or trying to figure out where the next session is airing. Race day just got way more convenient. Are you watching F1 on Apple TV this season? #formula1 #apple #appletv #gifted

Andru Edwards

345,892 views • 4 months ago

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Sagar

12,828 views • 1 month ago

For the first time, the latest LLMs run on the Apple Neural Engine — and NexaSDK is the only framework that makes it possible, powered by the NexaML engine. Last year, our two co-founders were invited by Apple DMLI team (Data & Machine Learning Innovation) to share their research about on-device multimodal model for local AI agents. One of the big questions in the room was: “Can the newest LLMs actually run on ANE?” At the time, nobody had a clear path. Today, that path exists. NexaSDK now runs Granite-4.0 (IBM), Qwen3 (Qwen), Gemma3 (Google), and Parakeet-v3 (NVIDIA) fully on Apple’s NPU — unlocking low-power, always-on, fast inference across Mac and iPhone. A new wave of NPU-first local AI apps is coming to Apple devices. Start with one line of code on Mac. iOS SDK coming soon.

For the first time, the latest LLMs run on the Apple Neural Engine — and NexaSDK is the only framework that makes it possible, powered by the NexaML engine. Last year, our two co-founders were invited by Apple DMLI team (Data & Machine Learning Innovation) to share their research about on-device multimodal model for local AI agents. One of the big questions in the room was: “Can the newest LLMs actually run on ANE?” At the time, nobody had a clear path. Today, that path exists. NexaSDK now runs Granite-4.0 (IBM), Qwen3 (Qwen), Gemma3 (Google), and Parakeet-v3 (NVIDIA) fully on Apple’s NPU — unlocking low-power, always-on, fast inference across Mac and iPhone. A new wave of NPU-first local AI apps is coming to Apple devices. Start with one line of code on Mac. iOS SDK coming soon.

NEXA AI

30,213 views • 8 months ago

Apple CEO : "Apple just made the Mac Mini small enough to disappear on your desk, but powerful enough to become your AI workstation." The new M4 Mac Mini starts at $599, comes with Apple Silicon, and pushes desktop compute into a box most people could hide behind a monitor. It is the next cheap entry point for local AI, automation, coding, content, and private workflows. Mac Mini + M4 + local models + low power = the new desk setup. Worth more than another $500 productivity course.

Apple CEO : "Apple just made the Mac Mini small enough to disappear on your desk, but powerful enough to become your AI workstation." The new M4 Mac Mini starts at $599, comes with Apple Silicon, and pushes desktop compute into a box most people could hide behind a monitor. It is the next cheap entry point for local AI, automation, coding, content, and private workflows. Mac Mini + M4 + local models + low power = the new desk setup. Worth more than another $500 productivity course.

Lummox

261,857 views • 1 month ago

Excited to introduce Nativ 🚀 Run frontier open models locally on your Mac. No accounts, no subscriptions, no cloud. ⚡ Built on mlx-vlm — multimodal + fastest on Apple Silicon 🔒 100% private — every token generated on your machine 🛠 Plug your coding agents into a local endpoint 📊 Live telemetry — tokens/sec, memory, usage 🆓 100% open source, MIT licensed, free forever Your Mac is more capable than you think. Stop renting intelligence. Download for macOS 👇 Github repo 🌟

Excited to introduce Nativ 🚀 Run frontier open models locally on your Mac. No accounts, no subscriptions, no cloud. ⚡ Built on mlx-vlm — multimodal + fastest on Apple Silicon 🔒 100% private — every token generated on your machine 🛠 Plug your coding agents into a local endpoint 📊 Live telemetry — tokens/sec, memory, usage 🆓 100% open source, MIT licensed, free forever Your Mac is more capable than you think. Stop renting intelligence. Download for macOS 👇 Github repo 🌟

Prince Canuma

115,001 views • 7 days ago

NVIDIA just built the laptop chip people are already calling the MacBook killer it is called RTX Spark, a single superchip that runs heavy creative work, real gaming, and private on-device AI agents, all on one machine but before you ditch your MacBook, here is the honest part nobody in the hype is telling you: > on raw memory bandwidth, a maxed macbook is actually still ahead, not behind > the RTX Spark laptops are not out yet, they ship this fall, apple is on shelves today > where nvidia truly wins is cuda, real RTX gaming, and 3D, things apple has no answer for > one is the proven machine you can buy now, the other is the more exciting bet for later so no, the macbook is not dead, and no, this is not just hype, the truth sits in between i broke down the full thing, every spec, who wins where, and which one is actually yours the honest breakdown is in the article below

NVIDIA just built the laptop chip people are already calling the MacBook killer it is called RTX Spark, a single superchip that runs heavy creative work, real gaming, and private on-device AI agents, all on one machine but before you ditch your MacBook, here is the honest part nobody in the hype is telling you: > on raw memory bandwidth, a maxed macbook is actually still ahead, not behind > the RTX Spark laptops are not out yet, they ship this fall, apple is on shelves today > where nvidia truly wins is cuda, real RTX gaming, and 3D, things apple has no answer for > one is the proven machine you can buy now, the other is the more exciting bet for later so no, the macbook is not dead, and no, this is not just hype, the truth sits in between i broke down the full thing, every spec, who wins where, and which one is actually yours the honest breakdown is in the article below

John Doe

107,985 views • 1 month ago

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched

Ejaaz

136,637 views • 1 month ago