Загрузка видео...

Не удалось загрузить видео

На главную

You can now run SQLCoder with a GUI on Apple Silicon or any NVIDIA GPU-enabled device! On Apple Silicon, just run CMAKE_ARGS="-DLLAMA_METAL=on" pip install "sqlcoder[llama-cpp]" sqlcoder launch The Apple Silicon version is not super accurate, but works great for simple questions – and everything happens (fairly fast) on just...

rishdotblog's profile picture

Rishabh Srivastava

12,553 subscribers

10,405 просмотров • 2 лет назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 6

Фото профиля Cris Lenta
Cris Lenta2 лет назад

This looks great! Have you tried also running it with .cpp models? Curious of the performance

Фото профиля Rishabh Srivastava
Rishabh Srivastava2 лет назад

Thank you! The demo above is for a llama.cpp model (with q5 km quantization). Haven't done a rigorous eval on those yet though – will do in a few hours!

Фото профиля Nerdynav
Nerdynav2 лет назад

hey great work, what's the license for this? Are code llama fine tunes allowed in their Meta's licensing?

Фото профиля Gautam Padiyar
Gautam Padiyar2 лет назад

GPU with 20gb memory not required?

Фото профиля Rishabh Srivastava
Rishabh Srivastava2 лет назад

7B model, so you only need ~16GB or so of memory to make it run without beam search

Фото профиля Harsh Patel
Harsh Patel2 лет назад

On Apple silicon how do I make it use the 7b-2 model? It is downloading the v1 I guess

Похожие видео

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:
0:20

The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:

Awni Hannun

42,761 просмотров • 1 год назад

Weekend plans? How about diving into LLMs on Apple devices? 🕷️💻 Just dropped my first tutorial: Run LLMs on your Mac with MLX! Perfect for AI beginners with Apple Silicon. More tech adventures coming soon! Please watch & subscribe:
11:04

Weekend plans? How about diving into LLMs on Apple devices? 🕷️💻 Just dropped my first tutorial: Run LLMs on your Mac with MLX! Perfect for AI beginners with Apple Silicon. More tech adventures coming soon! Please watch & subscribe:

qnguyen3

27,144 просмотров • 2 лет назад

Gemma 4 now works on-device using React Native! You can now run Gemma 4 fully offline in your cross-platform apps with local hardware acceleration: ⚡ Vulkan delegate on Android ⚡ MLX delegate on Apple Silicon See Gemma 4's vision and tool-use capabilities in action, instantly reading a flyer and scheduling a calendar event, 100% on-device.
8:59

Gemma 4 now works on-device using React Native! You can now run Gemma 4 fully offline in your cross-platform apps with local hardware acceleration: ⚡ Vulkan delegate on Android ⚡ MLX delegate on Apple Silicon See Gemma 4's vision and tool-use capabilities in action, instantly reading a flyer and scheduling a calendar event, 100% on-device.

Google Gemma

115,299 просмотров • 19 дней назад

Alibaba’s Qwen 3.5 is now running fully on-device on the iPhone 17 Pro. It outperforms models 4x its size, delivers strong visual understanding, and lets you switch reasoning on or off. This demo uses the 2B 6-bit version, optimized with MLX for Apple Silicon.
0:24

Alibaba’s Qwen 3.5 is now running fully on-device on the iPhone 17 Pro. It outperforms models 4x its size, delivers strong visual understanding, and lets you switch reasoning on or off. This demo uses the 2B 6-bit version, optimized with MLX for Apple Silicon.

Hugging Models

202,781 просмотров • 4 месяцев назад

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)
4:07

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Prince Canuma

123,480 просмотров • 1 год назад

Claude Code just got a lot easier to use 🙌 Just updated You can now install agents, commands, MCPs, and templates with a single command. Click on any card and it gives you the exact line to run. Everything is pulled from GitHub, the site runs on GitHub Pages, and downloads are logged with GitHub Actions.
1:05

Claude Code just got a lot easier to use 🙌 Just updated You can now install agents, commands, MCPs, and templates with a single command. Click on any card and it gives you the exact line to run. Everything is pulled from GitHub, the site runs on GitHub Pages, and downloads are logged with GitHub Actions.

Daniel San

135,238 просмотров • 1 год назад

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.
10:36

HERMES CAN NOW RUN FOR FREE FOREVER A new Gemma 4 update just removed the biggest problem with local AI agents. The Upgrade: → Gemma 4 runs around 90% faster on Apple Silicon using Ollama + MLX → Plug it into Hermes with a few terminal commands → Run private agent workflows without subscriptions, token limits, or API bills What It Can Actually Do: ✓ Build and preview apps locally ✓ Use /learn to turn tutorials into permanent skills ✓ Read files, analyse landing pages, summarise emails, and track AI news ✓ Run loops until a task is complete instead of waiting for another prompt ✓ Launch multiple Gemma 4 subagents to work on separate tasks in parallel No Apple Silicon? ✔ OpenRouter offers the 31B model through a free API The real win is not faster chat. It is having Hermes build, retry, check its work, and operate in the background without charging you for every loop.

Julian Goldie SEO

11,234 просмотров • 23 дней назад

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.
2:41

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

36,711 просмотров • 8 дней назад

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨
0:46

ParoQuant just got a big upgrade 🚀 ✅ Supports the new Qwen3.5 models ⚡ Now runs on MLX (fast local inference on Apple Silicon) 🧠 Preserves reasoning quality with 4-bit quantization We also built an agent demo running locally on my 4-year-old M2 Max. Can't wait to upgrade to an M5 Max and see what kind of magic we can do. ✨

Zhijian Liu

49,358 просмотров • 4 месяцев назад

gpt4o in ChatGPT vs Apple Intelligence (on device Foundation model) prompt: "what are the top 3 coffee shops in lower manhattan?" not saying one is better than the other but fun to test. reply with your prompts and ill run a side by side for you.
0:26

gpt4o in ChatGPT vs Apple Intelligence (on device Foundation model) prompt: "what are the top 3 coffee shops in lower manhattan?" not saying one is better than the other but fun to test. reply with your prompts and ill run a side by side for you.

jpa

248,720 просмотров • 1 год назад

Apple just upgraded the Formula 1 experience! 🏎️ Apple just made Formula One a lot easier to follow in the U.S. One place for the full race weekend. Practice, qualifying, sprint sessions, and every Grand Prix, live or on demand. The gifted package was cool, too. Apple TV 4K, pins, and a seriously clean F1 jacket. But the bigger story is what this means for fans. No more bouncing between apps or trying to figure out where the next session is airing. Race day just got way more convenient. Are you watching F1 on Apple TV this season? #formula1 #apple #appletv #gifted
1:02

Apple just upgraded the Formula 1 experience! 🏎️ Apple just made Formula One a lot easier to follow in the U.S. One place for the full race weekend. Practice, qualifying, sprint sessions, and every Grand Prix, live or on demand. The gifted package was cool, too. Apple TV 4K, pins, and a seriously clean F1 jacket. But the bigger story is what this means for fans. No more bouncing between apps or trying to figure out where the next session is airing. Race day just got way more convenient. Are you watching F1 on Apple TV this season? #formula1 #apple #appletv #gifted

Andru Edwards

345,892 просмотров • 4 месяцев назад

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!
0:28

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Sagar

12,828 просмотров • 1 месяц назад

For the first time, the latest LLMs run on the Apple Neural Engine — and NexaSDK is the only framework that makes it possible, powered by the NexaML engine. Last year, our two co-founders were invited by Apple DMLI team (Data & Machine Learning Innovation) to share their research about on-device multimodal model for local AI agents. One of the big questions in the room was: “Can the newest LLMs actually run on ANE?” At the time, nobody had a clear path. Today, that path exists. NexaSDK now runs Granite-4.0 (IBM), Qwen3 (Qwen), Gemma3 (Google), and Parakeet-v3 (NVIDIA) fully on Apple’s NPU — unlocking low-power, always-on, fast inference across Mac and iPhone. A new wave of NPU-first local AI apps is coming to Apple devices. Start with one line of code on Mac. iOS SDK coming soon.
1:42

For the first time, the latest LLMs run on the Apple Neural Engine — and NexaSDK is the only framework that makes it possible, powered by the NexaML engine. Last year, our two co-founders were invited by Apple DMLI team (Data & Machine Learning Innovation) to share their research about on-device multimodal model for local AI agents. One of the big questions in the room was: “Can the newest LLMs actually run on ANE?” At the time, nobody had a clear path. Today, that path exists. NexaSDK now runs Granite-4.0 (IBM), Qwen3 (Qwen), Gemma3 (Google), and Parakeet-v3 (NVIDIA) fully on Apple’s NPU — unlocking low-power, always-on, fast inference across Mac and iPhone. A new wave of NPU-first local AI apps is coming to Apple devices. Start with one line of code on Mac. iOS SDK coming soon.

NEXA AI

30,213 просмотров • 8 месяцев назад

Apple CEO : "Apple just made the Mac Mini small enough to disappear on your desk, but powerful enough to become your AI workstation." The new M4 Mac Mini starts at $599, comes with Apple Silicon, and pushes desktop compute into a box most people could hide behind a monitor. It is the next cheap entry point for local AI, automation, coding, content, and private workflows. Mac Mini + M4 + local models + low power = the new desk setup. Worth more than another $500 productivity course.
5:32

Apple CEO : "Apple just made the Mac Mini small enough to disappear on your desk, but powerful enough to become your AI workstation." The new M4 Mac Mini starts at $599, comes with Apple Silicon, and pushes desktop compute into a box most people could hide behind a monitor. It is the next cheap entry point for local AI, automation, coding, content, and private workflows. Mac Mini + M4 + local models + low power = the new desk setup. Worth more than another $500 productivity course.

Lummox

261,857 просмотров • 1 месяц назад

Excited to introduce Nativ 🚀 Run frontier open models locally on your Mac. No accounts, no subscriptions, no cloud. ⚡ Built on mlx-vlm — multimodal + fastest on Apple Silicon 🔒 100% private — every token generated on your machine 🛠 Plug your coding agents into a local endpoint 📊 Live telemetry — tokens/sec, memory, usage 🆓 100% open source, MIT licensed, free forever Your Mac is more capable than you think. Stop renting intelligence. Download for macOS 👇 Github repo 🌟
3:19

Excited to introduce Nativ 🚀 Run frontier open models locally on your Mac. No accounts, no subscriptions, no cloud. ⚡ Built on mlx-vlm — multimodal + fastest on Apple Silicon 🔒 100% private — every token generated on your machine 🛠 Plug your coding agents into a local endpoint 📊 Live telemetry — tokens/sec, memory, usage 🆓 100% open source, MIT licensed, free forever Your Mac is more capable than you think. Stop renting intelligence. Download for macOS 👇 Github repo 🌟

Prince Canuma

115,001 просмотров • 7 дней назад

NVIDIA just built the laptop chip people are already calling the MacBook killer it is called RTX Spark, a single superchip that runs heavy creative work, real gaming, and private on-device AI agents, all on one machine but before you ditch your MacBook, here is the honest part nobody in the hype is telling you: > on raw memory bandwidth, a maxed macbook is actually still ahead, not behind > the RTX Spark laptops are not out yet, they ship this fall, apple is on shelves today > where nvidia truly wins is cuda, real RTX gaming, and 3D, things apple has no answer for > one is the proven machine you can buy now, the other is the more exciting bet for later so no, the macbook is not dead, and no, this is not just hype, the truth sits in between i broke down the full thing, every spec, who wins where, and which one is actually yours the honest breakdown is in the article below
0:34

NVIDIA just built the laptop chip people are already calling the MacBook killer it is called RTX Spark, a single superchip that runs heavy creative work, real gaming, and private on-device AI agents, all on one machine but before you ditch your MacBook, here is the honest part nobody in the hype is telling you: > on raw memory bandwidth, a maxed macbook is actually still ahead, not behind > the RTX Spark laptops are not out yet, they ship this fall, apple is on shelves today > where nvidia truly wins is cuda, real RTX gaming, and 3D, things apple has no answer for > one is the proven machine you can buy now, the other is the more exciting bet for later so no, the macbook is not dead, and no, this is not just hype, the truth sits in between i broke down the full thing, every spec, who wins where, and which one is actually yours the honest breakdown is in the article below

John Doe

107,985 просмотров • 1 месяц назад

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched
1:30

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched

Ejaaz

136,637 просмотров • 1 месяц назад