Microsoft just a 1-bit LLM with 2B parameters that... can run on CPUs like Apple M2. BitNet b1.58 2B4T outperforms fp LLaMA 3.2 1B while using only 0.4GB memory versus 2GB and processes tokens 40% faster. 100% opensource.show more

You are not prepared for this, 250+ tokens/sec, 1B... model < 2GB memoryshow more

LightVAE + ComfyUI node: High-performance video VAE; runs 2–3x... faster using 50% less memory; LightTAE offers a 10+x speedup on just ~0.4GB VRAMshow more

LM Studio 0.3.4 ships with Apple MLX 🚢🍎 Run... on-device LLMs super fast, 100% locally and offline on your Apple Silicon Mac! Includes: > run Llama 3.2 1B at ~250 tok/sec (!) on M3 > enforce structured JSON responses > use via chat UI, or from your own code > run multiple models simultaneously > download any model from Hugging Face Video at 1x speed.show more

Llama 3.2 1B in 4-bit runs at ~60 toks/sec... with MLX Swift on my iPhone 15 pro. It's quite good and easily runs on-device:show more

GPT-4o level multimodal LLM running on your phone. MiniCPM-V... 4.5 outperforms GPT-4o, Gemini-2.0 Pro, and Qwen2.5-VL 72B on vision and language AI tasks. It can even understand videos and perform OCR in 30+ languages. And it's 100% Opensource.show more

How much faster is the new MacBook Pro for... AI inference? M4 Max is 27% faster with 72 tok/sec compared to 56 tok/sec of the M3 Max with MLX running Gemma 2 9B (4bit). The 27% speedup is the same with Llama-3.2-1b, Llama-3.2-3b and others. Next up: EXO Labs M4 cluster.show more

Latest mlx-lm has faster and lower memory prompt processing!... Thanks to causal fused attention from Jagrit Digani 7B 4-bit Mistral v3 can do ~30,000 tokens under a minute on my M4 Max laptop and only needs 8.5GB:show more

`transformers` + `torchao` quantization + `torch.compile` for faster inference... speed and less memory usage 🔥 Demo of "meta-llama/Meta-Llama-3.1-8B-Instruct" quantized in 4-bit weight-only :show more

Today Meta released "Code Llama", a large language model... fine-tuned for coding tasks. It's publicly available and can be used for commercial use! It outperforms GPT 3.5 and you can even run it locally on your Macbook using Ollama.show more

RAG is not Memory for AI Agents. 5 AI... memory engines to build agents that maintain long-term context and learn continuously: (Last 2 released just this month) 1. Zep builds and queries temporally-aware knowledge graphs that evolve with every interaction. 100% Opensource.show more

Llama 3.2 is the latest open-source AI model from... Meta, released only a few hours ago. Here is the 3B parameter model running on Akash Chat at 165 tokens/second, powered by NVIDIA A100s on Akash. Try Llama 3.2 for free, no sign-in required:show more

Microsoft is testing a new feature in Windows 11... called Low Latency Profile that gives a short boost to your CPU for 1 to 3 seconds when you open apps or menus. Early tests show Microsoft apps like Edge and Outlook open up to 40% faster, while the Start menu and other menus can load up to 70% faster, with CPU use jumping high for a few seconds then dropping right back down. Microsoft says this is normal because Apple does it in macOS and Linux, plus phones use the same type of quick CPU boosts for better speed. The feature is only in early Insider testing now, so it is not out for everyone yet.show more

You can run and monitor Claude Code from literally... anywhere. Start a coding session on your laptop, continue from your phone while on a walk, and get push notifications when Claude needs input. 100% Opensource and Free.show more

Meta released LongVU: a new video LM that can... handle long videos (great performance, battle-tested by me ⚔) TLDR; 1️⃣ downsample using DINOv2 to eliminate redundant scenes 🦖 2️⃣ fuse rest of the features using DINOv2 and SigLIP 3️⃣ select some tokens, pass to Qwen2/Llama-3.2-3Bshow more

RAG engine that just works for complex real-world documents.... It can extract knowledge from unlimited tokens across ANY data format - Word docs, Excel sheets, PDFs, and even scanned copies. And it's 100% Opensource.show more

Indian man finds romance with Fido while his H-1B processes

Perplexity's Sonar—built on Llama 3.3 70b—outperforms GPT-4o-mini and Claude... 3.5 Haiku while matching or surpassing top models like GPT-4o and Claude 3.5 Sonnet in user satisfaction. At 1200 tokens/second, Sonar is optimized for answer quality and speed.show more

I just created my own LaTeX-OCR app using Llama... 3.2 Vision! Upload the LaTeX code as an image, and it gives you the corresponding LaTeX code using Llama 3.2 multimodal! Here's what I used: - Ollama for serving Llama 3.2 vision locally - Streamlit for the UI Everything is just 50 lines of code! Find the code in the next tweet. -- Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs. Find me → Avi Chawlashow more

RAG is not Memory. AI agents need long-term memory... to maintain context and learn continuously. 5 frameworks for AI agent memory. 100% opensource. 1. Graphiti builds temporally-aware knowledge graphs for AI agents that change over time with evolving relationships and context.show more