`transformers` + `torchao` quantization + `torch.compile` for faster inference... speed and less memory usage 🔥 Demo of "meta-llama/Meta-Llama-3.1-8B-Instruct" quantized in 4-bit weight-only :show more

Mac Studio M2 Ultra is best Apple Silicon ever... (so far!) 🔥75 t/s and 0 throttling 🔥 Even after 20 mins of uninterrupted token generation with Apple MLX and mlx-community/Meta-Llama-3.1-8B-Instruct-8bit model!show more

Wild. Llama 4 Maverick just went Beast Mode 🔥... SambaNova partnered with AI at Meta to deliver the fastest Llama 4 inference on the market. Even the mighty Groq Inc can’t keep up! Let’s dive in ↓show more

smolagents is now available in ai-gradio pip install ai-gradio[smolagents]==0.2.1... then simply do import gradio as gr import ai_gradio gr.load( name='smolagents:meta-llama/Llama-3.1-8B-Instruct', src=ai_gradio.registry).launch()show more

We are thrilled to be a launch partner for... Meta Llama 3. Experience Llama 3 now at up to 350 tokens per second for Llama 3 8B and up to 150 tokens per second for Llama 3 70B, running in full FP16 precision on the Together API! 🤯show more

Introducing DeepThought-8B: Transparent reasoning model built on LLaMA-3.1 with... test-time compute scaling. - JSON-structured thought chains & controllable inference paths. - ~16GB VRAM, competitive w/ 70B models. - Open model weights, and inference scripts.show more

Llama 3.3 70B is live on AkashChat. The latest... state-of-the-art multilingual AI model released by Meta is as performant as Llama 3.1 405B at a fraction of the size. Try it today:show more

pip install spectralquant ✂️ Up to 6.62x KV cache... compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inferenceshow more

Meta just announced that Code Llama was now free... for both research and commercial. This might the strongest competitor to ChatGPT: ▸ Can generate, explain, and debug your code ▸ Handles input 100,000 tokens ▸ Free for research + commercial use ▸ Outperforms most open models ▸ Comes in 7B, 13B, and 34B ▸ Supports Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash Available in: ▸ Foundation base models (Code Llama) ▸ Python specializations (Code Llama - Python), ▸ Instruction-following models (Code Llama - Instruct)show more

.ollama is playing with AI at Meta Llama 4... Scout! 🤯 a perfect opportunity to test Ollama's giant super computer ✈️✈️✈️show more

Llama 3.2 is the latest open-source AI model from... Meta, released only a few hours ago. Here is the 3B parameter model running on Akash Chat at 165 tokens/second, powered by NVIDIA A100s on Akash. Try Llama 3.2 for free, no sign-in required:show more

Microsoft just a 1-bit LLM with 2B parameters that... can run on CPUs like Apple M2. BitNet b1.58 2B4T outperforms fp LLaMA 3.2 1B while using only 0.4GB memory versus 2GB and processes tokens 40% faster. 100% opensource.show more

How much faster is the new MacBook Pro for... AI inference? M4 Max is 27% faster with 72 tok/sec compared to 56 tok/sec of the M3 Max with MLX running Gemma 2 9B (4bit). The 27% speedup is the same with Llama-3.2-1b, Llama-3.2-3b and others. Next up: EXO Labs M4 cluster.show more

Multimodal Meta AI is rolling out widely on Ray-Ban... Meta starting today! It's a huge advancement for wearables & makes using AI more interactive & intuitive. Excited to share more on our multimodal work w/ Meta AI (& Llama 3), stay tuned for more updates coming soon.show more

AGI at home Running DeepSeek R1 across my 7... M4 Pro Mac Minis and 1 M4 Max MacBook Pro. Total unified memory = 496GB. Uses EXO Labs distributed inference with 4-bit quantization. Next goal is fp8 (requires >700GB)show more

The Meta Llama 3 Hackathon is this weekend in... SF with @Cerebral_Valley! Get on the list ➡️ What to expect • Two days of building alongside the best hackers in AI • Hands on support from the Llama team • Talks from some of the top names in the industryshow more

AI agents have arrived on Bittensor. Be the first... to launch one. It's fast and easy. TAO AGENTS - Smart contracts on Openτensor Foundaτion - Decentralized inference by Eternal AI - AI Model by AI at Meta (Llama 3.3) You can issue an instantly tradable token for your agent, too.show more

The first natively trained 1-bit model: BitNet 2B. Trained... on 4 trillion tokens. that can run on CPUs like Apple M2 Native 1.58-bit weights and 8bit activations W158A8 Outperforms LLaMA &close to Qwen 2.5 1.5B in while using only 0.4GB memory versus 2GB and processes tokens 40%show more

The easiest way to use this new model is... through HuggingChat with the link below. Just create a free account and select the model “nvidia/Llama-3.1-Nemotron-70B-Instruct-HF”. And you're ready to start chatting!show more

You can now run inference directly on the Llama... 4 Hugging Face model page – powered by Together AI!show more

MLX Swift LLM example works with: - Mistral /... Llama - Phi-2 - Qwen 1.5 - Starcoder 2 Quick-start: Qwen 1.5 0.5B runs pretty fast in 16-bit on my iPhone 14, no quantization needed:show more