正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Depth Anything 3 now runs as pure C++/ggml (ggml) . No Python, no PyTorch, no CUDA toolkit at inference, just one self-contained GGUF. It's faster than PyTorch on CPU! and ties speed on GPU. The CPU win came from the last place..I'd have looked. Quantized GGUF on Hugging Face🤗... show more

Ettore Di Giacinto

3,652 subscribers

34,915 次观看 • 4 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

parakeet.cpp: native C++/ggml (ggml) inference for NVIDIA AI Developer's Parakeet, one of the best speech-to-text models out there, from the LocalAI team. Every Parakeet model (TDT/CTC/RNNT/hybrid + cache-aware streaming), byte-for-byte identical output to NeMo, now running anywhere with no Python and even a bit faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIA AI Developer for releasing Parakeet! 🧵

parakeet.cpp: native C++/ggml (ggml) inference for NVIDIA AI Developer's Parakeet, one of the best speech-to-text models out there, from the LocalAI team. Every Parakeet model (TDT/CTC/RNNT/hybrid + cache-aware streaming), byte-for-byte identical output to NeMo, now running anywhere with no Python and even a bit faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIA AI Developer for releasing Parakeet! 🧵

Ettore Di Giacinto

55,955 次观看 • 5 个月前

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?

Alok

105,756 次观看 • 2 个月前

Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. Congratulations Georgi Gerganov ! This is a triumph.

Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. Congratulations Georgi Gerganov ! This is a triumph.

Nat Friedman

1,764,336 次观看 • 3 年前

Just dropped on HF — NeuTTS Air Next-gen on-device TTS that matches cloud-level quality while staying fully open source. > Real-time speech synthesis on CPU/GPU > 3-second voice cloning, no cloud or data upload > Compact: under 200 MB, runs on mobile and edge devices > Multilingual and expressive > Developed by Neuphonic , optimized for speed and fidelity

Just dropped on HF — NeuTTS Air Next-gen on-device TTS that matches cloud-level quality while staying fully open source. > Real-time speech synthesis on CPU/GPU > 3-second voice cloning, no cloud or data upload > Compact: under 200 MB, runs on mobile and edge devices > Multilingual and expressive > Developed by Neuphonic , optimized for speed and fidelity

steven

73,642 次观看 • 1 年前

llama.cpp isn't just for text LLMs anymore. Pure C++ zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!

llama.cpp isn't just for text LLMs anymore. Pure C++ zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!

Alok

47,881 次观看 • 2 个月前

8GB RTX 4060 Laptop. ~16GB Ornith IQ3_S GGUF. 44.5 tok/s streaming locally. FreeToken didn’t have the qwen35moe GGUF path needed for this model, so I implemented it and submitted PR #131 upstream. 👀 I also exposed the K/I GGUF quant types its CUDA kernels already supported through the Python loader. This is the result. Setup: Ornith-1.5-35B-A3B 35B MoE / ~3B active IQ3_S GGUF · ~16GB RTX 4060 Laptop · 8GB VRAM FreeToken · FlashInfer MoE backend: offload On the attached full-prompt run, I measured ~44.5 tok/s streaming on screen. In the same runs, FreeToken’s server decode counter reported 46.7 → 50.1 tok/s. VRAM stayed at 6,879 / 8,188 MiB, with 20GB host RAM and 98% GPU utilization during generation. Load time: 65s. My llama.cpp CPU reference on the same model was 11.1 tok/s. The interesting part isn’t just “35B on an 8GB GPU.” The code path itself was missing. FreeToken’s CUDA side already supported more GGUF quant types than its Python loader exposed, including the K/I quants needed for this IQ3_S model. And qwen35moe models like Ornith needed their own GGUF architecture/loading path. I implemented both pieces in my fork and opened them upstream in PR #131. Once loaded, FreeToken can use GPU + CPU + host RAM as an elastic MoE inference system instead of requiring the entire checkpoint to live in VRAM. That fits Ornith well because only a subset of its experts are active per token. Important caveat: PR #131 is still open. This is not merged upstream yet. I’m not calling this official Ornith support. I’m showing a working implementation, the code is public, and the attached terminal video is the actual run. PR #131: My fork: Next I’m going to test where this path holds up across more quants, longer context and real agent workloads.

8GB RTX 4060 Laptop. ~16GB Ornith IQ3_S GGUF. 44.5 tok/s streaming locally. FreeToken didn’t have the qwen35moe GGUF path needed for this model, so I implemented it and submitted PR #131 upstream. 👀 I also exposed the K/I GGUF quant types its CUDA kernels already supported through the Python loader. This is the result. Setup: Ornith-1.5-35B-A3B 35B MoE / ~3B active IQ3_S GGUF · ~16GB RTX 4060 Laptop · 8GB VRAM FreeToken · FlashInfer MoE backend: offload On the attached full-prompt run, I measured ~44.5 tok/s streaming on screen. In the same runs, FreeToken’s server decode counter reported 46.7 → 50.1 tok/s. VRAM stayed at 6,879 / 8,188 MiB, with 20GB host RAM and 98% GPU utilization during generation. Load time: 65s. My llama.cpp CPU reference on the same model was 11.1 tok/s. The interesting part isn’t just “35B on an 8GB GPU.” The code path itself was missing. FreeToken’s CUDA side already supported more GGUF quant types than its Python loader exposed, including the K/I quants needed for this IQ3_S model. And qwen35moe models like Ornith needed their own GGUF architecture/loading path. I implemented both pieces in my fork and opened them upstream in PR #131. Once loaded, FreeToken can use GPU + CPU + host RAM as an elastic MoE inference system instead of requiring the entire checkpoint to live in VRAM. That fits Ornith well because only a subset of its experts are active per token. Important caveat: PR #131 is still open. This is not merged upstream yet. I’m not calling this official Ornith support. I’m showing a working implementation, the code is public, and the attached terminal video is the actual run. PR #131: My fork: Next I’m going to test where this path holds up across more quants, longer context and real agent workloads.

Cruz

12,257 次观看 • 2 个月前

so we already have an open source alternative to jev... and it's 6-7x faster?! it's a typed-decision classification system: no chat, no generated text, just fast yes/no, scoring, or choice answers. > runs in under 1gb of memory > free on hugging face > runs on a laptop, or even a phone there's a real demo where it plays snake live, making a fresh decision every single move. two honest limits, straight from the project itself: > only 512-1,024 token context, some cases won't fit > weaker generalization than jev out of the box

so we already have an open source alternative to jev... and it's 6-7x faster?! it's a typed-decision classification system: no chat, no generated text, just fast yes/no, scoring, or choice answers. > runs in under 1gb of memory > free on hugging face > runs on a laptop, or even a phone there's a real demo where it plays snake live, making a fresh decision every single move. two honest limits, straight from the project itself: > only 512-1,024 token context, some cases won't fit > weaker generalization than jev out of the box

Simplifying AI

48,083 次观看 • 1 个月前

We are building “Open Source Nano Banana for Video” - here is open source demo v0.1 We are open sourcing Lucy Edit, the first foundation model for text-guided video editing! Get the model on Hugging Face 🤗, API on @FAL, and nodes on ComfyUI 🧵

We are building “Open Source Nano Banana for Video” - here is open source demo v0.1 We are open sourcing Lucy Edit, the first foundation model for text-guided video editing! Get the model on Hugging Face 🤗, API on @FAL, and nodes on ComfyUI 🧵

Decart

416,688 次观看 • 1 年前

Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.

Holy shit... Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.

Guri Singh

2,180,357 次观看 • 7 个月前

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference

pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): - KV cache integration via Hugging Face's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge) - Mistral 7B / Qwen 2.5 7B / Llama 3.1 8B verified - Pure PyTorch + future CUDA kernel support - Auto-calibration from a bundled corpus 📰 Paper: 💻 Code, quickstart, and benchmarks: #LLM #Inference #PyTorch #OpenSource #MachineLearning #LLM #KVCache #Inference

ani

17,980 次观看 • 4 个月前

They were building in stealth for 2 years, I was building in stealth for 2 hours… Happy to open source Qwen-2.5-1B-RLCD, 5x faster on-device inference for JSON workloads that need to be type-safe. ⚡️Demo below on a M4 MacBook⚡️ every LLM has the ability to efficiently batch inference every key of a JSON at the same time and generate probabilities from a set of possible categories. No new training required, but it’s easy to optimize if you need! On hugging face now!

They were building in stealth for 2 years, I was building in stealth for 2 hours… Happy to open source Qwen-2.5-1B-RLCD, 5x faster on-device inference for JSON workloads that need to be type-safe. ⚡️Demo below on a M4 MacBook⚡️ every LLM has the ability to efficiently batch inference every key of a JSON at the same time and generate probabilities from a set of possible categories. No new training required, but it’s easy to optimize if you need! On hugging face now!

Harsha Gundala

1,299,334 次观看 • 1 个月前

I don’t have the BRAM for a depth buffer so it’s time to get the memory controller up and running for this fjord torus. 100% FPGA logic 3D pipeline, no CPU.

I don’t have the BRAM for a depth buffer so it’s time to get the memory controller up and running for this fjord torus. 100% FPGA logic 3D pipeline, no CPU.

Ian Hanschen

29,249 次观看 • 8 个月前

Microsoft killed the GPU mafia 🤯 They finally open-sourced their 1-bit LLM inference framework called bitnet.cpp. It lets you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.

Microsoft killed the GPU mafia 🤯 They finally open-sourced their 1-bit LLM inference framework called bitnet.cpp. It lets you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.

Oliver Prompts

1,632,375 次观看 • 9 个月前

THIS ENGINEER SKIPPED 7,862 WORDS OF AI ROADMAP AND SHIPPED ON A WEEKEND a Pi Zero 2W, an IMX500 camera and a solar panel, that's the whole stack inference runs on-chip so there's no cloud, no latency and no monthly bill Spotted Dove at 0.95, King Parrot at 0.88, runs forever on sun code is open source and the 3D files are free, you just print the case and buy the parts and it works better than most paid wildlife monitoring services out there full video: code (open source): 3D print files: solar power board: save this so you actually have everything in one place when you decide to build it

THIS ENGINEER SKIPPED 7,862 WORDS OF AI ROADMAP AND SHIPPED ON A WEEKEND a Pi Zero 2W, an IMX500 camera and a solar panel, that's the whole stack inference runs on-chip so there's no cloud, no latency and no monthly bill Spotted Dove at 0.95, King Parrot at 0.88, runs forever on sun code is open source and the 3D files are free, you just print the case and buy the parts and it works better than most paid wildlife monitoring services out there full video: code (open source): 3D print files: solar power board: save this so you actually have everything in one place when you decide to build it

leopardracer

63,096 次观看 • 5 个月前

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

Step 4 to achieve truly serverless GPUs for AI inference: skip over unserializable inference engine setup steps like CUDA graph capture and Torch compilation by stacking GPU snapshots and CPU snapshots.

Charles 🎉 Frye

17,452 次观看 • 5 个月前

Microsoft just changed the game 🤯 They've open-sourced bitnet.cpp, a 1-bit LLM inference framework. It let's you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.

Microsoft just changed the game 🤯 They've open-sourced bitnet.cpp, a 1-bit LLM inference framework. It let's you run 100B parameter models on your local CPU without GPUs. - 6.17x faster inference - 82.2% less energy on CPUs 100% Open Source.

Oliver Prompts

115,338 次观看 • 18 天前

Someone just built a desktop app that that generates 3D models from images and runs 100% locally. It's called Modly. It runs entirely on your GPU, no cloud, no API bills. Just drop an image and get a 3D mesh. 100% Open Source.

Someone just built a desktop app that that generates 3D models from images and runs 100% locally. It's called Modly. It runs entirely on your GPU, no cloud, no API bills. Just drop an image and get a 3D mesh. 100% Open Source.

How To Prompt

238,264 次观看 • 6 个月前

Llama 2: Now on Hugging Chat 🤗🦙 Try out the 70B Chat model for free with super fast inference, web search, and powered by open-source tools! 👉

Llama 2: Now on Hugging Chat 🤗🦙 Try out the 70B Chat model for free with super fast inference, web search, and powered by open-source tools! 👉

Hugging Face

403,843 次观看 • 3 年前

New vs Old! Epyc 64-Core CPU vs DEC VAX! I like to compare weird things on an even playing field, and so I installed NetBSD 10.1 on both a VAX 4000-705A and on an Epyc 8534P, 64-core CPU with 128GB of RAM. The 1993 VAX runs at 112MHz (pretty fast for a VAX!) Then I set them both to building the NetBSD source tree. Epyc on the top, VAX on the bottom. And this VAX is at least 50x as fast as the original VAX 11/780!

New vs Old! Epyc 64-Core CPU vs DEC VAX! I like to compare weird things on an even playing field, and so I installed NetBSD 10.1 on both a VAX 4000-705A and on an Epyc 8534P, 64-core CPU with 128GB of RAM. The 1993 VAX runs at 112MHz (pretty fast for a VAX!) Then I set them both to building the NetBSD source tree. Epyc on the top, VAX on the bottom. And this VAX is at least 50x as fast as the original VAX 11/780!

Dave W Plummer

21,590 次观看 • 1 年前

We’re incredibly excited to launch Open Benchmarks Grants, a new program committing $3M in grants to fund new open source benchmarks advancing agentic AI. We’re partnering up with Hugging Face, Together AI, Prime Intellect, Factory HQ, Harbor Framework, and PyTorch to support academic and open-source teams as they define and advance the AI frontier.

We’re incredibly excited to launch Open Benchmarks Grants, a new program committing $3M in grants to fund new open source benchmarks advancing agentic AI. We’re partnering up with Hugging Face, Together AI, Prime Intellect, Factory HQ, Harbor Framework, and PyTorch to support academic and open-source teams as they define and advance the AI frontier.

Snorkel AI

664,985 次观看 • 8 个月前