Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

The GeoAI Python package now supports object detection using pre-trained models from the GeoDeep libarary ( The supported object types include cars, trees, birds, planes, aerovision, utilities, buildings, and roads. Try it out: GitHub: Notebook example: #geospatial #geoai #opensource #python

Qiusheng Wu

54,745 subscribers

87,303 просмотров • 7 месяцев назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

🚀 The #GeoAI Python package now supports feature segmentation from high-resolution satellite and aerial imagery using text prompts, such as trees, buildings, etc. It efficiently processes large datasets with automatic tiling and can save results as a single image. Stay tuned for more features coming soon! 📓 Access the notebook: 🛠️ Explore the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist: #GeoAI #geospatial #AI #Python #DeepLearning

🚀 The #GeoAI Python package now supports feature segmentation from high-resolution satellite and aerial imagery using text prompts, such as trees, buildings, etc. It efficiently processes large datasets with automatic tiling and can save results as a single image. Stay tuned for more features coming soon! 📓 Access the notebook: 🛠️ Explore the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist: #GeoAI #geospatial #AI #Python #DeepLearning

Qiusheng Wu

13,923 просмотров • 1 год назад

🚀 Exciting news! The #GeoAI Python package now lets you train land cover classification models with just one line of code. Leverage any PyTorch segmentation model from — with hundreds of image encoders & pretrained weights available. 📍 GitHub: 📓 Notebook: #GeoAI #Geospatial #DeepLearning

🚀 Exciting news! The #GeoAI Python package now lets you train land cover classification models with just one line of code. Leverage any PyTorch segmentation model from — with hundreds of image encoders & pretrained weights available. 📍 GitHub: 📓 Notebook: #GeoAI #Geospatial #DeepLearning

Qiusheng Wu

16,210 просмотров • 1 год назад

🚀 A sneak peek of a new feature in the #GeoAI Python package! Now you can train an image segmentation model for extracting features (e.g., buildings) from satellite or aerial imagery—all with just a few lines of code. 🛠️ Check out the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist: #GeoAI #geospatial #AI #Python #DeepLearning

🚀 A sneak peek of a new feature in the #GeoAI Python package! Now you can train an image segmentation model for extracting features (e.g., buildings) from satellite or aerial imagery—all with just a few lines of code. 🛠️ Check out the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist: #GeoAI #geospatial #AI #Python #DeepLearning

Qiusheng Wu

11,284 просмотров • 1 год назад

🚀 I spent the entire day training image segmentation models from scratch! I've created several pretrained models to detect features like buildings, cars, ships, solar panels, and wetlands. Video tutorials are coming soon! 🤗 Check out the pretrained models on Hugging Face: 🛠️ Check out the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist: #GeoAI #geospatial #AI #Python #DeepLearning

🚀 I spent the entire day training image segmentation models from scratch! I've created several pretrained models to detect features like buildings, cars, ships, solar panels, and wetlands. Video tutorials are coming soon! 🤗 Check out the pretrained models on Hugging Face: 🛠️ Check out the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist: #GeoAI #geospatial #AI #Python #DeepLearning

Qiusheng Wu

15,101 просмотров • 1 год назад

A sneak peek of a new feature in the #GeoAI Python package! 🎉 Now you can detect cars from georeferenced aerial imagery using deep learning—all with just a few lines of code. Stay tuned for an in-depth video tutorial coming soon! 🛠️ Explore the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist:

A sneak peek of a new feature in the #GeoAI Python package! 🎉 Now you can detect cars from georeferenced aerial imagery using deep learning—all with just a few lines of code. Stay tuned for an in-depth video tutorial coming soon! 🛠️ Explore the GitHub repository: 📚 Dive into the documentation: 📺 Check out the entire YouTube playlist:

Qiusheng Wu

45,737 просмотров • 1 год назад

📢Visualize hyperspectral data in 3D 🌍 with just a few lines of code using the HyperCoast #Python package . It supports popular datasets like AVIRIS, DESIS, EMIT, PACE, and NEON AOP. GitHub: Notebook: bingqing liu #geospaptial #hyperspectral #nasa #dataviz

📢Visualize hyperspectral data in 3D 🌍 with just a few lines of code using the HyperCoast #Python package . It supports popular datasets like AVIRIS, DESIS, EMIT, PACE, and NEON AOP. GitHub: Notebook: bingqing liu #geospaptial #hyperspectral #nasa #dataviz

Qiusheng Wu

13,503 просмотров • 2 лет назад

So it works now! If you have Bakery, can you try getting the latest github patch and using it? Add "Bakery Light Filter" component to an object to make it act like stained glass 😋 (video from RTPreview for fanciness, but it looks the same when baked) #gamedev

So it works now! If you have Bakery, can you try getting the latest github patch and using it? Add "Bakery Light Filter" component to an object to make it act like stained glass 😋 (video from RTPreview for fanciness, but it looks the same when baked) #gamedev

Mr F

236,686 просмотров • 1 год назад

Create Stunning Time-Series Satellite Images in Seconds! The GEE Data Catalogs Plugin v0.5 for QGIS is now available and it's a powerful upgrade. You can now create time-series satellite imagery with just a few clicks using a simple interface. The new version also supports direct downloads to your computer, making the workflow faster and more efficient. Key Features: - Access over 80 petabytes of satellite and geospatial datasets from Google Earth Engine - Generate animated time-series imagery effortlessly - Export results directly from QGIS to your local machine Useful Links: QGIS Plugin Page: GitHub Repository: Video Tutorial: #QGIS #geospatial #EarthEngine #Python #datascience #satellite

Create Stunning Time-Series Satellite Images in Seconds! The GEE Data Catalogs Plugin v0.5 for QGIS is now available and it's a powerful upgrade. You can now create time-series satellite imagery with just a few clicks using a simple interface. The new version also supports direct downloads to your computer, making the workflow faster and more efficient. Key Features: - Access over 80 petabytes of satellite and geospatial datasets from Google Earth Engine - Generate animated time-series imagery effortlessly - Export results directly from QGIS to your local machine Useful Links: QGIS Plugin Page: GitHub Repository: Video Tutorial: #QGIS #geospatial #EarthEngine #Python #datascience #satellite

Qiusheng Wu

14,171 просмотров • 9 месяцев назад

Predicting the next word "only" is sufficient for language models to learn a large body of knowledge that enables then to code, answer questions, understand many topics, chat, and so on. This is clear to many researchers now, and there are nice tutorials on why this works by Ilya Sutskever resorting to compression ( ) and by Geoffrey Hinton ( ). However, the emergence of types of understanding is not unique to language models. In by Misha Denil and Brandon Amos the authors trained models to predict the next few time stems of over a hundred robot hand sensors (Touch, Gyro, Accelerometer, Joint Info, Actuator Info, etc.). They ten found out that they could regress the shape of the thing the hand was touching from the activations of the neural networks using probes. That is, the model developed an internal representation of shapes even though it was simply used to predict "only" the next few senses. Awareness follows from simple predictions and interaction with the world.

Predicting the next word "only" is sufficient for language models to learn a large body of knowledge that enables then to code, answer questions, understand many topics, chat, and so on. This is clear to many researchers now, and there are nice tutorials on why this works by Ilya Sutskever resorting to compression ( ) and by Geoffrey Hinton ( ). However, the emergence of types of understanding is not unique to language models. In by Misha Denil and Brandon Amos the authors trained models to predict the next few time stems of over a hundred robot hand sensors (Touch, Gyro, Accelerometer, Joint Info, Actuator Info, etc.). They ten found out that they could regress the shape of the thing the hand was touching from the activations of the neural networks using probes. That is, the model developed an internal representation of shapes even though it was simply used to predict "only" the next few senses. Awareness follows from simple predictions and interaction with the world.

Nando de Freitas

134,455 просмотров • 2 лет назад

Blender just became a prompt box. With Kimi K3 connected through Blender MCP, you can describe a scene like this in plain English and let the model handle the ugly part: terrain, buildings, lighting, materials, camera movement, animation, and the Python scripts holding everything together. The interesting part is not the first render. Kimi can inspect the result, notice that the camera clips through a tree or the city looks suspiciously like plastic, then edit the actual Blender scene and render it again instead of restarting from zero. You still need taste, because “make it cinematic” remains one of humanity’s least useful instructions. But the distance between an empty Blender file and a fully editable 3D world just got embarrassingly small. A few years ago, creating this meant weeks of modeling, scripting, lighting, and animation. Now you can describe the world, watch Kimi build it, and spend your time fixing the final 10% instead of manually constructing the first 90%. Kimi K3 + Blender MCP is basically text-to-3D without trapping the result inside a useless generated video. Every object, material, light, camera, and keyframe stays editable.

Blender just became a prompt box. With Kimi K3 connected through Blender MCP, you can describe a scene like this in plain English and let the model handle the ugly part: terrain, buildings, lighting, materials, camera movement, animation, and the Python scripts holding everything together. The interesting part is not the first render. Kimi can inspect the result, notice that the camera clips through a tree or the city looks suspiciously like plastic, then edit the actual Blender scene and render it again instead of restarting from zero. You still need taste, because “make it cinematic” remains one of humanity’s least useful instructions. But the distance between an empty Blender file and a fully editable 3D world just got embarrassingly small. A few years ago, creating this meant weeks of modeling, scripting, lighting, and animation. Now you can describe the world, watch Kimi build it, and spend your time fixing the final 10% instead of manually constructing the first 90%. Kimi K3 + Blender MCP is basically text-to-3D without trapping the result inside a useless generated video. Every object, material, light, camera, and keyframe stays editable.

Rina

264,903 просмотров • 3 месяцев назад

Big moment for text-to-speech. Qwen just open-sourced a text-to-speech model that lets you clone voices, design new ones, and control speech using natural language. Let me explain what I mean: You can literally tell it "speak in a cheerful tone with slight nervousness," and it actually does that. No complex audio engineering needed. What makes this special: - 3-second voice cloning - Covers 10 languages: English, German, French, and more - Latency as low as 97ms for real-time applications - Supports both streaming and non-streaming generation The model comes in two sizes (0.6B and 1.7B parameters), so you can pick based on your hardware and quality needs. Three modes to work with: 1. Custom Voice: Use pre-built premium voices with instruction-based style control 2. Voice Design: Describe the voice you want in plain English (or Chinese), and the model creates it 3. Voice Clone: Provide a 3-second reference audio and clone that voice The best part? It integrates with vLLM for production deployment and has a simple Python package you can pip install. I've shared a link to the GitHub repo in the next tweet.

Big moment for text-to-speech. Qwen just open-sourced a text-to-speech model that lets you clone voices, design new ones, and control speech using natural language. Let me explain what I mean: You can literally tell it "speak in a cheerful tone with slight nervousness," and it actually does that. No complex audio engineering needed. What makes this special: - 3-second voice cloning - Covers 10 languages: English, German, French, and more - Latency as low as 97ms for real-time applications - Supports both streaming and non-streaming generation The model comes in two sizes (0.6B and 1.7B parameters), so you can pick based on your hardware and quality needs. Three modes to work with: 1. Custom Voice: Use pre-built premium voices with instruction-based style control 2. Voice Design: Describe the voice you want in plain English (or Chinese), and the model creates it 3. Voice Clone: Provide a 3-second reference audio and clone that voice The best part? It integrates with vLLM for production deployment and has a simple Python package you can pip install. I've shared a link to the GitHub repo in the next tweet.

Akshay 🚀

31,249 просмотров • 9 месяцев назад

Two weeks ago I fixed one of my teeth with algorithms I wrote a couple of years ago! I got hooked by 3D scanning when I started to work for a software shop in Zurich that was programming 3D computational geometry algorithms for denture scanning to produce crowns (and more). Back then, a typical reconstruction pipeline was like: scan the patient’s teeth using an intraoral scanner, reconstruct the surface mesh, design the restoration digitally, and finally mill the crown out of ceramic. We were working mostly with point clouds and meshes, but it wasn’t just math, it was craftsmanship translated into a digital process. Every micron mattered. You could literally see how a good algorithm meant a better fit in someone’s mouth. Gaussian Splatting isn’t about surface reconstruction, it’s about appearance reconstruction. It doesn’t care about explicit topology, it captures how light interacts with the scene. In a sense, it’s the opposite philosophy of the dental world: instead of modeling what the object is, it models how the object looks. 3D Gaussian Splatting enables applications like training self driving cars, teaching robots to understand their environment, creating virtual worlds, or monitoring real sites. It represents scenes as millions of small Gaussians rendered in real time without the need for meshes or textures. Coming from a world where precision geometry was everything, this shift felt natural. It’s still about reconstruction, but with a different goal: not manufacturing a perfect object, but reproducing how the world actually looks. Two weeks ago I got my first dental crown, made with the same software, reconstruction algorithms, and Swiss precision I once helped develop. I haven’t worked there in two years, but sitting in that chair and seeing the process from the other side was a proud moment. It reminded me why I love this field.

Two weeks ago I fixed one of my teeth with algorithms I wrote a couple of years ago! I got hooked by 3D scanning when I started to work for a software shop in Zurich that was programming 3D computational geometry algorithms for denture scanning to produce crowns (and more). Back then, a typical reconstruction pipeline was like: scan the patient’s teeth using an intraoral scanner, reconstruct the surface mesh, design the restoration digitally, and finally mill the crown out of ceramic. We were working mostly with point clouds and meshes, but it wasn’t just math, it was craftsmanship translated into a digital process. Every micron mattered. You could literally see how a good algorithm meant a better fit in someone’s mouth. Gaussian Splatting isn’t about surface reconstruction, it’s about appearance reconstruction. It doesn’t care about explicit topology, it captures how light interacts with the scene. In a sense, it’s the opposite philosophy of the dental world: instead of modeling what the object is, it models how the object looks. 3D Gaussian Splatting enables applications like training self driving cars, teaching robots to understand their environment, creating virtual worlds, or monitoring real sites. It represents scenes as millions of small Gaussians rendered in real time without the need for meshes or textures. Coming from a world where precision geometry was everything, this shift felt natural. It’s still about reconstruction, but with a different goal: not manufacturing a perfect object, but reproducing how the world actually looks. Two weeks ago I got my first dental crown, made with the same software, reconstruction algorithms, and Swiss precision I once helped develop. I haven’t worked there in two years, but sitting in that chair and seeing the process from the other side was a proud moment. It reminded me why I love this field.

MrNeRF

290,319 просмотров • 11 месяцев назад

A dead fruit fly was sliced into 66 pieces. Now its 166,700 neurons are driving robots. Nobody trained it, no dataset, no reward function, no GPU farm, just the wiring. The fly took a second to die, but copying it took 7 microscopes running nonstop for a year, shaving it 1/10,000 of a hair at a time and photographing every layer before the next cut. AI rebuilt it from the scraps, eyes to legs with the neck still attached, 124M synapses and 11,710 cell types, 44 years of human labor finished by machines. Scientists first tried this in 2008 and guessed it would take 500 people 10 years. Then the internet got the file and someone gave it a body. Camera → fly eyes → 166,700 spiking neurons → descending neurons → motors. One of those neurons is DNp01, the giant fiber, the cell that fires when your hand comes down. Push an object at the camera and the looming detectors light up, so the robot swerves. It took 0 training runs, just the same reflex that gets a fly out from under a swatter in 100 milliseconds. Then came the control test, where they kept all 138,639 neurons and shuffled where the synapses connect. The behavior died. The intelligence was never in the neurons, it was in the map. The fly is gone, but the flinch is still running.

A dead fruit fly was sliced into 66 pieces. Now its 166,700 neurons are driving robots. Nobody trained it, no dataset, no reward function, no GPU farm, just the wiring. The fly took a second to die, but copying it took 7 microscopes running nonstop for a year, shaving it 1/10,000 of a hair at a time and photographing every layer before the next cut. AI rebuilt it from the scraps, eyes to legs with the neck still attached, 124M synapses and 11,710 cell types, 44 years of human labor finished by machines. Scientists first tried this in 2008 and guessed it would take 500 people 10 years. Then the internet got the file and someone gave it a body. Camera → fly eyes → 166,700 spiking neurons → descending neurons → motors. One of those neurons is DNp01, the giant fiber, the cell that fires when your hand comes down. Push an object at the camera and the looming detectors light up, so the robot swerves. It took 0 training runs, just the same reflex that gets a fly out from under a swatter in 100 milliseconds. Then came the control test, where they kept all 138,639 neurons and shuffled where the synapses connect. The behavior died. The intelligence was never in the neurons, it was in the map. The fly is gone, but the flinch is still running.

Spike 1%

25,494 просмотров • 1 месяц назад

$this might be the E2B killer for AI agent sandboxes. forkd is an open-source microVM sandbox runtime built on Firecracker, made for AI agent fan-out, code interpreters, eval harnesses, anything that spins up a lot of short-lived sandboxes. before: every sandbox cold-boots its own VM and re-imports the whole runtime from scratch, numpy, torch, JIT compilation, model weights, all of it. now: forkd boots one parent VM once, warms it with your runtime already imported, then forks children from that snapshot using copy-on-write memory. the repo's own benchmark: spawning 100 sandboxes takes 101ms with forkd, versus 759ms for a raw Firecracker cold-boot, and well over a minute for Docker or gVisor. the SDK is a literal drop-in for E2B's Python client, so if you're already running code-interpreter agents on it, swapping the import line gets you a self-hosted runtime with the same isolation model at a fraction of the per-sandbox cost. pre-built recipes ship for e2b-style code interpreters, Jupyter kernels, SWE-bench coding agents, and Playwright browser fan-out$

this might be the E2B killer for AI agent sandboxes. forkd is an open-source microVM sandbox runtime built on Firecracker, made for AI agent fan-out, code interpreters, eval harnesses, anything that spins up a lot of short-lived sandboxes. before: every sandbox cold-boots its own VM and re-imports the whole runtime from scratch, numpy, torch, JIT compilation, model weights, all of it. now: forkd boots one parent VM once, warms it with your runtime already imported, then forks children from that snapshot using copy-on-write memory. the repo's own benchmark: spawning 100 sandboxes takes 101ms with forkd, versus 759ms for a raw Firecracker cold-boot, and well over a minute for Docker or gVisor. the SDK is a literal drop-in for E2B's Python client, so if you're already running code-interpreter agents on it, swapping the import line gets you a self-hosted runtime with the same isolation model at a fraction of the per-sandbox cost. pre-built recipes ship for e2b-style code interpreters, Jupyter kernels, SWE-bench coding agents, and Playwright browser fan-out

Oliver Prompts

20,005 просмотров • 2 месяцев назад

The architecture of this new world model is one of the most interesting things I've seen lately: Let me first explain how most world models work: They predict and render one frame at a time. If you are navigating in one of these worlds, and you look left, the model draws whatever looks right in the moment. Every time you change your viewpoint, the model has to imagine what should be there again, so it's very common for these models to "forget" what's in the world. For example, if you put a toy on the table, look away, then look back, the toy might not be there anymore. Tripo AI is releasing its Project Eden model, which works very differently: The model builds the world first, and then renders it based on that map. That map holds the real state of the world: the geometry, every object, where things are, what's already happened. The picture you see on screen gets generated from the map. This architecture flips the whole thing. Now, you get the following: 1. The world stops forgetting. Leave, come back, and the toy is still on the table because it lives in the map, not in the last frame you saw. 2. You can edit the world, and those changes persist for anyone who enters later. 3. Multiple people and AI agents can coexist in the world and see it from different perspectives. This is early research, but it's looking really promising. They just raised nearly $200M across two rounds to build it out. Tripo will be at SIGGRAPH 2026 (July 19–23, Los Angeles Convention Center). If you work in 3D, embodied AI, simulation, or anything spatial, go connect with them there.

The architecture of this new world model is one of the most interesting things I've seen lately: Let me first explain how most world models work: They predict and render one frame at a time. If you are navigating in one of these worlds, and you look left, the model draws whatever looks right in the moment. Every time you change your viewpoint, the model has to imagine what should be there again, so it's very common for these models to "forget" what's in the world. For example, if you put a toy on the table, look away, then look back, the toy might not be there anymore. Tripo AI is releasing its Project Eden model, which works very differently: The model builds the world first, and then renders it based on that map. That map holds the real state of the world: the geometry, every object, where things are, what's already happened. The picture you see on screen gets generated from the map. This architecture flips the whole thing. Now, you get the following: 1. The world stops forgetting. Leave, come back, and the toy is still on the table because it lives in the map, not in the last frame you saw. 2. You can edit the world, and those changes persist for anyone who enters later. 3. Multiple people and AI agents can coexist in the world and see it from different perspectives. This is early research, but it's looking really promising. They just raised nearly $200M across two rounds to build it out. Tripo will be at SIGGRAPH 2026 (July 19–23, Los Angeles Convention Center). If you work in 3D, embodied AI, simulation, or anything spatial, go connect with them there.

Santiago

30,244 просмотров • 4 месяцев назад

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

Jonathan Stephens

17,712 просмотров • 9 месяцев назад

llama.cpp isn't just for text LLMs anymore. Pure C++ zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!

llama.cpp isn't just for text LLMs anymore. Pure C++ zero shot voice cloning just officially landed in mainline. Text generation was only step one. If you’re building autonomous local AI agents, real time voice assistants, or edge workflows, instant low latency audio is the missing piece. Thanks to PR #26254, Alibaba’s state of the art Qwen3 TTS model family is now natively supported directly inside the llama.cpp repository under the multimodal (mtmd) framework. No Python bloat. No massive PyTorch CUDA overhead. Just raw, hyper optimized C++ running GGUF voice weights. Here is why this native update is a massive deal for the open source local AI stack: # Multimodal Architecture (.gguf + mmproj) Qwen3-TTS splits the workload between the base language model backbone and a multimodal projection adapter. llama.cpp handles this using the llama-tts binary, mapping the text model alongside its --mmproj projector to process audio tokens seamlessly. # Zero Shot Voice Cloning in Seconds You don't need fine tuning or massive dataset training. Feed the C++ engine a single 5 to 10 second .wav audio sample using the --tts-speaker-file flag, and it accurately clones the exact timbre, tone, and accent on the fly. # Real World T4 GPU Benchmark & Resource FootprintRunning the 1.7B Base model in 8-bit quantization (Q8_0): - VRAM Footprint: ~7 GB peak VRAM during active zero-shot cloning. - Audio Quality: Studio grade, natural-sounding voice output in seconds. • - Execution: Direct execution via native compiled binaries or sub process calls. # Coming Next to llama-server (PR #26603) Beyond CLI execution, a native POST /tts HTTP endpoint is currently being added to llama-server, which will soon allow you to trigger voice generation directly via standard REST API requests! # quick note on Colab compilation: Because this code was merged into mainline very recently, pre-built third-party binaries haven't fully caught up yet. Compiling llama-tts directly from source on Google Colab's free CPU instance can take about 1 hour (or ~1-2 minutes if targeting single GPU arch like -DCMAKE_CUDA_ARCHITECTURES=75). Be patient during the build step, or compile it locally on your own rig for instant execution! To test this out yourself, I built a zero config Google Colab notebook that compiles llama.cpp, downloads the Q8_0 GGUF files from HuggingFace, and spins up an interactive Gradio Studio UI so you can record/upload 3 second clips and clone voices in real time. Stop sleeping on native C++ audio. The era of bulky Python audio pipelines is officially over. Links to the free Google Colab notebook and the official ggml org GGUF HuggingFace model repository are in the replies below! available in q4 and q8 both variants, 1 GB and 1.85 GBs respectively (requires additional ~500MB mmproj gguf) Are you building local voice agents yet? What does your current audio stack look like? Drop your setups below!

Alok

47,881 просмотров • 2 месяцев назад

A good technical LLM interview question: Your RAG chatbot is working as expected locally. You deploy it behind a load balancer with 3 replicas. Users report that it forgets what they just asked, and answers get worse with each restart. Why did this happen? (answer below) A local setup has one process that owns everything. - The vector index is a variable in memory. - Conversation history is a Python list. - The documents are on local disk. You never treat any of them as infrastructure, because restarting rebuilds all three in seconds and there is only ever one copy. The setup does not carry over to production directly. The vector index might disappear on restart, so the app re-embeds everything on boot and serves empty results until it finishes. Conversation history may belong to one replica, so a follow-up routed elsewhere has no memory of the previous turn. Documents could be on whichever container ingested them, so the three replicas hold three different corpora. None of this is evident with one user and one process. So the actual work in shipping RAG is not just the retrieval logic, but also storing the vector index, the conversation history, and the documents outside the app, where every replica reads and writes the same copy. Which comes down to three requirements: > The vector store needs persistence and has to be reachable from every replica. pgvector inside Postgres keeps embeddings next to the rest of the data instead of adding another system to operate. > Conversation state has to be checkpointed outside the app. LangGraph writes its state to Postgres, so any replica can pick up a thread mid-conversation. > Docs need shared object storage, so ingestion happens once instead of once per replica. If you get those three right, the retrieval logic you wrote in the notebook works unchanged. To learn how all of it is wired together, Akamai's GitHub has a working reference implementation. - rag-langgraph-k8s-quickstart is an airline policy Q&A assistant built with FastAPI, LangChain, and LangGraph. Terraform provisions the LKE cluster, a Postgres instance with pgvector for embeddings, a second Postgres for LangGraph checkpointing, and an object storage bucket for the policy documents, in one apply. - akamai-workshop-ai-inference covers the next step, running the model yourself instead of calling an API, with prefill and decode, KV cache tradeoffs, and continuous batching under real concurrency. Both are available on Akamai's new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post assumes the retrieval logic was right to begin with, and that is doing a lot of work. Most RAG systems fail earlier, at the point where a chunk gets treated as a self-contained unit of meaning. I wrote about the two skills that fix that gap, and why the chunk is usually the wrong thing to embed. Read it below. Thanks to Akamai Cloud for partnering today!

A good technical LLM interview question: Your RAG chatbot is working as expected locally. You deploy it behind a load balancer with 3 replicas. Users report that it forgets what they just asked, and answers get worse with each restart. Why did this happen? (answer below) A local setup has one process that owns everything. - The vector index is a variable in memory. - Conversation history is a Python list. - The documents are on local disk. You never treat any of them as infrastructure, because restarting rebuilds all three in seconds and there is only ever one copy. The setup does not carry over to production directly. The vector index might disappear on restart, so the app re-embeds everything on boot and serves empty results until it finishes. Conversation history may belong to one replica, so a follow-up routed elsewhere has no memory of the previous turn. Documents could be on whichever container ingested them, so the three replicas hold three different corpora. None of this is evident with one user and one process. So the actual work in shipping RAG is not just the retrieval logic, but also storing the vector index, the conversation history, and the documents outside the app, where every replica reads and writes the same copy. Which comes down to three requirements: > The vector store needs persistence and has to be reachable from every replica. pgvector inside Postgres keeps embeddings next to the rest of the data instead of adding another system to operate. > Conversation state has to be checkpointed outside the app. LangGraph writes its state to Postgres, so any replica can pick up a thread mid-conversation. > Docs need shared object storage, so ingestion happens once instead of once per replica. If you get those three right, the retrieval logic you wrote in the notebook works unchanged. To learn how all of it is wired together, Akamai's GitHub has a working reference implementation. - rag-langgraph-k8s-quickstart is an airline policy Q&A assistant built with FastAPI, LangChain, and LangGraph. Terraform provisions the LKE cluster, a Postgres instance with pgvector for embeddings, a second Postgres for LangGraph checkpointing, and an object storage bucket for the policy documents, in one apply. - akamai-workshop-ai-inference covers the next step, running the model yourself instead of calling an API, with prefill and decode, KV cache tradeoffs, and continuous batching under real concurrency. Both are available on Akamai's new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post assumes the retrieval logic was right to begin with, and that is doing a lot of work. Most RAG systems fail earlier, at the point where a chunk gets treated as a self-contained unit of meaning. I wrote about the two skills that fix that gap, and why the chunk is usually the wrong thing to embed. Read it below. Thanks to Akamai Cloud for partnering today!

Akshay 🚀

31,971 просмотров • 1 месяц назад

I think one of the biggest problems with AI agents right now is how much control we give up just to use them. Most AI agent platforms want you to work inside their ecosystem, use their preferred models, and trust them with your files and data. That might be convenient, but I don't think it should be the only option. I recently came across PilotDeck, and I found its approach pretty interesting. It's an open-source AI agent workspace developed by Tsinghua University's THUNLP, ModelBest, OpenBMB, and AI9Stars. The idea is simple: instead of being tied to one AI provider, you should have more freedom to decide how your agent works. And there are a few things worth looking at here. First, model independence. Most people focus on which AI model is the smartest, but I think having the freedom to choose your model is equally important. PilotDeck supports different model providers and configurable API endpoints. This means you're not necessarily stuck with one company's models. You can choose a setup based on your budget, performance needs, or the type of task you're working on. For developers who regularly experiment with different AI models, I think that's a pretty useful approach. Second, privacy and control over your workspace. This is probably the part that interests me the most. AI agents are increasingly being used to handle research papers, financial documents, spreadsheets, presentations, and other personal or business files. And once you start giving an agent access to that kind of information, where your files are processed becomes an important question. PilotDeck supports local workspaces and offers options for local or private deployment. That gives users more flexibility over how their environment is configured, rather than relying entirely on a closed platform. Of course, privacy still depends on which model provider and API you connect. Using a local workspace doesn't automatically mean your data never leaves your device. But having the option to control the setup is a meaningful difference. Third, it's designed to actually complete work. Another thing I like about the approach is that PilotDeck isn't just trying to be another chatbot. It's built around an agent workflow where you describe a task, the system plans the steps, works with files, uses tools, and produces an output you can still edit. For example, imagine giving an AI agent multiple research documents and asking it to organize the information into an Excel spreadsheet and prepare a PowerPoint presentation. Instead of getting a long text response that you need to manually copy into different applications, the goal is to have the agent handle the workflow and deliver usable files. It also supports task history, result previews, switching models, and changing instructions while a task is running. I think this is where AI agents become more practical for everyday work. And finally, there's the open-source aspect. Personally, I think we're going to see more demand for AI tools that don't force users into one vendor's ecosystem. Models are improving quickly, and the best option today might not be the best option six months from now. Having an agent workspace that can work with different models makes a lot more sense to me than rebuilding your entire workflow every time you want to switch providers. There are still things I'd want to test properly, especially how reliable it is on longer tasks, how easy the setup is for non-technical users, and what the actual privacy trade-offs are with different model configurations. But I like the direction PilotDeck is taking. I don't think the future of AI agents should be about which company controls your entire workflow. It should be about how much control you have over the tools doing the work. And that's why I think open-source, model-independent projects like PilotDeck are worth paying attention to. If you're interested in building AI workflows with more flexibility, check it out. Website: GitHub:

I think one of the biggest problems with AI agents right now is how much control we give up just to use them. Most AI agent platforms want you to work inside their ecosystem, use their preferred models, and trust them with your files and data. That might be convenient, but I don't think it should be the only option. I recently came across PilotDeck, and I found its approach pretty interesting. It's an open-source AI agent workspace developed by Tsinghua University's THUNLP, ModelBest, OpenBMB, and AI9Stars. The idea is simple: instead of being tied to one AI provider, you should have more freedom to decide how your agent works. And there are a few things worth looking at here. First, model independence. Most people focus on which AI model is the smartest, but I think having the freedom to choose your model is equally important. PilotDeck supports different model providers and configurable API endpoints. This means you're not necessarily stuck with one company's models. You can choose a setup based on your budget, performance needs, or the type of task you're working on. For developers who regularly experiment with different AI models, I think that's a pretty useful approach. Second, privacy and control over your workspace. This is probably the part that interests me the most. AI agents are increasingly being used to handle research papers, financial documents, spreadsheets, presentations, and other personal or business files. And once you start giving an agent access to that kind of information, where your files are processed becomes an important question. PilotDeck supports local workspaces and offers options for local or private deployment. That gives users more flexibility over how their environment is configured, rather than relying entirely on a closed platform. Of course, privacy still depends on which model provider and API you connect. Using a local workspace doesn't automatically mean your data never leaves your device. But having the option to control the setup is a meaningful difference. Third, it's designed to actually complete work. Another thing I like about the approach is that PilotDeck isn't just trying to be another chatbot. It's built around an agent workflow where you describe a task, the system plans the steps, works with files, uses tools, and produces an output you can still edit. For example, imagine giving an AI agent multiple research documents and asking it to organize the information into an Excel spreadsheet and prepare a PowerPoint presentation. Instead of getting a long text response that you need to manually copy into different applications, the goal is to have the agent handle the workflow and deliver usable files. It also supports task history, result previews, switching models, and changing instructions while a task is running. I think this is where AI agents become more practical for everyday work. And finally, there's the open-source aspect. Personally, I think we're going to see more demand for AI tools that don't force users into one vendor's ecosystem. Models are improving quickly, and the best option today might not be the best option six months from now. Having an agent workspace that can work with different models makes a lot more sense to me than rebuilding your entire workflow every time you want to switch providers. There are still things I'd want to test properly, especially how reliable it is on longer tasks, how easy the setup is for non-technical users, and what the actual privacy trade-offs are with different model configurations. But I like the direction PilotDeck is taking. I don't think the future of AI agents should be about which company controls your entire workflow. It should be about how much control you have over the tools doing the work. And that's why I think open-source, model-independent projects like PilotDeck are worth paying attention to. If you're interested in building AI workflows with more flexibility, check it out. Website: GitHub:

Sally Stockholm

13,794 просмотров • 11 дней назад