Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

🚀 Major Update for ML-Sharp WebUI on Pinokio! 🌟 Native Mac Video Export (Zero CUDA) 🎥 10 New Camera Trajectories 💾 Low VRAM Mode (8GB GPU Support) ⚡️ Real-Time Previews Create stunning 3D gaussian splats from a single image.

franzipol's profile picture

franzipol

3,209 subscribers

22,749 görüntüleme • 6 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

🚀 Just created something completely new! I Rewrote Apple's Sharp to generate 3D Gaussian Splats from a single 360° equirectangular image made from FLUX 2 (Flex) One equirectangular image → Full spherical 3DGS world! ✨ New web interface - drag & drop your pano, select quality, download PLY ⚡ 2-3 sec on a older laptop Quality tiers: ▫️ Low: 500K splats ▫️ Medium: 2M splats ▫️ High: 4.7M splats ▫️ Ultra: 8.4M splats The pipeline: 360° depth estimation from equirectangular input Spherical projection maps pixels → 3D points Each point becomes a Gaussian with depth-scaled radius - Export to standard PLY for any 3DGS viewer (Unity) No multi-view capture. No COLMAP. No NeRF training. Just upload a pano, pick quality, get your splats. #3DGS #GaussianSplatting #ComputerVision #AI #madewithunity #Splats Sharp Monocular View Synthesis in Less Than a Second
0:27

🚀 Just created something completely new! I Rewrote Apple's Sharp to generate 3D Gaussian Splats from a single 360° equirectangular image made from FLUX 2 (Flex) One equirectangular image → Full spherical 3DGS world! ✨ New web interface - drag & drop your pano, select quality, download PLY ⚡ 2-3 sec on a older laptop Quality tiers: ▫️ Low: 500K splats ▫️ Medium: 2M splats ▫️ High: 4.7M splats ▫️ Ultra: 8.4M splats The pipeline: 360° depth estimation from equirectangular input Spherical projection maps pixels → 3D points Each point becomes a Gaussian with depth-scaled radius - Export to standard PLY for any 3DGS viewer (Unity) No multi-view capture. No COLMAP. No NeRF training. Just upload a pano, pick quality, get your splats. #3DGS #GaussianSplatting #ComputerVision #AI #madewithunity #Splats Sharp Monocular View Synthesis in Less Than a Second

Daniel Skaale

73,120 görüntüleme • 7 ay önce

I really want to show off the cutting edge tech the team is building and how we're turning AI image generation into real living environments using gaussian splats and at the :19 mark of this video you can see how we're blending splats together to create a real world. What this looks like from a gaming perspective is this all happens behind the players view to simulate a real world environment. As you move through the space, distance becomes a variable to consider on how images update and are experienced. We are creating a new reality in real time.
1:21

I really want to show off the cutting edge tech the team is building and how we're turning AI image generation into real living environments using gaussian splats and at the :19 mark of this video you can see how we're blending splats together to create a real world. What this looks like from a gaming perspective is this all happens behind the players view to simulate a real world environment. As you move through the space, distance becomes a variable to consider on how images update and are experienced. We are creating a new reality in real time.

𝔹𝕃𝕍ℂ𝕂𝕃!𝔾ℍ𝕋

25,084 görüntüleme • 1 ay önce

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.
0:20

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

60,866 görüntüleme • 1 ay önce

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!
0:30

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

95,991 görüntüleme • 7 ay önce

QVAC SDK 0.13.0 is live, and this version brings a lot of exciting updates! Local AI now plugs into your coding agent, ships as a desktop app in one command, and runs even more models. Highlights: NEW INTEGRATIONS - OpenCode and coding agents: the new @qvac/ai-sdk-provider makes QVAC a local provider. Less setup, same-model requests queue cleanly, and managed mode starts and supervises qvac serve for you. - Broader OpenAI-compatible API, validated across supported flows so covered capabilities stay consistent and testable. - Turn your QVAC project into a real desktop app for Mac, Windows, or Linux with a single command. The new Electron plugin handles the packaging and keeps the app small by including only what it needs. NEW MODELS - New pi0.5 model support - run a vision-language "robot brain" on a single ordinary graphics card, at full accuracy. - Image-to-video, fully local, via the Wan2.1 model in the Diffusion addon. - New BCI add-on: brain-computer interface transcription, fully local. Decode recorded neural signals into text on-device via the Whisper.cpp-based BCI model. IMPROVEMENTS - Whisper GPU transcription on Android, auto-picking the best backend (OpenCL on Adreno 700+, Vulkan elsewhere), unified on one ggml engine. - Parakeet steadier on mobile, with real end-of-utterance detection for streaming. - Supertonic TTS now runs full GPU across Metal, Vulkan, and OpenCL, with native streaming.
1:39

QVAC SDK 0.13.0 is live, and this version brings a lot of exciting updates! Local AI now plugs into your coding agent, ships as a desktop app in one command, and runs even more models. Highlights: NEW INTEGRATIONS - OpenCode and coding agents: the new @qvac/ai-sdk-provider makes QVAC a local provider. Less setup, same-model requests queue cleanly, and managed mode starts and supervises qvac serve for you. - Broader OpenAI-compatible API, validated across supported flows so covered capabilities stay consistent and testable. - Turn your QVAC project into a real desktop app for Mac, Windows, or Linux with a single command. The new Electron plugin handles the packaging and keeps the app small by including only what it needs. NEW MODELS - New pi0.5 model support - run a vision-language "robot brain" on a single ordinary graphics card, at full accuracy. - Image-to-video, fully local, via the Wan2.1 model in the Diffusion addon. - New BCI add-on: brain-computer interface transcription, fully local. Decode recorded neural signals into text on-device via the Whisper.cpp-based BCI model. IMPROVEMENTS - Whisper GPU transcription on Android, auto-picking the best backend (OpenCL on Adreno 700+, Vulkan elsewhere), unified on one ggml engine. - Parakeet steadier on mobile, with real end-of-utterance detection for streaming. - Supertonic TTS now runs full GPU across Metal, Vulkan, and OpenCL, with native streaming.

QVAC

20,922,918 görüntüleme • 1 ay önce

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:
1:07

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

SpAItial AI

176,105 görüntüleme • 7 ay önce

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito
0:38

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,763 görüntüleme • 1 yıl önce

3D scanning and rendering is moving so fast - got my splats up and running and I'm mind blown getting ~100fps for this complex 3D scene ⬇️ 🤯 1. WAY faster than NeRF: For comparison, NeRFs would takes around 10 seconds per frame (!) Instead I'm zipping around with FPV controls without breaking a sweat - though I do crash a few times towards the end of the video lol 2. Old Meets New: Gaussian Splatting is cool in that it fuses classical graphics and deep learning techniques. Like NeRFs, this is still a radiance field - just without the slower (ne)ural rendering part. 3. Explicit Representation: Instead you represent a 3D scene as a collection of ellipsoidal "splats" called gaussians. Each gaussian has a position, size, and color. Rendering in real-time is done by projecting into the image plane and alpha blending. 4. Photorealistic Effects: Gaussian splatting use spherical harmonics to represent the view-dependent effects and lighting - allowing surfaces to change color when viewed from different angles, enabling greater photorealism. It doesn't use a neural network, but the training loop is similar to deep learning. 5. Enables Direct Editing: But it's not just speed - with Gaussian Splatting you also get 3D editing support! So you can select, move, and delete stuff, even relight stuff. This type of editing has been more tedious to do with NeRFs and their implicit black box representations. 📲 More tests cooking! Much more to unpack here including simpler explanations. If you enjoyed this post, you might enjoy my feed: Bilawal Sidhu
0:35

3D scanning and rendering is moving so fast - got my splats up and running and I'm mind blown getting ~100fps for this complex 3D scene ⬇️ 🤯 1. WAY faster than NeRF: For comparison, NeRFs would takes around 10 seconds per frame (!) Instead I'm zipping around with FPV controls without breaking a sweat - though I do crash a few times towards the end of the video lol 2. Old Meets New: Gaussian Splatting is cool in that it fuses classical graphics and deep learning techniques. Like NeRFs, this is still a radiance field - just without the slower (ne)ural rendering part. 3. Explicit Representation: Instead you represent a 3D scene as a collection of ellipsoidal "splats" called gaussians. Each gaussian has a position, size, and color. Rendering in real-time is done by projecting into the image plane and alpha blending. 4. Photorealistic Effects: Gaussian splatting use spherical harmonics to represent the view-dependent effects and lighting - allowing surfaces to change color when viewed from different angles, enabling greater photorealism. It doesn't use a neural network, but the training loop is similar to deep learning. 5. Enables Direct Editing: But it's not just speed - with Gaussian Splatting you also get 3D editing support! So you can select, move, and delete stuff, even relight stuff. This type of editing has been more tedious to do with NeRFs and their implicit black box representations. 📲 More tests cooking! Much more to unpack here including simpler explanations. If you enjoyed this post, you might enjoy my feed: Bilawal Sidhu

Bilawal Sidhu

337,090 görüntüleme • 2 yıl önce

🚀 Introducing the Kling 3.0 Model: Everyone a Director. It’s Time. An all-in-one creative engine that enables truly native multimodal creation. - Superb Consistency: Your characters and elements, always locked in. - Flexible Video Production: Create 15s clips with precise control, excellent video realism and customizable multi-shots. - Upgraded Native Audio: Now supports multi-character reference, more languages and accents. - Enhanced Image Generation: 4K Image output, new image series mode and more cinematic visuals. Ultra subscribers get exclusive early access — now live on Web -> Kling AI. Get ready to unlock your creative potential with our most advanced model yet. [For the next 24 hours ONLY] Follow, Comment & Retweet to win exclusive early access - we will select 10 winners in this post from the top ten 'Most liked replies'. So which feature are you most excited about? Let us know in the comment! We will contact you via your DM.
1:54

🚀 Introducing the Kling 3.0 Model: Everyone a Director. It’s Time. An all-in-one creative engine that enables truly native multimodal creation. - Superb Consistency: Your characters and elements, always locked in. - Flexible Video Production: Create 15s clips with precise control, excellent video realism and customizable multi-shots. - Upgraded Native Audio: Now supports multi-character reference, more languages and accents. - Enhanced Image Generation: 4K Image output, new image series mode and more cinematic visuals. Ultra subscribers get exclusive early access — now live on Web -> Kling AI. Get ready to unlock your creative potential with our most advanced model yet. [For the next 24 hours ONLY] Follow, Comment & Retweet to win exclusive early access - we will select 10 winners in this post from the top ten 'Most liked replies'. So which feature are you most excited about? Let us know in the comment! We will contact you via your DM.

Kling AI

1,195,432 görüntüleme • 5 ay önce

We have SLAM on the Robocap! 🎉 Visualized with Rerun Using NVIDIA AI Developer cuVSLAM for GPU-accelerated multicamera tracking. I basically wrote zero code myself and fully used Claude Code for this. It worked because I had so many existing examples to point to that it just wrote everything the way I would have. A few technical wins: 1. Used rattler-build from prefix.dev to package the compiled cuVSLAM CUDA binaries, which made it SUPER easy to use across repos. This also means it works on the DGX Spark (ARM64) out of the box. 2. Zero-setup experience: git clone && pixi run track-robocap auto-downloads a 100MB dataset from HuggingFace and tracks frames. 3. Real-time 3D visualization with trajectories, landmarks, pose graphs, and video playback in Rerun. Still visual-only (not visual-inertial yet), and loop closure needs some debugging. Next steps are getting this into a Gradio interface, then into daggr, and extending it to work with other datasets from exoego-forge. The last piece I'm excited about: Rerun's RRD files now support layers for incremental data. Planning to build pipelines that go from raw sensor data → slam -> human pose → depth estimation → etc. Repo here:
0:21

We have SLAM on the Robocap! 🎉 Visualized with Rerun Using NVIDIA AI Developer cuVSLAM for GPU-accelerated multicamera tracking. I basically wrote zero code myself and fully used Claude Code for this. It worked because I had so many existing examples to point to that it just wrote everything the way I would have. A few technical wins: 1. Used rattler-build from prefix.dev to package the compiled cuVSLAM CUDA binaries, which made it SUPER easy to use across repos. This also means it works on the DGX Spark (ARM64) out of the box. 2. Zero-setup experience: git clone && pixi run track-robocap auto-downloads a 100MB dataset from HuggingFace and tracks frames. 3. Real-time 3D visualization with trajectories, landmarks, pose graphs, and video playback in Rerun. Still visual-only (not visual-inertial yet), and loop closure needs some debugging. Next steps are getting this into a Gradio interface, then into daggr, and extending it to work with other datasets from exoego-forge. The last piece I'm excited about: Rerun's RRD files now support layers for incremental data. Planning to build pipelines that go from raw sensor data → slam -> human pose → depth estimation → etc. Repo here:

Pablo Vela

50,805 görüntüleme • 4 ay önce

𝗗𝗼𝗿𝗺𝗮𝗻𝘁 𝗠𝗲𝗺𝗼𝗿𝘆 𝗩𝗶𝗲𝘄𝗲𝗿: Because I almost always have some gear with me—a drone, a mirrorless camera, or at least a phone—I can capture scenes on the spur of the moment. Here I am freely flying through 3D Gaussian Splats scenes trained from these impromptu captures. It has been several years now since I've developed a real passion for 3D scanning, photogrammetry, and—for the last two and a half years—radiance fields. We actively use these technologies on a professional level at Dpt. , but this is also a personal passion of mine. I love capturing scenes, places, people, moments in time—memories. I enjoy the process of doing it; yes, capturing several hundreds of photos at a time, going in circles around a 'subject' for 15 minutes—all of which can drive my family nuts when they're with me. I have fun pre-processing, aligning, and using datasets to finally see those memories form and then be able to explore them freely in real-time. This week, Meta announced Hyperscape during this year's Meta Connect event… "𝘈𝘯𝘥 𝘴𝘱𝘦𝘢𝘬𝘪𝘯𝘨 𝘰𝘧 𝘱𝘳𝘦𝘴𝘦𝘯𝘤𝘦, 𝘸𝘦’𝘳𝘦 𝘪𝘮𝘱𝘳𝘰𝘷𝘪𝘯𝘨 𝘵𝘩𝘢𝘵, 𝘵𝘰𝘰. 𝘞𝘦’𝘳𝘦 𝘸𝘰𝘳𝘬𝘪𝘯𝘨 𝘵𝘰 𝘣𝘳𝘪𝘯𝘨 𝘱𝘩𝘰𝘵𝘰𝘳𝘦𝘢𝘭𝘪𝘴𝘵𝘪𝘤 𝘴𝘱𝘢𝘤𝘦𝘴 𝘪𝘯𝘵𝘰 𝘵𝘩𝘦 𝘮𝘦𝘵𝘢𝘷𝘦𝘳𝘴𝘦, 𝘦𝘯𝘢𝘣𝘭𝘪𝘯𝘨 𝘢 𝘱𝘳𝘰𝘧𝘰𝘶𝘯𝘥 𝘯𝘦𝘸 𝘸𝘢𝘺 𝘵𝘰 𝘣𝘦 𝘵𝘰𝘨𝘦𝘵𝘩𝘦𝘳 𝘪𝘯 𝘴𝘱𝘢𝘤𝘦𝘴 𝘵𝘩𝘢𝘵 𝘭𝘰𝘰𝘬 𝘢𝘯𝘥 𝘧𝘦𝘦𝘭 𝘭𝘪𝘬𝘦 𝘺𝘰𝘶’𝘳𝘦 𝘱𝘩𝘺𝘴𝘪𝘤𝘢𝘭𝘭𝘺 𝘵𝘩𝘦𝘳𝘦—𝘸𝘦 𝘤𝘢𝘭𝘭 𝘪𝘵 𝘏𝘺𝘱𝘦𝘳𝘴𝘤𝘢𝘱𝘦." …it's exciting to see major players like Meta pushing the boundaries of immersive experiences. Their announcement of Hyperscape resonates deeply with the work I've been doing and my passion for capturing and recreating photorealistic environments and memories. It feels like we're all moving toward a future where the lines between the physical and digital worlds blur, allowing us to share spaces and moments in unprecedented ways. I'm thrilled to be contributing my own explorations to this ever-evolving field. The attached video is a screen recording of myself exploring and flying through volumetric captures—dormant memories—in real-time using an app I crafted for this purpose. The first part shows me moving from one memory to another (btw, this works in tethered VR), and in the second part of the video at , I explore two memories simultaneously, rendering two 3D Gaussian Splats of the same location onto two monitors. #gaussiansplatting #radiancefield #digitaltwin #hyperscape #memories
2:51

𝗗𝗼𝗿𝗺𝗮𝗻𝘁 𝗠𝗲𝗺𝗼𝗿𝘆 𝗩𝗶𝗲𝘄𝗲𝗿: Because I almost always have some gear with me—a drone, a mirrorless camera, or at least a phone—I can capture scenes on the spur of the moment. Here I am freely flying through 3D Gaussian Splats scenes trained from these impromptu captures. It has been several years now since I've developed a real passion for 3D scanning, photogrammetry, and—for the last two and a half years—radiance fields. We actively use these technologies on a professional level at Dpt. , but this is also a personal passion of mine. I love capturing scenes, places, people, moments in time—memories. I enjoy the process of doing it; yes, capturing several hundreds of photos at a time, going in circles around a 'subject' for 15 minutes—all of which can drive my family nuts when they're with me. I have fun pre-processing, aligning, and using datasets to finally see those memories form and then be able to explore them freely in real-time. This week, Meta announced Hyperscape during this year's Meta Connect event… "𝘈𝘯𝘥 𝘴𝘱𝘦𝘢𝘬𝘪𝘯𝘨 𝘰𝘧 𝘱𝘳𝘦𝘴𝘦𝘯𝘤𝘦, 𝘸𝘦’𝘳𝘦 𝘪𝘮𝘱𝘳𝘰𝘷𝘪𝘯𝘨 𝘵𝘩𝘢𝘵, 𝘵𝘰𝘰. 𝘞𝘦’𝘳𝘦 𝘸𝘰𝘳𝘬𝘪𝘯𝘨 𝘵𝘰 𝘣𝘳𝘪𝘯𝘨 𝘱𝘩𝘰𝘵𝘰𝘳𝘦𝘢𝘭𝘪𝘴𝘵𝘪𝘤 𝘴𝘱𝘢𝘤𝘦𝘴 𝘪𝘯𝘵𝘰 𝘵𝘩𝘦 𝘮𝘦𝘵𝘢𝘷𝘦𝘳𝘴𝘦, 𝘦𝘯𝘢𝘣𝘭𝘪𝘯𝘨 𝘢 𝘱𝘳𝘰𝘧𝘰𝘶𝘯𝘥 𝘯𝘦𝘸 𝘸𝘢𝘺 𝘵𝘰 𝘣𝘦 𝘵𝘰𝘨𝘦𝘵𝘩𝘦𝘳 𝘪𝘯 𝘴𝘱𝘢𝘤𝘦𝘴 𝘵𝘩𝘢𝘵 𝘭𝘰𝘰𝘬 𝘢𝘯𝘥 𝘧𝘦𝘦𝘭 𝘭𝘪𝘬𝘦 𝘺𝘰𝘶’𝘳𝘦 𝘱𝘩𝘺𝘴𝘪𝘤𝘢𝘭𝘭𝘺 𝘵𝘩𝘦𝘳𝘦—𝘸𝘦 𝘤𝘢𝘭𝘭 𝘪𝘵 𝘏𝘺𝘱𝘦𝘳𝘴𝘤𝘢𝘱𝘦." …it's exciting to see major players like Meta pushing the boundaries of immersive experiences. Their announcement of Hyperscape resonates deeply with the work I've been doing and my passion for capturing and recreating photorealistic environments and memories. It feels like we're all moving toward a future where the lines between the physical and digital worlds blur, allowing us to share spaces and moments in unprecedented ways. I'm thrilled to be contributing my own explorations to this ever-evolving field. The attached video is a screen recording of myself exploring and flying through volumetric captures—dormant memories—in real-time using an app I crafted for this purpose. The first part shows me moving from one memory to another (btw, this works in tethered VR), and in the second part of the video at , I explore two memories simultaneously, rendering two 3D Gaussian Splats of the same location onto two monitors. #gaussiansplatting #radiancefield #digitaltwin #hyperscape #memories

Hugues Bruyère

16,304 görüntüleme • 1 yıl önce

I turned a single renovation photo into a cinematic time-lapse video for $10 of credits. These videos are crushing on social media because humans naturally want to see what the construction turns into. The dwell time is nuts. So... now we're using these videos to help promote real estate development and landscaping companies. Here's how it works: → Drop one finished renovation image into a customGPT that reverse-engineers the full construction sequence → Calico AI generates each stage of the renovation with consistent aerial camera angles → Kling 3.0 animates the timelapse transitions between frames with realistic construction movement → ElevenLabs generates a matching soundtrack → Stitch in CapCut and export The result: real estate developers and landscaping companies get portfolio-quality timelapse content from a single photo. No production team. Landscaping firms can use these for ads, social content, client pitches, and website videos that actually convert. Comment "RENOVATION" and I'll send you the exact system, custom GPT, and all prompts (must be following so I can DM you!).
9:47

I turned a single renovation photo into a cinematic time-lapse video for $10 of credits. These videos are crushing on social media because humans naturally want to see what the construction turns into. The dwell time is nuts. So... now we're using these videos to help promote real estate development and landscaping companies. Here's how it works: → Drop one finished renovation image into a customGPT that reverse-engineers the full construction sequence → Calico AI generates each stage of the renovation with consistent aerial camera angles → Kling 3.0 animates the timelapse transitions between frames with realistic construction movement → ElevenLabs generates a matching soundtrack → Stitch in CapCut and export The result: real estate developers and landscaping companies get portfolio-quality timelapse content from a single photo. No production team. Landscaping firms can use these for ads, social content, client pitches, and website videos that actually convert. Comment "RENOVATION" and I'll send you the exact system, custom GPT, and all prompts (must be following so I can DM you!).

David Roberts

64,267 görüntüleme • 4 ay önce

Kasia v0.2.0 is here—our biggest update yet! 🚀 We’ve been busy building. Here’s what’s new: Payments are Here! 💸 Send $KAS directly in chat. Seamless, in-message payments. Send custom messages with a payment. KNS Integration 📛 No more long addresses. Use .kas names or just the name to message and pay. Go Mobile! 📱 Fully mobile-friendly. Kasia on the go. Native apps coming soon™. Major UI/UX Overhaul ✨ - Redesigned home screen & message bubbles - Uniquely generated avatars on desktop to visually ID contacts - Replaced alerts with smooth toast notifications - No more estimate fee flickering with a clean redesign Advanced Wallet Features 🛠️ - UTXO Compounding to reduce fees - Priority Fee Selector for custom speed - Custom Kaspa Node URL support Media & Performance Boosts 📸⚡ - Image compression & resizing on upload - take photos from your phone and send - Faster message fetching with optimized decryption - Handshake flow overhaul and improvements - Improved Export / Import functionality. Thanks for supporting the future of decentralized private communication. Check out v0.2.0 and tell us what you think!
0:31

Kasia v0.2.0 is here—our biggest update yet! 🚀 We’ve been busy building. Here’s what’s new: Payments are Here! 💸 Send $KAS directly in chat. Seamless, in-message payments. Send custom messages with a payment. KNS Integration 📛 No more long addresses. Use .kas names or just the name to message and pay. Go Mobile! 📱 Fully mobile-friendly. Kasia on the go. Native apps coming soon™. Major UI/UX Overhaul ✨ - Redesigned home screen & message bubbles - Uniquely generated avatars on desktop to visually ID contacts - Replaced alerts with smooth toast notifications - No more estimate fee flickering with a clean redesign Advanced Wallet Features 🛠️ - UTXO Compounding to reduce fees - Priority Fee Selector for custom speed - Custom Kaspa Node URL support Media & Performance Boosts 📸⚡ - Image compression & resizing on upload - take photos from your phone and send - Faster message fetching with optimized decryption - Handshake flow overhaul and improvements - Improved Export / Import functionality. Thanks for supporting the future of decentralized private communication. Check out v0.2.0 and tell us what you think!

Kasia

67,998 görüntüleme • 1 yıl önce

We are excited to share our work “Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones” published in IEEE Transactions on Robotics IEEE Transactions on Robotics (T-RO), which tackles sharp radiance field reconstruction under agile drone motion, where RGB frames are heavily motion-blurred and pose priors become unreliable! 4 years in the making! Code & dataset released! PDF: Code & Dataset: Full Narrated Video: High-speed flight is essential for time- and battery-constrained missions (e.g., inspection, exploration, search & rescue). However, fast motion corrupts visual data with severe motion blur and introduces drift/noise in visual-inertial odometry, making NeRF-based 3D reconstruction particularly brittle. We propose a unified framework that leverages asynchronous #EventCamera streams together with motion-blurred frames to reconstruct high-fidelity radiance fields from agile drone flights. Our key idea is to embed event-image fusion directly into radiance field optimization while jointly refining a shared, continuous-time camera trajectory initialized from event-based VIO. This enables us to recover sharp radiance fields and accurate trajectories without ground-truth supervision during training. We validate our method on synthetic data and on real sequences captured by a drone flying up to 2 m/s. Despite severe blur and noisy pose priors, our method preserves fine scene details and achieves a performance gain of over 50% on real-world data compared to state-of-the-art methods. Kudos to Rong Zou and Marco Cannici! Marco Cannici Reference: Rong Zou*, Marco Cannici*, Davide Scaramuzza Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones IEEE Transactions on Robotics (T-RO), 2026 NCCR Robotics European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub
1:02

We are excited to share our work “Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones” published in IEEE Transactions on Robotics IEEE Transactions on Robotics (T-RO), which tackles sharp radiance field reconstruction under agile drone motion, where RGB frames are heavily motion-blurred and pose priors become unreliable! 4 years in the making! Code & dataset released! PDF: Code & Dataset: Full Narrated Video: High-speed flight is essential for time- and battery-constrained missions (e.g., inspection, exploration, search & rescue). However, fast motion corrupts visual data with severe motion blur and introduces drift/noise in visual-inertial odometry, making NeRF-based 3D reconstruction particularly brittle. We propose a unified framework that leverages asynchronous #EventCamera streams together with motion-blurred frames to reconstruct high-fidelity radiance fields from agile drone flights. Our key idea is to embed event-image fusion directly into radiance field optimization while jointly refining a shared, continuous-time camera trajectory initialized from event-based VIO. This enables us to recover sharp radiance fields and accurate trajectories without ground-truth supervision during training. We validate our method on synthetic data and on real sequences captured by a drone flying up to 2 m/s. Despite severe blur and noisy pose priors, our method preserves fine scene details and achieves a performance gain of over 50% on real-world data compared to state-of-the-art methods. Kudos to Rong Zou and Marco Cannici! Marco Cannici Reference: Rong Zou*, Marco Cannici*, Davide Scaramuzza Event-Aided Sharp Radiance Field Reconstruction for Fast-Flying Drones IEEE Transactions on Robotics (T-RO), 2026 NCCR Robotics European Research Council (ERC) AUTOASSESS UZH IfI University of Zurich UZH Science Prophesee SynSense UZH Space Hub

Davide Scaramuzza

11,989 görüntüleme • 4 ay önce

VoxCPM 2 just dropped by OpenBMB Only 2B-param open-source TTS (Text-to-Speech) model built for production-grade multilingual voice work. Apache-2.0 license, Can run on only 8GB VRAM. • Eliminates the "robotic" feel of traditional TTS, delivering prosody and emotional depth suitable for high-stakes professional environments like filmmaking, gaming, animation, and audiobooks. • 30-language multilingual: no language tag needed, just type in a supported language and generate directly. • Voice design: create a brand-new voice from a text description alone, like age, tone, pace, or emotion. No reference audio required. Describe the desired voice characteristics (gender, age, tone, emotion, pace …) in Control Instruction, and VoxCPM2 will craft a unique voice from your description alone. • Controllable cloning: clone from a short clip, then steer delivery style without losing the speaker’s core voice. • Ultimate cloning: use reference audio + transcript for continuation-style cloning that keeps the tiny vocal details. • 48kHz output: takes 16kHz reference audio and produces studio-quality speech without an external upsampler. • Real-time ready: around 0.3 RTF on RTX 4090, even lower with Nano-VLLM. • Commercial use: Apache-2.0 licensed. Developer-Friendly Infrastructure: - Native Torch Inference: Direct support for PyTorch-based workflows. - Training Flexibility: Supports both full-parameter and LoRA fine-tuning for specific domain adaptation. - Production Readiness: Compatible with voxcpm-nanovllm for large-scale, high-concurrency deployment.
1:58

VoxCPM 2 just dropped by OpenBMB Only 2B-param open-source TTS (Text-to-Speech) model built for production-grade multilingual voice work. Apache-2.0 license, Can run on only 8GB VRAM. • Eliminates the "robotic" feel of traditional TTS, delivering prosody and emotional depth suitable for high-stakes professional environments like filmmaking, gaming, animation, and audiobooks. • 30-language multilingual: no language tag needed, just type in a supported language and generate directly. • Voice design: create a brand-new voice from a text description alone, like age, tone, pace, or emotion. No reference audio required. Describe the desired voice characteristics (gender, age, tone, emotion, pace …) in Control Instruction, and VoxCPM2 will craft a unique voice from your description alone. • Controllable cloning: clone from a short clip, then steer delivery style without losing the speaker’s core voice. • Ultimate cloning: use reference audio + transcript for continuation-style cloning that keeps the tiny vocal details. • 48kHz output: takes 16kHz reference audio and produces studio-quality speech without an external upsampler. • Real-time ready: around 0.3 RTF on RTX 4090, even lower with Nano-VLLM. • Commercial use: Apache-2.0 licensed. Developer-Friendly Infrastructure: - Native Torch Inference: Direct support for PyTorch-based workflows. - Training Flexibility: Supports both full-parameter and LoRA fine-tuning for specific domain adaptation. - Production Readiness: Compatible with voxcpm-nanovllm for large-scale, high-concurrency deployment.

Rohan Paul

13,541 görüntüleme • 3 ay önce

🚨 SIGGRAPH Asia 2025 Paper Alert 🚨 ➡️Paper Title: WorldExplorer: Towards Generating Fully Navigable 3D Scenes 🌟Few pointers from the paper 🎯Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce stretched-out and noisy artifacts when moving beyond central or panoramic perspectives. 🎯 To this end, authors of this paper proposed “WorldExplorer”, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. 🎯They initialize their scenes by creating multi-view consistent images corresponding to a 360 degree panorama. 🎯Then, they expanded it by leveraging video diffusion models in an iterative scene generation pipeline. 🎯Concretely, they generated multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. 🎯Their novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. 🎯Finally,they fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. 🎯Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. 🎯They believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments. 🏢Organization: TU München 🧙Paper Authors: Manuel-Andreas Schneider, Lukas Höllein , Matthias Niessner 📝 Read the Full Paper here: 🗂️ Project Page: 🧑‍💻 Code: 🎥 Be sure to watch the attached Technical Summary Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #SIGGRAPHAsia2025
3:36

🚨 SIGGRAPH Asia 2025 Paper Alert 🚨 ➡️Paper Title: WorldExplorer: Towards Generating Fully Navigable 3D Scenes 🌟Few pointers from the paper 🎯Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce stretched-out and noisy artifacts when moving beyond central or panoramic perspectives. 🎯 To this end, authors of this paper proposed “WorldExplorer”, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. 🎯They initialize their scenes by creating multi-view consistent images corresponding to a 360 degree panorama. 🎯Then, they expanded it by leveraging video diffusion models in an iterative scene generation pipeline. 🎯Concretely, they generated multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. 🎯Their novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. 🎯Finally,they fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. 🎯Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. 🎯They believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments. 🏢Organization: TU München 🧙Paper Authors: Manuel-Andreas Schneider, Lukas Höllein , Matthias Niessner 📝 Read the Full Paper here: 🗂️ Project Page: 🧑‍💻 Code: 🎥 Be sure to watch the attached Technical Summary Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #SIGGRAPHAsia2025

naveen manwani

10,578 görüntüleme • 10 ay önce

🚀 🚀Excited to announce the technical report of MiniCPM-o 4.5! MiniCPM-o 4.5 transitions #AI interaction from traditional turn-based processing to a real-time, native full-duplex stream-based paradigm. 🌊 The Omni-Flow Framework Instead of traditional VAD-based workarounds, we introduce the #Omni-#Flow framework. This unified stream paradigm aligns video, audio, and text on a synchronized millisecond timeline. • Native Full-Duplex: Simultaneous perception and response. • Proactive Interaction: Natively manages turn-taking without external VAD, supports proactive reminding. 📉 9B Scale, SOTA Performance MiniCPM-o 4.5 demonstrates SOTA multimodal intelligence at its scale: • Multimodal Benchmarks: Comparable to #Gemini 2.5 Flash on MMBench EN (87.6) and MathVista (80.1). • Streaming Evaluation: 54.4% win rate on LiveSports-3K-CC, surpassing specialized models. 💻 The Ultimate Edge AI — Fully Functional without Network Connection We are providing one-click installers for Windows (12G VRAM,RTX 5070) and macOS (M1-M5 Max/ M5 Pro). • Local API Support: Deploy your own inference server to integrate native full-duplex into custom apps. • Free Access: We are offering free community API services for exploration. • 100% Private: Your data never leaves your machine. Deploy in under 10 minutes. 🛠️👇 👐 Join the Open Future The weights are open. The protocol is public. 📄 Technical Report: 💻 GitHub: 🤗 HuggingFace: 🌐 Web Demo: #MiniCPMo #OpenSourceAI #EdgeAI #MachineLearning #ComputerVision #LLM
2:08

🚀 🚀Excited to announce the technical report of MiniCPM-o 4.5! MiniCPM-o 4.5 transitions #AI interaction from traditional turn-based processing to a real-time, native full-duplex stream-based paradigm. 🌊 The Omni-Flow Framework Instead of traditional VAD-based workarounds, we introduce the #Omni-#Flow framework. This unified stream paradigm aligns video, audio, and text on a synchronized millisecond timeline. • Native Full-Duplex: Simultaneous perception and response. • Proactive Interaction: Natively manages turn-taking without external VAD, supports proactive reminding. 📉 9B Scale, SOTA Performance MiniCPM-o 4.5 demonstrates SOTA multimodal intelligence at its scale: • Multimodal Benchmarks: Comparable to #Gemini 2.5 Flash on MMBench EN (87.6) and MathVista (80.1). • Streaming Evaluation: 54.4% win rate on LiveSports-3K-CC, surpassing specialized models. 💻 The Ultimate Edge AI — Fully Functional without Network Connection We are providing one-click installers for Windows (12G VRAM,RTX 5070) and macOS (M1-M5 Max/ M5 Pro). • Local API Support: Deploy your own inference server to integrate native full-duplex into custom apps. • Free Access: We are offering free community API services for exploration. • 100% Private: Your data never leaves your machine. Deploy in under 10 minutes. 🛠️👇 👐 Join the Open Future The weights are open. The protocol is public. 📄 Technical Report: 💻 GitHub: 🤗 HuggingFace: 🌐 Web Demo: #MiniCPMo #OpenSourceAI #EdgeAI #MachineLearning #ComputerVision #LLM

OpenBMB

147,824 görüntüleme • 2 ay önce

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal
0:39

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,106 görüntüleme • 1 yıl önce

A new 30-minute presentation from Ashok Elluswamy, Tesla’s VP of AI, has been released, where he talks about FSD, AI and the team’s latest progress. Highlight from the presentation: • Tesla's vehicle fleet can provide 500 years of driving data every single day. Curse of Dimensionality: • 8 cameras at high frame rate = billions of tokens per 30 seconds of driving context. • Tesla must compress and extract the right correlations between sensory input and control actions. Data Advantage: • Tesla has access to a “Niagara Falls of data” — hundreds of years’ worth of collective fleet driving. • Uses smart data triggers to capture rare corner cases (e.g., complex intersections, unpredictable behavior). Quality and Efficiency: • Extracts only the essential data needed to train models efficiently. Debugging and Interpretability: • Even though the system is end-to-end, Tesla can still prompt the model to output interpretable data: 3D occupancy, road boundaries, objects, signs, traffic lights, etc. • Natural language querying: ask the model why it made a certain decision. • These auxiliary predictions don’t drive the car but help engineers debug and ensure safety. Tesla’s Advanced Gaussian Splatting (3D Scene Modeling): • Tesla developed a custom, ultra-fast Gaussian splatting system to reconstruct 3D scenes from limited camera views. • Produces crisp, accurate 3D renderings even from few camera angles — far better than standard NeRF/splatting approaches. • Enables rapid visual debugging of the driving environment in 3D. Evaluation & World Models: • Evaluation is the hardest challenge: models may perform well offline but fail in real-world conditions. • Tesla builds balanced, diverse evaluation datasets focusing on edge cases — not just easy highway driving. Introduced a learned world simulator (neural network-generated video engine): • Can simulate 8 Tesla camera feeds simultaneously — fully synthetic. • Used for testing, training, and reinforcement learning. • Allows adversarial event injection (e.g., adding a pedestrian or vehicle cutting in). • Enables replaying past failures to verify new model improvements. • Can run in near real-time, letting testers “drive” inside a simulated world. What’s Next: • Scale robotaxi service globally. • Unlock full autonomy across the entire Tesla fleet. • Cybercab: next-gen 2-seat vehicle designed specifically for robotaxi use, targeting lowest transportation cost (cheaper than public transit). • Same neural networks will power Optimus humanoid robot. • The same video generation system is now being applied to Optimus. • The system can simulate and plan movement for robots, adapting easily to new forms. via the International Conference on Computer Vision (ICCV). Full presentation:
0:29

A new 30-minute presentation from Ashok Elluswamy, Tesla’s VP of AI, has been released, where he talks about FSD, AI and the team’s latest progress. Highlight from the presentation: • Tesla's vehicle fleet can provide 500 years of driving data every single day. Curse of Dimensionality: • 8 cameras at high frame rate = billions of tokens per 30 seconds of driving context. • Tesla must compress and extract the right correlations between sensory input and control actions. Data Advantage: • Tesla has access to a “Niagara Falls of data” — hundreds of years’ worth of collective fleet driving. • Uses smart data triggers to capture rare corner cases (e.g., complex intersections, unpredictable behavior). Quality and Efficiency: • Extracts only the essential data needed to train models efficiently. Debugging and Interpretability: • Even though the system is end-to-end, Tesla can still prompt the model to output interpretable data: 3D occupancy, road boundaries, objects, signs, traffic lights, etc. • Natural language querying: ask the model why it made a certain decision. • These auxiliary predictions don’t drive the car but help engineers debug and ensure safety. Tesla’s Advanced Gaussian Splatting (3D Scene Modeling): • Tesla developed a custom, ultra-fast Gaussian splatting system to reconstruct 3D scenes from limited camera views. • Produces crisp, accurate 3D renderings even from few camera angles — far better than standard NeRF/splatting approaches. • Enables rapid visual debugging of the driving environment in 3D. Evaluation & World Models: • Evaluation is the hardest challenge: models may perform well offline but fail in real-world conditions. • Tesla builds balanced, diverse evaluation datasets focusing on edge cases — not just easy highway driving. Introduced a learned world simulator (neural network-generated video engine): • Can simulate 8 Tesla camera feeds simultaneously — fully synthetic. • Used for testing, training, and reinforcement learning. • Allows adversarial event injection (e.g., adding a pedestrian or vehicle cutting in). • Enables replaying past failures to verify new model improvements. • Can run in near real-time, letting testers “drive” inside a simulated world. What’s Next: • Scale robotaxi service globally. • Unlock full autonomy across the entire Tesla fleet. • Cybercab: next-gen 2-seat vehicle designed specifically for robotaxi use, targeting lowest transportation cost (cheaper than public transit). • Same neural networks will power Optimus humanoid robot. • The same video generation system is now being applied to Optimus. • The system can simulate and plan movement for robots, adapting easily to new forms. via the International Conference on Computer Vision (ICCV). Full presentation:

Sawyer Merritt

1,287,003 görüntüleme • 9 ay önce