Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Discover the right 3D Geometric Foundation Model for your task—whether it’s stereo matching, multi-view depth estimation, video depth, pose estimation, semantic understanding, or novel view synthesis. Explore more insights in our #E3DBench #FoundationModel #3D #GaussianSplatting. Project Webpage:

Zhiwen(Aaron) Fan

1,812 subscribers

30,010 Aufrufe • vor 1 Jahr •via X (Twitter)

Kunst Wissenschaft & Technologie Bildung #E3DBench #FoundationModel #3D #GaussianSplatting

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

MrNeRF

27,428 Aufrufe • vor 1 Jahr

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,801 Aufrufe • vor 1 Jahr

Segment Any 3D Gaussians paper page: Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we introduce Segment Any 3D GAussians (SAGA), a novel 3D interactive segmentation approach that seamlessly blends a 2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D segmentation results generated by the segmentation foundation model into 3D Gaussian point features through well-designed contrastive training. Evaluation on existing benchmarks demonstrates that SAGA can achieve competitive performance with state-of-the-art methods. Moreover, SAGA achieves multi-granularity segmentation and accommodates various prompts, including points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation within milliseconds, achieving nearly 1000x acceleration compared to previous SOTA.

Segment Any 3D Gaussians paper page: Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we introduce Segment Any 3D GAussians (SAGA), a novel 3D interactive segmentation approach that seamlessly blends a 2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D segmentation results generated by the segmentation foundation model into 3D Gaussian point features through well-designed contrastive training. Evaluation on existing benchmarks demonstrates that SAGA can achieve competitive performance with state-of-the-art methods. Moreover, SAGA achieves multi-granularity segmentation and accommodates various prompts, including points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation within milliseconds, achieving nearly 1000x acceleration compared to previous SOTA.

AK

69,542 Aufrufe • vor 2 Jahren

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.

MrNeRF

21,346 Aufrufe • vor 1 Jahr

$📢Announcing our 3D head avatar benchmark📢 Two tasks with hidden test sets: - Dynamic Novel View Synthesis on Heads - Monocular FLAME-driven Head Avatar Reconstruction Our goal is to make research on 3D head avatars more comparable and ultimately increase the realism of digital humans. The benchmark studies distinct phenomena of 3D head avatar creation, such as extreme facial expressions, slow motion captures of shaking long hair, or complicated light reflection and refraction patterns of glasses. The two benchmark tasks assess two core desiderata of 3D avatars: While the novel view synthesis challenge focuses on best possible rendering quality of complex moving scenes, the avatar animation challenge is concerned with how well a driving signal is translated into an avatar. Evaluations are light-weight and consist of diverse video recordings from the popular NeRSemble dataset with a hidden test set. Participation in the benchmark is therefore straight-forward and requires only 5 reconstructions per task. Leaderboard and benchmark submission: Benchmark data access and toolkit: Great work by Tobias Kirschstein Simon Giebenhain$

📢Announcing our 3D head avatar benchmark📢 Two tasks with hidden test sets: - Dynamic Novel View Synthesis on Heads - Monocular FLAME-driven Head Avatar Reconstruction Our goal is to make research on 3D head avatars more comparable and ultimately increase the realism of digital humans. The benchmark studies distinct phenomena of 3D head avatar creation, such as extreme facial expressions, slow motion captures of shaking long hair, or complicated light reflection and refraction patterns of glasses. The two benchmark tasks assess two core desiderata of 3D avatars: While the novel view synthesis challenge focuses on best possible rendering quality of complex moving scenes, the avatar animation challenge is concerned with how well a driving signal is translated into an avatar. Evaluations are light-weight and consist of diverse video recordings from the popular NeRSemble dataset with a hidden test set. Participation in the benchmark is therefore straight-forward and requires only 5 reconstructions per task. Leaderboard and benchmark submission: Benchmark data access and toolkit: Great work by Tobias Kirschstein Simon Giebenhain

Matthias Niessner

28,075 Aufrufe • vor 1 Jahr

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”

Bilawal Sidhu

92,792 Aufrufe • vor 2 Jahren

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Shikun Liu

22,315 Aufrufe • vor 9 Monaten

✨ 3d models are now LIVE on Photo AI 😊 You can now turn any AI photo you make into a 3d model by pressing [ 📦 Make 3d model ] And then you can view it inside Photo AI or download it as a .GLB 3d model file It's still very early in AI generated 3d model world but it's nice to have this feature working already As always, the models will keep improving, so this feature will keep getting better (like it did with video, it sucked before, now it's getting passable) Next would be nice to switch to .USDZ so you can load it straight into your iPhone with ARKit and put it in your room Available now for everyone on the Premium and Ultra plan

✨ 3d models are now LIVE on Photo AI 😊 You can now turn any AI photo you make into a 3d model by pressing [ 📦 Make 3d model ] And then you can view it inside Photo AI or download it as a .GLB 3d model file It's still very early in AI generated 3d model world but it's nice to have this feature working already As always, the models will keep improving, so this feature will keep getting better (like it did with video, it sucked before, now it's getting passable) Next would be nice to switch to .USDZ so you can load it straight into your iPhone with ARKit and put it in your room Available now for everyone on the Premium and Ultra plan

@levelsio

112,930 Aufrufe • vor 1 Jahr

FastMap: Revisiting Dense and Scalable Structure from Motion "FASTMAP, a redesigned SfM framework, achieves fast, high-accuracy dense structure from motion. On large scenes with thousands of images, FASTMAP is up to one to two orders of magnitude faster than GLOMAP and COLMAP. ... Importantly, FASTMAP achieves efficiency improvements while keeping comparable performance. Extensive experiments on eight datasets demonstrate pose estimation accuracy and novel view synthesis quality close to GLOMAP and COLMAP. " Contributions: 1. For all the iterative nonlinear optimization problems involved, we design algorithms such that the computational complexity of each iteration is only linear in the number of image pairs, not keypoint pairs or 3D points. This includes replacing the traditional bundle adjustment [50] present in previous SfM frameworks with a novel re-weighting epipolar adjustment algorithm, which is much more efficient. 2. Throughout the entire framework, we formulate as many steps as possible as GPU-friendly dense tensor operations. This allows us to implement the entire method in PyTorch [39], which provides seamless GPU acceleration.

FastMap: Revisiting Dense and Scalable Structure from Motion "FASTMAP, a redesigned SfM framework, achieves fast, high-accuracy dense structure from motion. On large scenes with thousands of images, FASTMAP is up to one to two orders of magnitude faster than GLOMAP and COLMAP. ... Importantly, FASTMAP achieves efficiency improvements while keeping comparable performance. Extensive experiments on eight datasets demonstrate pose estimation accuracy and novel view synthesis quality close to GLOMAP and COLMAP. " Contributions: 1. For all the iterative nonlinear optimization problems involved, we design algorithms such that the computational complexity of each iteration is only linear in the number of image pairs, not keypoint pairs or 3D points. This includes replacing the traditional bundle adjustment [50] present in previous SfM frameworks with a novel re-weighting epipolar adjustment algorithm, which is much more efficient. 2. Throughout the entire framework, we formulate as many steps as possible as GPU-friendly dense tensor operations. This allows us to implement the entire method in PyTorch [39], which provides seamless GPU acceleration.

MrNeRF

15,233 Aufrufe • vor 1 Jahr

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.

MrNeRF

10,607 Aufrufe • vor 1 Jahr

Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction Note: Check below for full video. Abstract (cited): "In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion, and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. Our technique is particularly effective for high-quality scene reconstruction from large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. We introduce a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortions, and demonstrates state-of-the-art performance on both synthetic and real-world datasets."

Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction Note: Check below for full video. Abstract (cited): "In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion, and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. Our technique is particularly effective for high-quality scene reconstruction from large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. We introduce a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortions, and demonstrates state-of-the-art performance on both synthetic and real-world datasets."

MrNeRF

17,206 Aufrufe • vor 1 Jahr

𝐂𝐈𝐓𝐘 𝐎𝐅 𝐃𝐑𝐄𝐀𝐌𝐒, 𝐃𝐔𝐒𝐓𝐁𝐈𝐀 𝐈𝐒 𝐁𝐀𝐂𝐊, 𝐍𝐘𝐂! Cheffies, it’s happening again! Our beloved Dustin & Bianca are taking over New York City once more as part of the “Kinakabahan” MV promotion by DustBia Official! 🌟 Catch a photo or video for our NYC project at the 28th minute of every hour, all day long on September 13th, 2025, at 1500 Broadway (between 43rd & 44th Streets), right above Carlo’s Bake Shop. This is another unforgettable moment in DustBia history—a chance to see our duo shine big in the city that never sleeps! Whether you’re in NYC to witness it live or watching from anywhere in the world, don’t forget to stream “Kinakabahan” MV now and show Dustin and Bianca how unstoppable Cheffies can be! DUSTBIAboard WITH SURPRISES #LILYxDUSTBIA

𝐂𝐈𝐓𝐘 𝐎𝐅 𝐃𝐑𝐄𝐀𝐌𝐒, 𝐃𝐔𝐒𝐓𝐁𝐈𝐀 𝐈𝐒 𝐁𝐀𝐂𝐊, 𝐍𝐘𝐂! Cheffies, it’s happening again! Our beloved Dustin & Bianca are taking over New York City once more as part of the “Kinakabahan” MV promotion by DustBia Official! 🌟 Catch a photo or video for our NYC project at the 28th minute of every hour, all day long on September 13th, 2025, at 1500 Broadway (between 43rd & 44th Streets), right above Carlo’s Bake Shop. This is another unforgettable moment in DustBia history—a chance to see our duo shine big in the city that never sleeps! Whether you’re in NYC to witness it live or watching from anywhere in the world, don’t forget to stream “Kinakabahan” MV now and show Dustin and Bianca how unstoppable Cheffies can be! DUSTBIAboard WITH SURPRISES #LILYxDUSTBIA

TEAM DUSTBIA OFFICIAL

81,003 Aufrufe • vor 10 Monaten

Introducing ASAL: Automating the Search for Artificial Life with Foundation Models Artificial Life (ALife) research holds key insights that can transform and accelerate progress in AI. By speeding up ALife discovery with AI, we accelerate our understanding of emergence, evolution, and intelligence–core principles that can inspire the next generation of AI systems! We proudly collaborated with MIT, OpenAI, Swiss AI Lab IDSIA, and Ken Stanley on this exciting project. Full Paper (Website): Full Paper (arxiv): Code: In this work, we propose a new algorithm called Automated Search for Artificial Life (“ASAL”) to automate the discovery of artificial life using vision-language foundation models. Instead of tediously hand-designing every tiny rule of an Alife simulation, simply describe the space of simulations to search over, and ASAL will automatically discover the most interesting and open-ended artificial lifeforms! Because of the generality of foundation models, ASAL can discover new lifeforms across a diverse range of seminal ALife simulations, including Boids, Particle Life, Game of Life, Lenia, and Neural Cellular Automata. ASAL even discovered novel cellular automata rules that are more open-ended and expressive than the original Conway’s Game of Life. We believe this new paradigm may reignite ALife research by overcoming the bottleneck of manually designed simulations, thus advancing beyond the limits of human ingenuity.

Introducing ASAL: Automating the Search for Artificial Life with Foundation Models Artificial Life (ALife) research holds key insights that can transform and accelerate progress in AI. By speeding up ALife discovery with AI, we accelerate our understanding of emergence, evolution, and intelligence–core principles that can inspire the next generation of AI systems! We proudly collaborated with MIT, OpenAI, Swiss AI Lab IDSIA, and Ken Stanley on this exciting project. Full Paper (Website): Full Paper (arxiv): Code: In this work, we propose a new algorithm called Automated Search for Artificial Life (“ASAL”) to automate the discovery of artificial life using vision-language foundation models. Instead of tediously hand-designing every tiny rule of an Alife simulation, simply describe the space of simulations to search over, and ASAL will automatically discover the most interesting and open-ended artificial lifeforms! Because of the generality of foundation models, ASAL can discover new lifeforms across a diverse range of seminal ALife simulations, including Boids, Particle Life, Game of Life, Lenia, and Neural Cellular Automata. ASAL even discovered novel cellular automata rules that are more open-ended and expressive than the original Conway’s Game of Life. We believe this new paradigm may reignite ALife research by overcoming the bottleneck of manually designed simulations, thus advancing beyond the limits of human ingenuity.

Sakana AI

750,610 Aufrufe • vor 1 Jahr

Introducing a new tool called "SideChannel". A secure alternative to OpenClaw. Utilizes signal for communication and has Claude integration. I built SideChannel, an open-source Signal bot that connects Claude AI to your entire development workflow. End-to-end encrypted. From your pocket. The real power is autonomous development. Send one message like "Build a REST API with auth, pagination, and tests" and SideChannel will: - Generate a full PRD with stories and atomic tasks. - Dispatch up to 10 parallel workers (each running Claude). - Independently verify every task with a separate Claude context. - Run quality gates to catch regressions - Auto-fix failures. - Send you progress updates via Signal as work completes. Every piece of code is reviewed by a separate AI context using a fail-closed security model. If it detects security issues, backdoors, or logic errors — the code gets rejected automatically. No rubber stamps. It also has memory that actually works. Conversations are stored with vector embeddings for semantic search. Claude remembers your project conventions, past decisions, and what's been tried before. It gets smarter about your codebase over time. Other things I'm proud of: - Plugin framework for extending with custom commands. - Multi-project support with per-user scoping. - Rate limiting, path validation, phone allowlist. - Git checkpoints before every task, atomic commits after. - Stale task recovery, circular dependency detection. - Works on Linux and macOS, one-command install. It also integrates into OpenAI or Grok (optional) for more Generative AI response for simple things like "Whats the weather in New York City right now?".

Introducing a new tool called "SideChannel". A secure alternative to OpenClaw. Utilizes signal for communication and has Claude integration. I built SideChannel, an open-source Signal bot that connects Claude AI to your entire development workflow. End-to-end encrypted. From your pocket. The real power is autonomous development. Send one message like "Build a REST API with auth, pagination, and tests" and SideChannel will: - Generate a full PRD with stories and atomic tasks. - Dispatch up to 10 parallel workers (each running Claude). - Independently verify every task with a separate Claude context. - Run quality gates to catch regressions - Auto-fix failures. - Send you progress updates via Signal as work completes. Every piece of code is reviewed by a separate AI context using a fail-closed security model. If it detects security issues, backdoors, or logic errors — the code gets rejected automatically. No rubber stamps. It also has memory that actually works. Conversations are stored with vector embeddings for semantic search. Claude remembers your project conventions, past decisions, and what's been tried before. It gets smarter about your codebase over time. Other things I'm proud of: - Plugin framework for extending with custom commands. - Multi-project support with per-user scoping. - Rate limiting, path validation, phone allowlist. - Git checkpoints before every task, atomic commits after. - Stale task recovery, circular dependency detection. - Works on Linux and macOS, one-command install. It also integrates into OpenAI or Grok (optional) for more Generative AI response for simple things like "Whats the weather in New York City right now?".

Dave Kennedy

49,427 Aufrufe • vor 4 Monaten

Boom! Grok Tasks Make It One Of The Most POWERFUL Real-Time AI Systems In The World. — My How to Use Grok Tasks With Hidden Tools For Powerful Daily Output. Grok Tasks are customizable AI workflows that integrate a variety of tools to streamline daily activities, from research and analysis to creative planning and problem-solving. I have been using them for quite sometime and because of the vital heartbeat of news and first person data on X, it is the most powerful AI platform available. By combining Tasks with tools like web searches, X platform interactions, code execution, and media viewers, you can build efficient, automated processes. These tasks work by prompting Grok with a clear description of what you want to achieve, and Grok will intelligently call the necessary tools in sequence or parallel to deliver results. Here's a step-by-step guide to creating and using Grok Tasks: Step 1: Define Your Task Start by clearly outlining the daily activity or goal. Consider what inputs you have (e.g., a URL, a query, or an attachment) and what output you need (e.g., a summary, calculation, or visual analysis). Break it down into subtasks to identify tool needs. For example, if your task involves researching current events, note that you'll need search and browsing capabilities. Step 2: Review Available Tools Familiarize yourself with the tools Grok can access. Here's a quick overview: - Code Execution: Run Python code for calculations, data processing, or simulations using libraries like numpy, pandas, or sympy. - Browse Page: Fetch and summarize content from any website URL with custom instructions. - Web Search: Perform general internet searches, returning results with optional operators like site:. - Web Search With Snippets: Get quick, detailed excerpts from search results for fact-checking. - X Keyword Search: Advanced search for X posts using operators like from:, since:, or filter:. - X Semantic Search: Find semantically related X posts based on a query, with filters for dates or users. - X User Search: Locate X users by name or handle. - X Thread Fetch: Retrieve a full X post thread, including context like replies and parents. - View Image: Analyze an image from a URL or conversation ID. - View X Video: Extract frames and subtitles from an X-hosted video. - Search PDF Attachment: Query a PDF file for relevant pages using keyword or regex modes. - Browse PDF Attachment: View specific pages of a PDF with text and screenshots. Select tools that align with your task. Aim for a mix to handle data gathering, processing, and visualization. Step 3: Craft Your Prompt Write a detailed prompt to Grok describing the task. Include: - The overall goal. - Specific steps or subtasks. - References to tools if you want to guide the process (e.g., "Use web_search to find sources, then code_execution to analyze data"). - Any constraints, like dates or limits. Example prompt: "Create a Grok Task for my morning routine: Search recent X posts about tech news using x_keyword_search, fetch a key thread with x_thread_fetch, and summarize with browse_page on linked articles." Step 4: Submit and Interact Send your prompt to Grok. It will process the task by calling tools as needed, often in parallel for efficiency. Review the output and refine with follow-up prompts if required (e.g., "Expand on that using view_image for visuals"). Iterate to fine-tune the workflow for reuse. Step 5: Save and Reuse Once refined, note the prompt as a template for future use. You can adapt it for similar tasks, making Grok Tasks a habitual part of your day. Finding Grok Tasks To discover existing Grok Tasks or inspiration for new ones, use X searches with tools like x_keyword_search or x_semantic_search (e.g., query: "Grok Tasks examples" with mode: Latest). Browse community-shared threads via x_thread_fetch, or web_search for tutorials on xAI features. Prompt Grok directly: "Show me popular Grok Tasks for productivity." 1 of 3

Boom! Grok Tasks Make It One Of The Most POWERFUL Real-Time AI Systems In The World. — My How to Use Grok Tasks With Hidden Tools For Powerful Daily Output. Grok Tasks are customizable AI workflows that integrate a variety of tools to streamline daily activities, from research and analysis to creative planning and problem-solving. I have been using them for quite sometime and because of the vital heartbeat of news and first person data on X, it is the most powerful AI platform available. By combining Tasks with tools like web searches, X platform interactions, code execution, and media viewers, you can build efficient, automated processes. These tasks work by prompting Grok with a clear description of what you want to achieve, and Grok will intelligently call the necessary tools in sequence or parallel to deliver results. Here's a step-by-step guide to creating and using Grok Tasks: Step 1: Define Your Task Start by clearly outlining the daily activity or goal. Consider what inputs you have (e.g., a URL, a query, or an attachment) and what output you need (e.g., a summary, calculation, or visual analysis). Break it down into subtasks to identify tool needs. For example, if your task involves researching current events, note that you'll need search and browsing capabilities. Step 2: Review Available Tools Familiarize yourself with the tools Grok can access. Here's a quick overview: - Code Execution: Run Python code for calculations, data processing, or simulations using libraries like numpy, pandas, or sympy. - Browse Page: Fetch and summarize content from any website URL with custom instructions. - Web Search: Perform general internet searches, returning results with optional operators like site:. - Web Search With Snippets: Get quick, detailed excerpts from search results for fact-checking. - X Keyword Search: Advanced search for X posts using operators like from:, since:, or filter:. - X Semantic Search: Find semantically related X posts based on a query, with filters for dates or users. - X User Search: Locate X users by name or handle. - X Thread Fetch: Retrieve a full X post thread, including context like replies and parents. - View Image: Analyze an image from a URL or conversation ID. - View X Video: Extract frames and subtitles from an X-hosted video. - Search PDF Attachment: Query a PDF file for relevant pages using keyword or regex modes. - Browse PDF Attachment: View specific pages of a PDF with text and screenshots. Select tools that align with your task. Aim for a mix to handle data gathering, processing, and visualization. Step 3: Craft Your Prompt Write a detailed prompt to Grok describing the task. Include: - The overall goal. - Specific steps or subtasks. - References to tools if you want to guide the process (e.g., "Use web_search to find sources, then code_execution to analyze data"). - Any constraints, like dates or limits. Example prompt: "Create a Grok Task for my morning routine: Search recent X posts about tech news using x_keyword_search, fetch a key thread with x_thread_fetch, and summarize with browse_page on linked articles." Step 4: Submit and Interact Send your prompt to Grok. It will process the task by calling tools as needed, often in parallel for efficiency. Review the output and refine with follow-up prompts if required (e.g., "Expand on that using view_image for visuals"). Iterate to fine-tune the workflow for reuse. Step 5: Save and Reuse Once refined, note the prompt as a template for future use. You can adapt it for similar tasks, making Grok Tasks a habitual part of your day. Finding Grok Tasks To discover existing Grok Tasks or inspiration for new ones, use X searches with tools like x_keyword_search or x_semantic_search (e.g., query: "Grok Tasks examples" with mode: Latest). Browse community-shared threads via x_thread_fetch, or web_search for tutorials on xAI features. Prompt Grok directly: "Show me popular Grok Tasks for productivity." 1 of 3

Brian Roemmele

152,242 Aufrufe • vor 6 Monaten

I Built Tiktok for video games in 72hrs and It went so VIRAL that investors started hitting my dms but it gets even crazier.. because this $1M app idea just turned into a $100M app idea the #1 suggestion i got from you guys on X Was to add multiplayer or local playing with friends to the games Bc that’s what actually makes Tiktok so addictive It’s the interaction with other users. comments, likes, reposts it's all multi-player. but of course that is a complicated task to add to our little app here. very complicated... Thats when the co-founder of reached out And said they are woking on a multiplayer feature for Rork To make this process an extremely easy. This is why Rork is the #1 app builder right now as far as updates on the app →i've added 5 more games to the feed (mostly classics!) → Started implementing the multiplayer UI to invite friends/match-making if you have any ideas for games or for the app drop your suggestions below!

I Built Tiktok for video games in 72hrs and It went so VIRAL that investors started hitting my dms but it gets even crazier.. because this $1M app idea just turned into a $100M app idea the #1 suggestion i got from you guys on X Was to add multiplayer or local playing with friends to the games Bc that’s what actually makes Tiktok so addictive It’s the interaction with other users. comments, likes, reposts it's all multi-player. but of course that is a complicated task to add to our little app here. very complicated... Thats when the co-founder of reached out And said they are woking on a multiplayer feature for Rork To make this process an extremely easy. This is why Rork is the #1 app builder right now as far as updates on the app →i've added 5 more games to the feed (mostly classics!) → Started implementing the multiplayer UI to invite friends/match-making if you have any ideas for games or for the app drop your suggestions below!

Ernesto Lopez

74,985 Aufrufe • vor 6 Monaten

🚀 Introducing EgoExo Forge - built on top of Rerun, Gradio, and Hugging Face hub (I’ll be in San Francisco July 21–29 — if you’re into robotics, egocentric AI, large-scale data collection, or just want to chat, DM me!) In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some version of imitation learning from human examples will be the most scalable + clean way to train humanoid robots 🤖 (similar to what Tesla did for Full Self Driving). Teleop is too expensive to collect a large enough dataset in a reasonable manner, so passive collection via egocentric (and in certain cases, exocentric) views feels like the right bet. Over the past few months, I've been trying to build out the scaffolding for this and using Rerun as my underlying infrastructure. Data being collected needs to be easily inspectable + time series and rerun provides the right tooling for this. My goal is to first build out a ground truth representative dataset from already existing open source data, generate some reasonable baselines, and then go out and collect my own data that adheres to the defined schema. 🔍 Starting with open-source datasets 1. EgoDex from Apple 2. HOCap from Nvidia and the University of Texas at Dallas 3. Assembly101 from Meta All these different datasets have different sensor configurations + annotations, so my goal with egoexo-forge is to have one consistent labeling scheme + data layout. I built a data pipeline that aligns all of the different datasets in one general schema assuming the COCO133 keypoint layout that allows for exo+ego, ego only, or exo only Since the scaffolding is already there, it becomes MUCH easier to add other datasets. So the next ones that I'll be including are HD-EPIC kitchens dataset, HOT3D, and finally my own personal iPhone + insta360 go collection method. Once I have a diverse variety of datasets, I'll double down on what I believe to be the key algorithms required to make useful data for imitation learning 📊 1. Camera Pose estimation via SLAM/SFM for ego perspective (and automatic calibration for exo) 2. Human pose estimation for both egocentric + exocentric views 3. Metric 3D reconstruction + object tracking I'll be setting up reasonable open-source baselines for each of these to validate that these datasets work, and then finally try to use the generated datasets for some imitation learning via the pi0-lerobot repo I've been working on. I plan on making a blog post + providing more info on all of this in the near future so stay tuned

🚀 Introducing EgoExo Forge - built on top of Rerun, Gradio, and Hugging Face hub (I’ll be in San Francisco July 21–29 — if you’re into robotics, egocentric AI, large-scale data collection, or just want to chat, DM me!) In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some version of imitation learning from human examples will be the most scalable + clean way to train humanoid robots 🤖 (similar to what Tesla did for Full Self Driving). Teleop is too expensive to collect a large enough dataset in a reasonable manner, so passive collection via egocentric (and in certain cases, exocentric) views feels like the right bet. Over the past few months, I've been trying to build out the scaffolding for this and using Rerun as my underlying infrastructure. Data being collected needs to be easily inspectable + time series and rerun provides the right tooling for this. My goal is to first build out a ground truth representative dataset from already existing open source data, generate some reasonable baselines, and then go out and collect my own data that adheres to the defined schema. 🔍 Starting with open-source datasets 1. EgoDex from Apple 2. HOCap from Nvidia and the University of Texas at Dallas 3. Assembly101 from Meta All these different datasets have different sensor configurations + annotations, so my goal with egoexo-forge is to have one consistent labeling scheme + data layout. I built a data pipeline that aligns all of the different datasets in one general schema assuming the COCO133 keypoint layout that allows for exo+ego, ego only, or exo only Since the scaffolding is already there, it becomes MUCH easier to add other datasets. So the next ones that I'll be including are HD-EPIC kitchens dataset, HOT3D, and finally my own personal iPhone + insta360 go collection method. Once I have a diverse variety of datasets, I'll double down on what I believe to be the key algorithms required to make useful data for imitation learning 📊 1. Camera Pose estimation via SLAM/SFM for ego perspective (and automatic calibration for exo) 2. Human pose estimation for both egocentric + exocentric views 3. Metric 3D reconstruction + object tracking I'll be setting up reasonable open-source baselines for each of these to validate that these datasets work, and then finally try to use the generated datasets for some imitation learning via the pi0-lerobot repo I've been working on. I plan on making a blog post + providing more info on all of this in the near future so stay tuned

Pablo Vela

32,085 Aufrufe • vor 1 Jahr

$KNDX 🤖 Theres 3 big narratives that are sending coins left right and centre rn. 🚀 #AI, #Gamefi, & #NFTs 🔹Theres 50% mindshare for #AI. 🤖 🔹#GameFi mcap is hitting ATH's with #OfftheGrid, $XBG and $SUPER making spectacular moves. 🎮 🔹NFTs and the #Metaverse are making a strong comeback with $APE up 100% over the weekend. 🐵 What if there's a project that touches all these trending narratives with groundbreaking technology to disrupt all 3 of them? 🔥 💡- That's where $KNDX comes in. -💡 Kondux is a cutting-edge Web3 SaaS platform, combining NVIDIA’s Omniverse, AI, Blockchain, and dynamic NFTs to revolutionize secure asset management across industries. 👏 Their flagship product, kNFTs, are 3D digital assets usable across Metaverse and Gaming platforms, AR/VR/XR environments, and manufacturing applications. Kondux’s scalable model opens new revenue streams by enabling effective digital asset monetization. 💰 Kondux is the first Web3 project to integrate VFX pipelines with NVIDIA’s Omniverse and bringing it onto the Blockchain. ⛓️ It is also the only Web3 project with a *Select Status Partnership* with NVIDIA, operating under NVIDIA NDAs and working with them directly for more than 2 years. About their NVIDIA Integrations: 🤖 🔹There are three areas of the Kondux tech stack that coincide with three divisions of NVIDIA: 📡GDN (Graphics Delivery Network, the backbone of GeForce Now) 💡Omniverse for 3D aspects such as, geospatial data, real world physics, lighting, and raytracing 🤖NVIDIA AI Foundation, which covers many aspects of #AI, including inference and deployment scaling. The convergence of all these components lie within .USD file format . 🔹 They are the first blockchain project to integrate NVIDIA’s Omniverse Cloud and Graphics Delivery Network (GDN) to provide high-quality 3D content accessible on any device without requiring high-end hardware. 🔹 This setup streamlines content management, democratises access to resource-intensive 3D content, and enables real-time interaction with 3D NFTs. Now, I haven’t seen any crypto project so deeply connected with NVIDIA and NVIDIA technology. GDN is a HUGE competitive advantage. With it, the need for #GPU’s basically goes out the window. 🤯 Now lets take a look at some of the other main features... 👀 OpenUSD (Universal Scene Description): 📽️ 🔹 Kondux is leveraging USD technology, developed by Pixar and used by Meta, Apple, Microsoft and other industry leaders to enhance 3D graphics and interoperability within its creative ecosystem. 🔹 Originally created for high-end film production, USD now supports a variety of applications, including gaming and virtual reality, making it a key asset for Kondux. kNFT's: 🎨 🔹 Kondux is pioneering a new category of NFTs known as kNFTs, which aim to redefine NFT utility through innovative features. 🔹 A standout feature is the upgradeable aspect provided by Kondux DNA, allowing kNFTs to transform and combine with other NFTs, creating limitless possibilities in art, gaming, and music. 🔹Through the Kondux AI portal it will be possible to communicate with kNFTs. They can learn and adapt. This AI technology is revolutionary because it makes human to kNFT interaction possible, turning it into a unique, personalized experience. Check out the clip of kNFTs in Unreal Engine 5 gameplay below. 👇 Kondux is a very obvious utility play with huge upside because it’s multi narrative. 📈 It's seriously groundbreaking stuff that they’re about to launch. 🚀 After speaking with the team there’s no doubt in my mind this will do crazy big numbers in the next months. 🤑

$KNDX 🤖 Theres 3 big narratives that are sending coins left right and centre rn. 🚀 #AI, #Gamefi, & #NFTs 🔹Theres 50% mindshare for #AI. 🤖 🔹#GameFi mcap is hitting ATH's with #OfftheGrid, $XBG and $SUPER making spectacular moves. 🎮 🔹NFTs and the #Metaverse are making a strong comeback with $APE up 100% over the weekend. 🐵 What if there's a project that touches all these trending narratives with groundbreaking technology to disrupt all 3 of them? 🔥 💡- That's where $KNDX comes in. -💡 Kondux is a cutting-edge Web3 SaaS platform, combining NVIDIA’s Omniverse, AI, Blockchain, and dynamic NFTs to revolutionize secure asset management across industries. 👏 Their flagship product, kNFTs, are 3D digital assets usable across Metaverse and Gaming platforms, AR/VR/XR environments, and manufacturing applications. Kondux’s scalable model opens new revenue streams by enabling effective digital asset monetization. 💰 Kondux is the first Web3 project to integrate VFX pipelines with NVIDIA’s Omniverse and bringing it onto the Blockchain. ⛓️ It is also the only Web3 project with a Select Status Partnership with NVIDIA, operating under NVIDIA NDAs and working with them directly for more than 2 years. About their NVIDIA Integrations: 🤖 🔹There are three areas of the Kondux tech stack that coincide with three divisions of NVIDIA: 📡GDN (Graphics Delivery Network, the backbone of GeForce Now) 💡Omniverse for 3D aspects such as, geospatial data, real world physics, lighting, and raytracing 🤖NVIDIA AI Foundation, which covers many aspects of #AI, including inference and deployment scaling. The convergence of all these components lie within .USD file format . 🔹 They are the first blockchain project to integrate NVIDIA’s Omniverse Cloud and Graphics Delivery Network (GDN) to provide high-quality 3D content accessible on any device without requiring high-end hardware. 🔹 This setup streamlines content management, democratises access to resource-intensive 3D content, and enables real-time interaction with 3D NFTs. Now, I haven’t seen any crypto project so deeply connected with NVIDIA and NVIDIA technology. GDN is a HUGE competitive advantage. With it, the need for #GPU’s basically goes out the window. 🤯 Now lets take a look at some of the other main features... 👀 OpenUSD (Universal Scene Description): 📽️ 🔹 Kondux is leveraging USD technology, developed by Pixar and used by Meta, Apple, Microsoft and other industry leaders to enhance 3D graphics and interoperability within its creative ecosystem. 🔹 Originally created for high-end film production, USD now supports a variety of applications, including gaming and virtual reality, making it a key asset for Kondux. kNFT's: 🎨 🔹 Kondux is pioneering a new category of NFTs known as kNFTs, which aim to redefine NFT utility through innovative features. 🔹 A standout feature is the upgradeable aspect provided by Kondux DNA, allowing kNFTs to transform and combine with other NFTs, creating limitless possibilities in art, gaming, and music. 🔹Through the Kondux AI portal it will be possible to communicate with kNFTs. They can learn and adapt. This AI technology is revolutionary because it makes human to kNFT interaction possible, turning it into a unique, personalized experience. Check out the clip of kNFTs in Unreal Engine 5 gameplay below. 👇 Kondux is a very obvious utility play with huge upside because it’s multi narrative. 📈 It's seriously groundbreaking stuff that they’re about to launch. 🚀 After speaking with the team there’s no doubt in my mind this will do crazy big numbers in the next months. 🤑

Altcoin Miyagi🇯🇵

17,303 Aufrufe • vor 1 Jahr