Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as...

TencentHunyuan's profile picture

Tencent Hy

39,481 subscribers

20,792 Aufrufe • vor 5 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Spatial reconstruction is a long-context problem: real scenes come with hundreds of images. But O(N²) transformer-based models don’t scale efficiently. Introducing: 🤐ZipMap (CVPR ’26): Linear-Time, Stateful 3D Reconstruction via Test-Time Training (TTT). ZipMap “zips” a large image collection into an implicit TTT scene state in a single linear-time operation. The state will then be decoded into spatial outputs, and can be queried efficiently for novel-view geometry and appearance (~100 FPS) ZipMap is not only much faster (>20× faster than VGGT), but also matches or surpasses the accuracy of all SOTA models.
0:35

Spatial reconstruction is a long-context problem: real scenes come with hundreds of images. But O(N²) transformer-based models don’t scale efficiently. Introducing: 🤐ZipMap (CVPR ’26): Linear-Time, Stateful 3D Reconstruction via Test-Time Training (TTT). ZipMap “zips” a large image collection into an implicit TTT scene state in a single linear-time operation. The state will then be decoded into spatial outputs, and can be queried efficiently for novel-view geometry and appearance (~100 FPS) ZipMap is not only much faster (>20× faster than VGGT), but also matches or surpasses the accuracy of all SOTA models.

Haian Jin

79,033 Aufrufe • vor 5 Monaten

HunyuanWorld-Voyager is here and fully open-source! The world’s first ultra-long-range world model with native 3D reconstruction, redefining AI-driven spatial intelligence for VR, gaming, and simulations. ✅Direct 3D Output: Exports point cloud videos to 3D formats without tools like COLMAP, enabling instant 3D application use. ✅Innovative 3D Memory: Introduces a scalable world caching mechanism, ensuring geometric consistency across any camera trajectory. ✅Top-Ranked Performance: #1 on Stanford’s WorldScore, excelling in video generation and 3D reconstruction benchmarks.( Built on HunyuanWorld 1.0, Voyager blends video generation with 3D modeling, delivering camera-controlled, high-fidelity RGB-D sequences. Control scenes via keyboard or joystick for unmatched 3D consistency. Explore now: 🌐Project Page: 🔗GitHub: 🤗HuggingFace: 📝Technical Details:
0:37

HunyuanWorld-Voyager is here and fully open-source! The world’s first ultra-long-range world model with native 3D reconstruction, redefining AI-driven spatial intelligence for VR, gaming, and simulations. ✅Direct 3D Output: Exports point cloud videos to 3D formats without tools like COLMAP, enabling instant 3D application use. ✅Innovative 3D Memory: Introduces a scalable world caching mechanism, ensuring geometric consistency across any camera trajectory. ✅Top-Ranked Performance: #1 on Stanford’s WorldScore, excelling in video generation and 3D reconstruction benchmarks.( Built on HunyuanWorld 1.0, Voyager blends video generation with 3D modeling, delivering camera-controlled, high-fidelity RGB-D sequences. Control scenes via keyboard or joystick for unmatched 3D consistency. Explore now: 🌐Project Page: 🔗GitHub: 🤗HuggingFace: 📝Technical Details:

Tencent Hy

198,391 Aufrufe • vor 11 Monaten

🚀🚀🚀Introducing HY World 1.5 (WorldPlay)! We have now open-sourced the most systemized, comprehensive real-time world model framework in the industry. In HY World 1.5, we develop WorldPlay, a streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency, resolving the trade-off between speed and memory that limits current methods. You can generate and explore 3D worlds simply by inputting text or images. Walk, look around, and interact like you're playing a game. Highlights: 🔹Real-Time: Generates long-horizon streaming video at 24 FPS with superior consistency. 🔹Geometric Consistency: Achieved using a Reconstituted Context Memory mechanism to dynamically rebuild context from past frames to alleviate memory attenuation 🔹Robust Control: Uses a Dual Action Representation for robust response to user keyboard and mouse inputs. 🔹Versatile Applications: Supports both first-person and third-person perspectives, enabling applications like promptable events and infinite world extension. 👉🏻Try it now: 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical Report:
1:06

🚀🚀🚀Introducing HY World 1.5 (WorldPlay)! We have now open-sourced the most systemized, comprehensive real-time world model framework in the industry. In HY World 1.5, we develop WorldPlay, a streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency, resolving the trade-off between speed and memory that limits current methods. You can generate and explore 3D worlds simply by inputting text or images. Walk, look around, and interact like you're playing a game. Highlights: 🔹Real-Time: Generates long-horizon streaming video at 24 FPS with superior consistency. 🔹Geometric Consistency: Achieved using a Reconstituted Context Memory mechanism to dynamically rebuild context from past frames to alleviate memory attenuation 🔹Robust Control: Uses a Dual Action Representation for robust response to user keyboard and mouse inputs. 🔹Versatile Applications: Supports both first-person and third-person perspectives, enabling applications like promptable events and infinite world extension. 👉🏻Try it now: 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical Report:

Tencent Hy

189,802 Aufrufe • vor 8 Monaten

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.
0:40

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

MrNeRF

17,786 Aufrufe • vor 1 Jahr

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.
1:29

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

149,079 Aufrufe • vor 1 Jahr

Volumetric Capture is Hard There is no magic solution. Anyone interested in spatial R&D can get access to some awesome datasets online to test with, which can be especially interesting when processed with 3DGS and played back as 4DGS. However you need a ton of compute power. (multiple 3080, 3090+, RTX5000+ etc) The Necessity of Multiple Cameras for Volumetric Capture Consider, for instance, this very useful fixed rig dataset comprising 20 camera streams provided by Facebook Research Or this datasets comprising 40 camera streams provided by Google The attached video demonstrates FB's RAW 3DGS data played back in real-time in our custom Sibr viewer. It's noticable how quickly the quality can breakdown out of the camera group frustrum. Even at this quality it takes a lot of time and energy to process and playback. It's not simple. Let alone the expertise required by the original research team who captured the data. Doing this with 20 cameras is hard. 150+ cameras even harder. Now imagine trying to achieve it with a single camera phone to playback on a Meta Quest or Spatial headset. #volumetriccaptureishard #camerasdontseearoundcorners #spatial
1:16

Volumetric Capture is Hard There is no magic solution. Anyone interested in spatial R&D can get access to some awesome datasets online to test with, which can be especially interesting when processed with 3DGS and played back as 4DGS. However you need a ton of compute power. (multiple 3080, 3090+, RTX5000+ etc) The Necessity of Multiple Cameras for Volumetric Capture Consider, for instance, this very useful fixed rig dataset comprising 20 camera streams provided by Facebook Research Or this datasets comprising 40 camera streams provided by Google The attached video demonstrates FB's RAW 3DGS data played back in real-time in our custom Sibr viewer. It's noticable how quickly the quality can breakdown out of the camera group frustrum. Even at this quality it takes a lot of time and energy to process and playback. It's not simple. Let alone the expertise required by the original research team who captured the data. Doing this with 20 cameras is hard. 150+ cameras even harder. Now imagine trying to achieve it with a single camera phone to playback on a Meta Quest or Spatial headset. #volumetriccaptureishard #camerasdontseearoundcorners #spatial

Infinite-Realities

129,521 Aufrufe • vor 2 Jahren

Do Vision-Language Models represent space, and how? Spatial terms like "left" or "right" may not be enough to match images with spatial descriptions, as we often overlook the different frames of reference (FoR) used by speakers and listeners. See Figure 1 for examples! Introducing the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to assess the spatial reasoning capabilities of VLMs. COMFORT includes systematically designed datasets and metrics that evaluate model performance, and their deeper linguistic competence, specifically the spatial knowledge encoded in their internal representations. Find out more in the video teaser! Almost all VLMs prefer the egocentric relative FoR with reflected transform, similar to English. Yet, we reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. A shortened version will appear in Pluralistic Alignment Workshop Pluralistic Alignment Workshop #NeurIPS2024. It seems that the ArXiv moderators put it on hold and are eager to give it a thorough read first🤣! So here is the Paper/Code/Data: This collaboration turns out to be amazing, jointly led by Brian Zheyuan Zhang, @Hu_FY_ Jayjun Lee, with so many contributions and insights from Freda Shi, Parisa Kordjamshidi Michigan SLED Lab. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning!
0:30

Do Vision-Language Models represent space, and how? Spatial terms like "left" or "right" may not be enough to match images with spatial descriptions, as we often overlook the different frames of reference (FoR) used by speakers and listeners. See Figure 1 for examples! Introducing the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to assess the spatial reasoning capabilities of VLMs. COMFORT includes systematically designed datasets and metrics that evaluate model performance, and their deeper linguistic competence, specifically the spatial knowledge encoded in their internal representations. Find out more in the video teaser! Almost all VLMs prefer the egocentric relative FoR with reflected transform, similar to English. Yet, we reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. A shortened version will appear in Pluralistic Alignment Workshop Pluralistic Alignment Workshop #NeurIPS2024. It seems that the ArXiv moderators put it on hold and are eager to give it a thorough read first🤣! So here is the Paper/Code/Data: This collaboration turns out to be amazing, jointly led by Brian Zheyuan Zhang, @Hu_FY_ Jayjun Lee, with so many contributions and insights from Freda Shi, Parisa Kordjamshidi Michigan SLED Lab. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning!

Martin Ziqiao Ma

35,565 Aufrufe • vor 1 Jahr

NVIDIA AI Open-Sources ViPE (Video Pose Engine): A Powerful and Versatile 3D Video Annotation Tool for Spatial AI ViPE integrates bundle adjustment with dense optical flow, sparse keypoint tracking, and metric depth priors to estimate camera intrinsics, poses, and dense depth maps at 3–5 FPS on a single GPU. It significantly improves over prior uncalibrated pose estimation methods, achieving 18% and 50% error reduction on TUM and KITTI benchmarks, respectively, and shows robustness to dynamic scenes and diverse camera models. Beyond the method, the NVIDIA team also released a large-scale dataset comprising ~100K real-world internet videos, 1M AI-generated videos, and 2K panoramic videos (≈96M frames) annotated with metric depth and poses. This dataset and engine aim to accelerate training for spatial AI tasks such as 3D reconstruction, video generation, and robotics.... full analysis: paper: codes: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA Robotics NVIDIA AIDev NVIDIAdeveloper
1:13

NVIDIA AI Open-Sources ViPE (Video Pose Engine): A Powerful and Versatile 3D Video Annotation Tool for Spatial AI ViPE integrates bundle adjustment with dense optical flow, sparse keypoint tracking, and metric depth priors to estimate camera intrinsics, poses, and dense depth maps at 3–5 FPS on a single GPU. It significantly improves over prior uncalibrated pose estimation methods, achieving 18% and 50% error reduction on TUM and KITTI benchmarks, respectively, and shows robustness to dynamic scenes and diverse camera models. Beyond the method, the NVIDIA team also released a large-scale dataset comprising ~100K real-world internet videos, 1M AI-generated videos, and 2K panoramic videos (≈96M frames) annotated with metric depth and poses. This dataset and engine aim to accelerate training for spatial AI tasks such as 3D reconstruction, video generation, and robotics.... full analysis: paper: codes: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA Robotics NVIDIA AIDev NVIDIAdeveloper

Marktechpost AI Dev News ⚡

217,453 Aufrufe • vor 11 Monaten

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.
1:43

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

AK

40,485 Aufrufe • vor 2 Jahren

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

Loopy Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency paper page: With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion. In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.
0:45

Loopy Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency paper page: With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion. In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.

AK

128,852 Aufrufe • vor 1 Jahr

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, @YifanLu2024 , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at xiaojie@stanford.edu. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.
0:42

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, @YifanLu2024 , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at [email protected]. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.

evo-devo

42,268 Aufrufe • vor 9 Monaten

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.
0:50

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 Aufrufe • vor 3 Jahren

Introducing FoundationMotion. A large-scale, video-derived motion annotation dataset & auto-labeling pipeline + advanced models for motion understanding. Fully open-source: code, datasets, and models, free to use and build on. Understanding motion is core to physical reasoning, yet today’s leading models still struggle with simple spatial actions like “turn right” or “move up” or “flip the toast” - mainly due to the lack of large, fine-grained motion datasets. We present FoundationMotion, a fully automated pipeline that: • detects & tracks objects in videos • extracts trajectories • uses LLMs + frames to generate rich motion captions & QA pairs → creating large-scale, high-quality motion datasets at scale. After fine-tuning the open-source models Qwen and NVILA on our annotations, these models now outperform the closed-source Gemini-3-Flash and GPT-5.1 on spatial understanding tasks across autonomous driving, robotics, and everyday scenarios. 📜Paper: 🌐Webpage: 💻 Code: 🕸️Model: 📊 Dataset: 👉 Interactive Demo: Let’s move research forward together. FoundationMotion is also referred to as Wolf V2 🐺, the second chapter in the Wolf series:
1:25

Introducing FoundationMotion. A large-scale, video-derived motion annotation dataset & auto-labeling pipeline + advanced models for motion understanding. Fully open-source: code, datasets, and models, free to use and build on. Understanding motion is core to physical reasoning, yet today’s leading models still struggle with simple spatial actions like “turn right” or “move up” or “flip the toast” - mainly due to the lack of large, fine-grained motion datasets. We present FoundationMotion, a fully automated pipeline that: • detects & tracks objects in videos • extracts trajectories • uses LLMs + frames to generate rich motion captions & QA pairs → creating large-scale, high-quality motion datasets at scale. After fine-tuning the open-source models Qwen and NVILA on our annotations, these models now outperform the closed-source Gemini-3-Flash and GPT-5.1 on spatial understanding tasks across autonomous driving, robotics, and everyday scenarios. 📜Paper: 🌐Webpage: 💻 Code: 🕸️Model: 📊 Dataset: 👉 Interactive Demo: Let’s move research forward together. FoundationMotion is also referred to as Wolf V2 🐺, the second chapter in the Wolf series:

Boyi Li

66,999 Aufrufe • vor 8 Monaten

Beyoncé in her 2021 interview with Harper’s BAZAAR said: One of my favorite quotes is from the inventor Charles Kettering. It goes “Our imagination is the only limit to what we can hope to have in the FUTURE.” Kettering was the founder of Delco, and head of research at General Motors. Key aspects of Futurism were celebrating change, technical progress of the modern machine age, and new technology. Let me refresh everyone’s memory on how Beyoncé worked with Spatial Labs for Ivy Park, incorporating augmented reality. And the CEO of Spatial Labs teased Spatial Labs Music for 2023. In the same interview, she said “My wish for the FUTURE is to continue to do everything everyone thinks I can’t do.” Then she goes on to become the first ever Black woman to oversee the couture offering from an historic Parisian house in 2023. Olivier said that the making of RENAISSANCE COUTURE involved 3D-printing, complex constructions, mathematics, tech and engineering that went into the construction of each piece. And in the British Vogue July 2022 issue, the sheer contrast between Beyoncé sitting on a horse and on a motorcycle [styled like Niobe, played by Jada Pinkett Smith, who appeared in three sequels of The Matrix: Reloaded, Revolutions and Resurrections], hinting at Futurism and this being a 3-act project.
0:29

Beyoncé in her 2021 interview with Harper’s BAZAAR said: One of my favorite quotes is from the inventor Charles Kettering. It goes “Our imagination is the only limit to what we can hope to have in the FUTURE.” Kettering was the founder of Delco, and head of research at General Motors. Key aspects of Futurism were celebrating change, technical progress of the modern machine age, and new technology. Let me refresh everyone’s memory on how Beyoncé worked with Spatial Labs for Ivy Park, incorporating augmented reality. And the CEO of Spatial Labs teased Spatial Labs Music for 2023. In the same interview, she said “My wish for the FUTURE is to continue to do everything everyone thinks I can’t do.” Then she goes on to become the first ever Black woman to oversee the couture offering from an historic Parisian house in 2023. Olivier said that the making of RENAISSANCE COUTURE involved 3D-printing, complex constructions, mathematics, tech and engineering that went into the construction of each piece. And in the British Vogue July 2022 issue, the sheer contrast between Beyoncé sitting on a horse and on a motorcycle [styled like Niobe, played by Jada Pinkett Smith, who appeared in three sequels of The Matrix: Reloaded, Revolutions and Resurrections], hinting at Futurism and this being a 3-act project.

🐝

311,499 Aufrufe • vor 3 Jahren

Spatial AI is increasingly important, and the newest papers from #NVIDIAResearch, 3DGRT and 3DGUT, represent significant advancements in enabling researchers and developers to explore and innovate with 3D Gaussian Splatting techniques. 💎 3DGRT (Gaussian Ray Tracing) ➡️ - Enables secondary lighting effects such as refractions, shadows, depth of field, and mirrors. - Provides a foundation for further research in ray tracing-based rendering of Gaussian representations. - Achieves real-time rendering performance on modern GPUs. ✨ 3DGUT (Gaussain Unscented Transform) ➡️ - Support for distorted camera models and time-dependent effects like rolling shutter. - Enables phenomena such as reflections and refractions while retaining real-time efficiency. 📥 Code:
0:27

Spatial AI is increasingly important, and the newest papers from #NVIDIAResearch, 3DGRT and 3DGUT, represent significant advancements in enabling researchers and developers to explore and innovate with 3D Gaussian Splatting techniques. 💎 3DGRT (Gaussian Ray Tracing) ➡️ - Enables secondary lighting effects such as refractions, shadows, depth of field, and mirrors. - Provides a foundation for further research in ray tracing-based rendering of Gaussian representations. - Achieves real-time rendering performance on modern GPUs. ✨ 3DGUT (Gaussain Unscented Transform) ➡️ - Support for distorted camera models and time-dependent effects like rolling shutter. - Enables phenomena such as reflections and refractions while retaining real-time efficiency. 📥 Code:

NVIDIA AI Developer

27,760 Aufrufe • vor 1 Jahr

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease — a project I’m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ — including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didn’t build everything from scratch — we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: • scArches to build and map into a reference scRNA-seq atlas of human skin: • NicheCompass to identify and characterize spatial niches: • MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive — it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and I’m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and I’m especially excited for the community to see more of his work soon — stay tuned. The data and pre-trained models will be released soon. Preprint:
1:09

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease — a project I’m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ — including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didn’t build everything from scratch — we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: • scArches to build and map into a reference scRNA-seq atlas of human skin: • NicheCompass to identify and characterize spatial niches: • MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive — it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and I’m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and I’m especially excited for the community to see more of his work soon — stay tuned. The data and pre-trained models will be released soon. Preprint:

Mo Lotfollahi

11,813 Aufrufe • vor 4 Monaten

GeoLibre v1.2.0 is here! GeoLibre is a free and open-source, lightweight, cloud-native GIS platform for visualizing, exploring, and analyzing geospatial data. One application that runs everywhere: in your web browser, as a native desktop app, on your phone, and inside a Jupyter notebook. No account, no server, no cost. Everything runs locally and your data stays private. This release packs in 35+ pull requests of new capabilities. A few highlights: - Run SQL right in the browser. The SQL Workspace pairs DuckDB Spatial with a new in-browser PostGIS engine (PGlite), so you can query layers, local files, and remote URLs without a server. - A smarter attribute table. Add fields, run a field calculator, and explore your data with a built-in Charts panel (histogram, scatter, bar, line, and box plots). - More ways to add data. OpenStreetMap PBF extracts, Cloud-Optimized NetCDF/HDF via kerchunk, georeferenced video overlays, authenticated 3D Tiles, and a Layer builder for custom overlays. - Better visualization. Heatmap rendering, point clustering, and H3 hexagonal grids for spatial binning. - New analysis and routing. A Directions plugin, plus Spatial Join, Select by Value, and Select by Location vector tools. - Print and share. A print layout composer that exports your map to PNG or PDF. - Work faster. A command palette (Ctrl/Cmd + K), global keyboard shortcuts, and undo/redo for layer and style operations. - Built for everyone. New internationalization framework, an accessibility pass with automated axe checks, an installable offline-capable PWA web build, React error boundaries, and Playwright end-to-end tests. Try the live demo: Star it on GitHub: Docs and roadmap: Release notes: #GIS #OpenSource #Geospatial #MapLibre #WebGIS #DuckDB #GeoLibre
1:25

GeoLibre v1.2.0 is here! GeoLibre is a free and open-source, lightweight, cloud-native GIS platform for visualizing, exploring, and analyzing geospatial data. One application that runs everywhere: in your web browser, as a native desktop app, on your phone, and inside a Jupyter notebook. No account, no server, no cost. Everything runs locally and your data stays private. This release packs in 35+ pull requests of new capabilities. A few highlights: - Run SQL right in the browser. The SQL Workspace pairs DuckDB Spatial with a new in-browser PostGIS engine (PGlite), so you can query layers, local files, and remote URLs without a server. - A smarter attribute table. Add fields, run a field calculator, and explore your data with a built-in Charts panel (histogram, scatter, bar, line, and box plots). - More ways to add data. OpenStreetMap PBF extracts, Cloud-Optimized NetCDF/HDF via kerchunk, georeferenced video overlays, authenticated 3D Tiles, and a Layer builder for custom overlays. - Better visualization. Heatmap rendering, point clustering, and H3 hexagonal grids for spatial binning. - New analysis and routing. A Directions plugin, plus Spatial Join, Select by Value, and Select by Location vector tools. - Print and share. A print layout composer that exports your map to PNG or PDF. - Work faster. A command palette (Ctrl/Cmd + K), global keyboard shortcuts, and undo/redo for layer and style operations. - Built for everyone. New internationalization framework, an accessibility pass with automated axe checks, an installable offline-capable PWA web build, React error boundaries, and Playwright end-to-end tests. Try the live demo: Star it on GitHub: Docs and roadmap: Release notes: #GIS #OpenSource #Geospatial #MapLibre #WebGIS #DuckDB #GeoLibre

Qiusheng Wu

39,959 Aufrufe • vor 2 Monaten

How to think about the Agentic OS 8:10 - From early experiments to an entirely new OS 12:51 - Early bet on spatial UI and what it taught us 14:59 - The paradigm shift: from chatting with models to deploying agents 17:25 - Why siloed AI apps are a dead end 19:30 - Rethinking the browser for an agentic world 22:12 - The Agentic OS: browser + IDE + payments in one stack 24:33 - Unifying all data, compute, and software on one network 28:10 - Spatial interfaces: why the future isn't chat-based 31:38 - Demo: Exa MCP pulling live data straight into Notion 33:01 - Demo: Parallel browsers and CLIs composing together 35:22 - "There is no next IDE", they all collapse into one 35:47 - Memory as an open protocol, not a company lock-in 37:14 - Demo: How users control and shape agent memory 39:33 - Dynamic cognition: agents that learn and orchestrate across CLIs 41:47 - The Matrix: a Google-scale discovery layer for tools 46:10 - Programmable agents, fair-price auctions, and spot compute 49:40 - Agent-to-agent micropayments: thinking beyond the ad model 51:52 - Why we need the broadband infra for agentic commerce 58:30 - Closing remarks and the journey ahead [This was recorded on Aug 27th, 2025 in San Francisco]
1:04:50

How to think about the Agentic OS 8:10 - From early experiments to an entirely new OS 12:51 - Early bet on spatial UI and what it taught us 14:59 - The paradigm shift: from chatting with models to deploying agents 17:25 - Why siloed AI apps are a dead end 19:30 - Rethinking the browser for an agentic world 22:12 - The Agentic OS: browser + IDE + payments in one stack 24:33 - Unifying all data, compute, and software on one network 28:10 - Spatial interfaces: why the future isn't chat-based 31:38 - Demo: Exa MCP pulling live data straight into Notion 33:01 - Demo: Parallel browsers and CLIs composing together 35:22 - "There is no next IDE", they all collapse into one 35:47 - Memory as an open protocol, not a company lock-in 37:14 - Demo: How users control and shape agent memory 39:33 - Dynamic cognition: agents that learn and orchestrate across CLIs 41:47 - The Matrix: a Google-scale discovery layer for tools 46:10 - Programmable agents, fair-price auctions, and spot compute 49:40 - Agent-to-agent micropayments: thinking beyond the ad model 51:52 - Why we need the broadband infra for agentic commerce 58:30 - Closing remarks and the journey ahead [This was recorded on Aug 27th, 2025 in San Francisco]

Varun

22,963 Aufrufe • vor 4 Monaten

Nimo Infinity - Introducing the World’s First AI Browser for Productivity. Nimo We’re reinventing the GUI for AI. As Andrej Karpathy recently pointed out at Y Combinator AI Startup School, the GUI for AI hasn’t been invented yet. We filed a patent back in 2017 for an intelligent GUI for smartphones, but the technology wasn’t ready at the time. Since AI became real, all those ideas came rushing back. We started building Nimo Infinity over the past few months, thinking differently to design a rich, functional interface for all your AI Apps and Agents in a single canvas, with a deep focus on productivity. Our mission is to build an entirely new AI OS that is secure, transparent, and interactive. Today, we are taking the first step with an AI Browser. Key Highlights: 1. An Infinite Canvas for all your AI Apps & Agents 2. Memory and Context 3. Custom Chromium Engine We’ve spent the last 7 years working on Spatial OS and ultra-portable spatial computers. That experience has shaped how we approach the AI Browser with a completely new interface, purpose-built for AI. #AI #AIBrowser #GUI
0:54

Nimo Infinity - Introducing the World’s First AI Browser for Productivity. Nimo We’re reinventing the GUI for AI. As Andrej Karpathy recently pointed out at Y Combinator AI Startup School, the GUI for AI hasn’t been invented yet. We filed a patent back in 2017 for an intelligent GUI for smartphones, but the technology wasn’t ready at the time. Since AI became real, all those ideas came rushing back. We started building Nimo Infinity over the past few months, thinking differently to design a rich, functional interface for all your AI Apps and Agents in a single canvas, with a deep focus on productivity. Our mission is to build an entirely new AI OS that is secure, transparent, and interactive. Today, we are taking the first step with an AI Browser. Key Highlights: 1. An Infinite Canvas for all your AI Apps & Agents 2. Memory and Context 3. Custom Chromium Engine We’ve spent the last 7 years working on Spatial OS and ultra-portable spatial computers. That experience has shaped how we approach the AI Browser with a completely new interface, purpose-built for AI. #AI #AIBrowser #GUI

rohildev

27,673 Aufrufe • vor 1 Jahr