Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Spatial reconstruction is a long-context problem: real scenes come with hundreds of images. But O(N²) transformer-based models don’t scale efficiently. Introducing: 🤐ZipMap (CVPR ’26): Linear-Time, Stateful 3D Reconstruction via Test-Time Training (TTT). ZipMap “zips” a large image collection into an implicit TTT scene state in a single linear-time operation.... show more

Haian Jin

1,258 subscribers

80,294 Aufrufe • vor 7 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

29 Kommentare

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

🧵2/N Key to the speedup: we replace all global attention with TTT layers. Concretely, we first “zip” visual tokens into fast-weight MLPs by updating the weights with all tokens, then applying the updated MLPs back to the tokens. This update-then-apply mechanism plays a role analogous to vanilla bidirectional self-attention, while reducing the computational cost from O(N²) to O(N).

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

🧵3/N Fast speed without sacrificing quality: With careful designs of TTT, our model shows strong results across multiple benchmarks for both short and long sequence input, while being significantly faster. Left: Our runtime scales linearly; Given 750 images: <10s (ours) vs >200s (VGGT). Right: Our accuracy matches or surpasses quadratic-time SOTA (e.g., VGGT, π³).

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

🧵4/N Stateful reconstruction: the updated TTT fast weight can be viewed as an implicit scene state. In our paper, we demonstrate the benefits of having a stateful representation in real-time scene state querying and its extension to sequential streaming reconstruction.

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

For more results and technical details, please visit our project page: This work was done while I was a student researcher at @GoogleDeepMind. I’m very grateful for the inspiring research environment and the invaluable support from my collaborators @ChrisWu6080, @tianyuanzhang99, @RuiqiGao, @jon_barron, @Jimantha, and @holynski_.

Profilbild von ZotGuard

ZotGuardvor 7 Monaten

VERY COOL!

Profilbild von Mayank Bhaskar

Mayank Bhaskarvor 7 Monaten

Congratulations on the @CVPR 2026 acceptance! Would you or any of your co-authors like to present your papers in the @Cohere_Labs Computer Vision community? Cohere Labs Community Page: Here's the playlist to previous talks, if you are interested:

Profilbild von dailypapers.io

dailypapers.iovor 7 Monaten

this is great work

Profilbild von Aryan Pandit ✈️ ECCV '26 🇸🇪

Aryan Pandit ✈️ ECCV '26 🇸🇪vor 7 Monaten

Great work, seems really interesting!

Profilbild von Ang Cao

Ang Caovor 7 Monaten

Very cool work

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

Thank you, Ang!

Profilbild von ζ Pedram ζ

ζ Pedram ζvor 7 Monaten

Woah

Profilbild von Alexandre Devaux

Alexandre Devauxvor 7 Monaten

Brilliant work! Looking forward to trying the code 🔥

Profilbild von 🎶

🎶vor 7 Monaten

Cool

Profilbild von Yutang Lin

Yutang Linvor 7 Monaten

Great work, congrats!

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

Thank you, Yutang!

Profilbild von Wally

Wallyvor 7 Monaten

Very cool! Been exploring AI-generated impostor pipelines for real-time character rendering and the main pain point is view interpolation artifacts. This looks it could sidestep that entirely, looking forward to play around with the model!

Profilbild von 实践哥 Li

实践哥 Livor 7 Monaten

I love the zip idea

Profilbild von Brad Dangerfield

Brad Dangerfieldvor 7 Monaten

Sweet!

Profilbild von Utkarsh Mishra

Utkarsh Mishravor 7 Monaten

This is one of the most amazing things I've ever seen!

Profilbild von Sam Abdi

Sam Abdivor 7 Monaten

Seems really interesting and promising. Good work.

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

Thank you, Sam!

Profilbild von Daniel Sungho Jung

Daniel Sungho Jungvor 7 Monaten

May i ask if there is any reason why robot and human motion were in main qualitative results? Any advantage of ZipMap in such scenarios?

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

Hi! We put them to just show how our methods work for dynamic scenes. We don’t mean to specifically emphasize it.

Profilbild von Daniel Sungho Jung

Daniel Sungho Jungvor 7 Monaten

Thanks for the reply! I was curious as I am doing the tasks. Looking forward to visiting your poster at CVPR2026 :)

Profilbild von Nigel Higgs

Nigel Higgsvor 7 Monaten

Hit me with an update when I can use the code

Profilbild von Haian Jin

Haian Jinvor 7 Monaten

They will be released soon!

Profilbild von Nigel Higgs

Nigel Higgsvor 7 Monaten

Awesome 👏🏽 just curious how well do you think this would work on video game footage?

Profilbild von Artem Sawistki

Artem Sawistkivor 7 Monaten

That's dope! Can't wait to see the code

Profilbild von MistFaccoco

MistFaccocovor 6 Monaten

nice work!

Ähnliche Videos

Mind blown by this new CVPR 2026 paper: tttLRM🤯 It introduces a Test-Time Training layer for long-context autoregressive 3D reconstruction with linear complexity. Multiple image observations are compressed into fast weights that keep updating during inference, forming an implicit scene representation that can be decoded into explicit formats such as 3D Gaussian Splats. Progressive 3D reconstruction from streaming inputs is getting wild🥳It’s interesting to see how AI inference and real-world capture are starting to converge.

Mind blown by this new CVPR 2026 paper: tttLRM🤯 It introduces a Test-Time Training layer for long-context autoregressive 3D reconstruction with linear complexity. Multiple image observations are compressed into fast weights that keep updating during inference, forming an implicit scene representation that can be decoded into explicit formats such as 3D Gaussian Splats. Progressive 3D reconstruction from streaming inputs is getting wild🥳It’s interesting to see how AI inference and real-world capture are starting to converge.

KIRI Engine - 3D Scanner App

16,949 Aufrufe • vor 7 Monaten

Glob3R: Global Structure-from-Motion With 3D Foundation Models HKUST Spatial Artificial Intelligence Lab · Alibaba Group · Nanjin University · Fudan University Glob3R is a scalable global 3D reconstruction framework built on geometric foundation models. It converts dense feed-forward predictions into reliable multi-view tracks and jointly refines camera poses and scene geometry through motion averaging and bundle adjustment. Glob3R improves reconstruction accuracy and consistency on long sequences, large-scale scenes, and unordered image collections.

Glob3R: Global Structure-from-Motion With 3D Foundation Models HKUST Spatial Artificial Intelligence Lab · Alibaba Group · Nanjin University · Fudan University Glob3R is a scalable global 3D reconstruction framework built on geometric foundation models. It converts dense feed-forward predictions into reliable multi-view tracks and jointly refines camera poses and scene geometry through motion averaging and bundle adjustment. Glob3R improves reconstruction accuracy and consistency on long sequences, large-scale scenes, and unordered image collections.

Ryohei Sasaki@engineer

18,157 Aufrufe • vor 3 Monaten

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

Tencent Hy

20,792 Aufrufe • vor 7 Monaten

Want to reconstruct very large-scale scenes accurately? Check out our Scal3R (CVPR'26 Highlight) — scaling up VGGT with TTT Layers: Split image sequences into chunks, compute TTT gradients per chunk, and then average gradients across chunks. Code is out:

Want to reconstruct very large-scale scenes accurately? Check out our Scal3R (CVPR'26 Highlight) — scaling up VGGT with TTT Layers: Split image sequences into chunks, compute TTT gradients per chunk, and then average gradients across chunks. Code is out:

Sida Peng

52,286 Aufrufe • vor 6 Monaten

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,934 Aufrufe • vor 1 Jahr

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

Stan Szymanowicz

33,180 Aufrufe • vor 7 Monaten

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

AK

179,061 Aufrufe • vor 3 Jahren

InSpatio-WorldFM. transforms single photos into multi-view consistent 3D worlds. - Explicit 3D anchors + implicit neural state. - Zero-drift spatial reasoning. Real-time interactive exploration.

InSpatio-WorldFM. transforms single photos into multi-view consistent 3D worlds. - Explicit 3D anchors + implicit neural state. - Zero-drift spatial reasoning. Real-time interactive exploration.

Wildminder

10,808 Aufrufe • vor 7 Monaten

Introducing VGGT (CVPR'25), a feedforward Transformer that directly infers all key 3D attributes from one, a few, or hundreds of images, in seconds! No expensive optimization needed, yet delivers SOTA results for: ✅ Camera Pose Estimation ✅ Multi-view Depth Estimation ✅ Dense Point Cloud Reconstruction ✅ Point Tracking Project Page: Code & Weights:

Introducing VGGT (CVPR'25), a feedforward Transformer that directly infers all key 3D attributes from one, a few, or hundreds of images, in seconds! No expensive optimization needed, yet delivers SOTA results for: ✅ Camera Pose Estimation ✅ Multi-view Depth Estimation ✅ Dense Point Cloud Reconstruction ✅ Point Tracking Project Page: Code & Weights:

Jianyuan

203,723 Aufrufe • vor 1 Jahr

VAST AI releases Triplane Meets Gaussian Splatting on Hugging Face Fast and Generalizable Single-View 3D Reconstruction with Transformers demo: TGS enables fast reconstruction from single-view image. It builds the 3D representation upon a hybrid Triplane-Gaussian representation by evaluating a transformer-based framework, from which 3D Gaussians would be decoded

VAST AI releases Triplane Meets Gaussian Splatting on Hugging Face Fast and Generalizable Single-View 3D Reconstruction with Transformers demo: TGS enables fast reconstruction from single-view image. It builds the 3D representation upon a hybrid Triplane-Gaussian representation by evaluating a transformer-based framework, from which 3D Gaussians would be decoded

AK

113,242 Aufrufe • vor 2 Jahren

Introducing SAM 3D, the newest addition to the SAM collection, bringing common sense 3D understanding of everyday images. SAM 3D includes two models: 🛋️ SAM 3D Objects for object and scene reconstruction 🧑‍🤝‍🧑 SAM 3D Body for human pose and shape estimation Both models achieve state-of-the-art performance transforming static 2D images into vivid, accurate reconstructions. 🔗 Learn more:

Introducing SAM 3D, the newest addition to the SAM collection, bringing common sense 3D understanding of everyday images. SAM 3D includes two models: 🛋️ SAM 3D Objects for object and scene reconstruction 🧑‍🤝‍🧑 SAM 3D Body for human pose and shape estimation Both models achieve state-of-the-art performance transforming static 2D images into vivid, accurate reconstructions. 🔗 Learn more:

AI at Meta

861,594 Aufrufe • vor 11 Monaten

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

📢Happy to present Convex Splatting, a novel way for 3D reconstruction based on 3D smooth convexes. For the first time, a splatting-based method reaches the quality of NeRF sota methods but with real-time rendering and few primitives!! I expect this to replace Gaussian splatting for 3D in the coming months. CODE RELEASED TODAY! joint work with collaborators from Université de Liège Visual Geometry Group (VGG) , KAUST Computer Vision Lab (IVUL) a thread 🧵 1/n

Abdullah Hamdi

71,661 Aufrufe • vor 1 Jahr

Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where we make linear attention sparse in space, recurrent in time, and parallel in training, enabling the first purely-linear-attention-based neural network for asynchronous object detection with #EventCameras, outperforming the previous best asynchronous method with 20x less computation with truly event-by-event inference on CPU! Code released! Paper: Code: Video: Event cameras promise extremely low-latency vision, but to fully exploit them, the neural network must be low-latency too. We introduce #SpatiallySparseLinearAttention (#SSLA) for asynchronous object detection directly from raw events. Linear attention is particularly appealing for event cameras: it can be trained efficiently in parallel on long event sequences, while at inference it operates recurrently, updating its prediction every time a new event arrives. The problem is that conventional linear attention updates its entire state for every event. For object detection, where fine spatial resolution matters, this quickly becomes expensive. Our key idea is simple: an event only carries information about a small spatial region, so why update the entire spatial state? SSLA updates only the relevant parts of the state, enabling fine-grained spatial representations while keeping per-event computation low. We achieve: - >20× lower per-event computation than the strongest prior asynchronous baseline - State-of-the-art accuracy among asynchronous object detection methods - Truly event-by-event inference on CPU, designed to preserve the latency advantage of event cameras Come to our poster on Friday September 11, 2026 from 4-6pm at ExHall #389 Reference: Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention ECCV, 2026 Prophesee SynSense University of Zurich UZH Science European Research Council (ERC) UZHai UZH IfI Tesla BYD #EventCameras #ComputerVision #Robotics #DeepLearning #NeuromorphicVision #AI

Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where we make linear attention sparse in space, recurrent in time, and parallel in training, enabling the first purely-linear-attention-based neural network for asynchronous object detection with #EventCameras, outperforming the previous best asynchronous method with 20x less computation with truly event-by-event inference on CPU! Code released! Paper: Code: Video: Event cameras promise extremely low-latency vision, but to fully exploit them, the neural network must be low-latency too. We introduce #SpatiallySparseLinearAttention (#SSLA) for asynchronous object detection directly from raw events. Linear attention is particularly appealing for event cameras: it can be trained efficiently in parallel on long event sequences, while at inference it operates recurrently, updating its prediction every time a new event arrives. The problem is that conventional linear attention updates its entire state for every event. For object detection, where fine spatial resolution matters, this quickly becomes expensive. Our key idea is simple: an event only carries information about a small spatial region, so why update the entire spatial state? SSLA updates only the relevant parts of the state, enabling fine-grained spatial representations while keeping per-event computation low. We achieve: - >20× lower per-event computation than the strongest prior asynchronous baseline - State-of-the-art accuracy among asynchronous object detection methods - Truly event-by-event inference on CPU, designed to preserve the latency advantage of event cameras Come to our poster on Friday September 11, 2026 from 4-6pm at ExHall #389 Reference: Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention ECCV, 2026 Prophesee SynSense University of Zurich UZH Science European Research Council (ERC) UZHai UZH IfI Tesla BYD #EventCameras #ComputerVision #Robotics #DeepLearning #NeuromorphicVision #AI

Davide Scaramuzza

52,943 Aufrufe • vor 2 Monaten

SIFU Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction Creating high-quality 3D models of clothed humans from single images for real-world applications is crucial.

SIFU Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction Creating high-quality 3D models of clothed humans from single images for real-world applications is crucial.

AK

49,541 Aufrufe • vor 2 Jahren

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

Matthias Niessner

18,447 Aufrufe • vor 5 Monaten

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,681 Aufrufe • vor 3 Jahren

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

AK

76,380 Aufrufe • vor 3 Jahren

(1/N) Will this be the BERT/GPT moment for 3D vision？ Finally, unsupervised pre-training for 3D works. Led by Qitao Zhao , we present E-RayZer — a fully self-supervised 3D reconstruction model that: 🔥Matches or surpasses supervised methods like VGGT 👀Learns transferable 3D representations, outperforming CroCo, VideoMAE, and DINO 📈Scales with more unlabeled data A new recipe for scalable 3D foundation models.

(1/N) Will this be the BERT/GPT moment for 3D vision？ Finally, unsupervised pre-training for 3D works. Led by Qitao Zhao , we present E-RayZer — a fully self-supervised 3D reconstruction model that: 🔥Matches or surpasses supervised methods like VGGT 👀Learns transferable 3D representations, outperforming CroCo, VideoMAE, and DINO 📈Scales with more unlabeled data A new recipe for scalable 3D foundation models.

Hanwen Jiang

58,767 Aufrufe • vor 10 Monaten

This is SO crazy - The new "Spatial Scenes" feature turns your photos into frozen moments in time — making Vision Pro feel more like a real-life time machine than ever before. You can quite literally step into these photos, since they’re not just 3D versions but actually fully spatial. That means you can now look behind objects and change your perspective. If this isn’t pure sci-fi or magic come to life, I don’t know what is. I can’t stop smiling — we’re living in the craziest of all timelines.

This is SO crazy - The new "Spatial Scenes" feature turns your photos into frozen moments in time — making Vision Pro feel more like a real-life time machine than ever before. You can quite literally step into these photos, since they’re not just 3D versions but actually fully spatial. That means you can now look behind objects and change your perspective. If this isn’t pure sci-fi or magic come to life, I don’t know what is. I can’t stop smiling — we’re living in the craziest of all timelines.

Phil Traut ᯅ

89,103 Aufrufe • vor 1 Jahr