MLLMs are great at understanding videos, but struggle with... spatial reasoning—like estimating distances or tracking objects across time. the bottleneck? getting precise 3D spatial annotations on real videos is expensive and error-prone. introducing SIMS-V 🤖 [1/n]show more

Google 3D Maps aren’t as accurate as you think.... Everyone knows Google dominates mapping—but here’s what they won’t tell you. Their 3D maps are built with car-mounted cameras, capturing images every ~10 meters. That’s fine for basic navigation, but it’s nowhere near enough for AR, robotics, autonomous systems, or AI-driven spatial intelligence. Now, imagine a map so detailed it captures the world at sub-5cm accuracy. That’s OVRMaps. 🌐 OVER 3D maps are built with 400-1,000 images per 300 sqm, taken from multiple angles at pedestrian level. The result? A new era of hyper-precise localization that changes everything. Why does this matter? ↳ VPS that actually works – Real-world AR anchoring, digital twins, and AI-powered spatial computing with pinpoint accuracy. ↳ Next-gen precision – Essential for ride-sharing, robotics, smart cities, and asset tracking—where even a small error makes a huge difference. ↳ AI-Driven Spatial Intelligence – Our maps fuel Large Geo-Spatial Models (LGMs), the AI revolution enabling machines to understand, navigate, and interact with the real world. ↳ The Spatial Computing revolution – Merging immersive experiences with real-world precision, unlocking applications beyond what Google’s dataset can support. OVRMaps are built for the future. 🌍 Start mapping today:show more

I can't believe that Nvidia looked at this "AI... on top of games filter" and said to themselves this is the future of gaming. Like it or not, this is where Nvidia is heading and they're calling it neural rendering with DLSS 5. The examples they've showed reminded me a lot of those AI generated filter videos on top of GTA V, except that now it is supposed to run in real-time. Honestly, I don't like this current look at allshow more

Selling candles on Etsy, Amazon, Shopify, or eBay? Then... you know the struggle — product videos are expensive and a huge time sink. I tried Pollo AI's "Photo to Video Ads" feature right inside the Mobile APP with just one candle photo. A few minutes later, that single image turned into a cinematic, cozy video ad that looked ready to go live. No studio. No crew. No editing skills. All from my phone. At this point, Pollo Agent honestly feels like my creative team. AI-generated product ads are getting scary good.show more

Today, we released Lyra 2.0, a framework for generating... persistent, explorable 3D worlds at scale, from NVIDIA Research. Generating large-scale, complex environments is difficult for AI models. Current models often “forget” what spaces look like and lose track of movement over time, causing objects to shift, blur, or appear inconsistent. This prevents them from creating the reliable 3D environments required for downstream simulations. Lyra 2.0 solves these issues by: ✅ Maintaining per-frame 3D geometry to retrieve past frames and establish spatial correspondences ✅ Using self-augmented training to correct its own temporal drifting. Lyra 2.0 turns an image into a 3D world you can walk through, look back, and drop a robot into for real-time rendering, simulation, and immersive applications. ➡️ Learn more: 📄 Read the paper:show more

Robots can now reconstruct 3D scenes in real time... from a single RGB camera. [📍 Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like π³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: • Up to 30 FPS • 10 to 15x speedup • Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes • Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperial’s Dyson Robotics Lab and many others who contributed to this! 📍 Save projects page + paper for later: Video: ——- if it matters in AI or Robotics you'll read it here first:show more

Check out this Stereo4D paper from Google DeepMind. It's... a pretty clever approach to a persistent problem in computer vision -- getting good training data for how things move in 3D. The key insight is using VR180 videos -- those stereo fisheye videos we launched back in 2017 for YouTubeVR. It was always clear that structured stereo datasets would be valuable for computer vision -- and we launched some powerful VR tools with it back in 2017 (link below). But what's the game changer now in 2024 is the scale -- they're providing 110K high quality clips :-) That's the kind of massive, real-world AI dataset that was just a dream back then! They're using it to train this model called DynaDUSt3R that can predict both 3D structure and motion from video frames. Which means it tracks how objects move between frames while simultaneously reconstructing their 3D shape. And given we're dealing with real stereoscopic content, results are notably better than synthetic data, giving you a faithful rendition of the real-world with a diverse set of subject matter. It's one of those through lines when tackling a timeless mission like mapping the world or spatial computing -- VR content created for immersion becoming the foundation for teaching machines to understand how the world moves. Sometimes innovation chains together in unexpected ways! Links to projects below⛓️show more

This is some quietly impressive work on making video... world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.show more

Create a 3D model from a single image, set... of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”show more

Depth Any Video with Scalable Synthetic Data AI physicists... and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.show more

Does LLM really need to be a helpful assistant... all the time? No. If you want to simulate people, “perfectly helpful” could be the wrong objective. Meet OdysSim, a journey toward LLMs beyond assistants, as behavioral foundation models (10B tokens of real human behavior; 23 sim benchmarks, finally in one place. new open models: outperform or on par with GPT-5.5, Gemini 3.1, or Claude Opus 4.7 in many behavior-sim dimensions). Human behavior simulation is becoming essential. Agent evaluation needs realistic users before real users show up. Medical and classroom training need realistic patients and students. Social science needs synthetic participants at scale. But real people are not ideal assistants. Real patients panic or ignore good advice. Real students misunderstand. Real customers are vague, picky, impatient, or simply leave. Human behavior is messy, diverse, and often imperfect. Frontier LLMs are getting better at math, code, and long-horizon tasks. They are NOT getting better at simulating human behavior. If anything, they drift the other way: more assistant-ish, more homogeneous, fewer of the errors and quirks real humans show. This is no accident. The whole pipeline is built for helpfulness and task success, not behavioral realism. And you can't prompt your way out of that. So we rethink the recipe from scratch and release: 🧠 The OdysSim corpus: 21.4M real human interactions (~10B tokens) from 62 sources, every conversation retrofitted with social grounding (who is talking, and why) 📏 SOUL-Index: 23 human-behavior benchmarks unified into one suite across 5 axes 🤖 OSim-8B: open weights; tops more SOUL-Index benchmarks than any frontier model, acts more like a real user than any of them on τ-bench (nearly matching real humans in the reaction dimension), and writes far more human-like text along the way.show more

Streaming iPhone data in real-time directly to Rerun 🚀... The collection process is one of the most frustrating parts of building imitation-learning datasets. I’ve got a little army of sensors—📱 iPhone, iPad, Quest 3—but getting them temporally aligned, spatially aligned, AND seeing real-time feedback while recording is tough. I stumbled on a great library from Cake Lab (WPI) called ARFlow. It’s a thin client built on Unity’s ARFoundation that connects over gRPC to a server running Rerun for live data logging. I forked it to: - Log the SLAM translation poses, and - Upgrade rerun to v0.23 for my use case. So far, it works well, but there are still a few hitches: 1. Right now, it’s solid on iPhone and iPad; my Quest 3 client is still slow and not super reliable. 2. I’m using an older ARFlow branch focused on real-time streaming only—no spatial or temporal sync yet. Unity builds for iOS keep failing. 🛠️ 3. Nothing is saved locally to the client, so packet loss is a risk on shaky networks. There’s huge potential in tapping the ubiquitous sensors we carry around every day, and ARFlow is a big step toward making that easyshow more

I see a lot of people commenting without even... reading the caption or checking who I am. I’m a professional stunt rider, a movie stunt double and a content creator. I play characters, I create staged scenes because that is literally my job. Everything is planned, supervised and controlled. But even with all that, some people still rush into the comments to throw insults without understanding what they’re actually watching. Before getting angry, judging or making accusations, take a few seconds to look into what you’re seeing. It avoids a lot of unfairness and it shows respect for the people who spend time and energy creating entertaining content. And everything here is real work. It’s real riding, real effort and hours of preparation. There is never any artificial intelligence in my videos. I don’t blame anyone, but I kindly remind you to be careful with what you think you’re seeing before judging, because reality is not always what it looks like.show more

I've been researching Agents for the past 6 months... and collected 40+ materials on the most capable architectures & implementations. The intent was to publish a comprehensive overview, like I did on RAG techniques, but been too busy with so sharing it here. There are some great intro lectures by Andrew Ng to start with. The following types of Agentic architectures are covered: 🤖 Chain of thought (Plan & Execute agent) 🤖 Tooling operators (An agent upon a set of tools, routing to them) - good for connecting external data storage & APIs, pretty fast and robust 🤖 ReAct (Thought - Action - Observation) - capable of iteratively executing complex tasks or answering complex queries 🤖 Self-Reflection - (Action - Observation / Evaluation - Reflection - Planning) - adds some quality and reasoning clarity compared to the ReAct scheme, might be slower 🤖 Agent upon agents (A multiagent scheme) - a quite complex setting, slow, but capable of executing very complex multistep tasks, not super robust as loops are a frequent issue. Most successful projects: AutoGPT, AgentGPT, MemGPT, GPT-Researcher, CrewAI, MetaGPT. There are also some arXiv papers & blog posts on the most important architectures. 🔗 All the materials are here: 🧠 The best part is there is a co-pilot to chat with all this knowledge! If you’d like to add some valuable publications on Agents to this collection - just share a link in the comments 👇show more

MARVEL JUST PLUCKED $5,000,000 OUT OF THEIR VFX BUDGET... TO BUY OUT A 20-YEAR-OLD WHO REBUILT SPIDER-MAN IN 120 HOURS Disney spent $250,000,000 and forced 400 animators into crunch to render a masked, miserable hero. A broke student booted a single neural engine and built a maskless, overjoyed superhero in 5 days with zero render software. Marvel panicked, realized his pipeline beat their internal renders, and dropped a $5M buyout. Here is the exact technical stack that forced a $150B studio to write the check: > ZERO MOCAP HARDWARE - Driven by open-source spatial pose estimators. No physical tracking suits or clean plates required. > MID-AIR FACIAL LOCK - Locked consistent face features across high-speed lighting using a hyper-focused ControlNet depth harness. > LIQUIDATED CGI PIPELINES - Replaced 3D environments, manual keyframes, and render farms with real-time latent frame interpolation.show more

Introducing Kaleido💮 from AI at Meta — a universal... generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:show more

📍Theory of Space (accepted at #ICLR2026) Theory of Mind... → hidden mental states Theory of Space → hidden spatial beliefs from passive observers “What do I know?” to active explorers “What don’t I know, and how do I reduce that uncertainty?” Theory of Space is to evaluate if foundation models can actively construct, revise, and exploit internal spatial beliefs. We quantify Active-Passive Gap. Not just measure task accuracy, but how much uncertainty is reduced per step, and how many steps are needed in total for agents to build stable spatial beliefs. Exploration should prioritize information gain and reduce uncertainty per step. Instead, we observe LLMs/VLMs explore redundantly with stalled belief updates. Key findings: 1. Active agents perform worse than rule based programs 2. Cognitive Map Failures & Belief Drift (beliefs about previously observed objects degrades over time; new updates corrupt earlier correct perceptions) 3. Poor Vision Identification & Belief Inertia in Belief Revision Website: Code: Data: Theory of Space is a joint effort of Northwestern Engineering, Stanford AI Lab, Allen School, Cornell Computer Science. Led by the amazing WilliamZhang, jointly done with Zihan Huang, yue wang, Jieyu Zhang, Lester Xue, @wzihanw, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, Ranjay Krishna, Jiajun Wu, Fei-Fei Lishow more

2023 was the year of AI avatars 2024 was... the year of AI photos 2025 was the year of AI videos And I think it's becoming clear now that 2026 will be the year of AI world models Fully interactive explorable 3d worlds generated from one or multiple 2d images or a prompt In turn these 2d images can then be generated by AI too So soon you can generate fully explorable virtual 3d worlds based on your own imagination Next will be figuring out how to make those worlds interactive This is World Labs (unaffiliated, but I like it) As always a lot of big AI model companies are now working on the same thing: 3d world models, only World Labs has a real properly working demo (for now) Very exciting time again!show more

3D Gaussian Splatting for Real-Time Radiance Field Rendering paper... page: Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.show more