Загрузка видео...

Не удалось загрузить видео

На главную

Check out TriPlaneNet! From a single image, we predict EG3D latents & offsets, thus obtaining high-fidelity 3D models. Works in real time! Great work by our stellar MA student Ananta R. Bhattarai advised by Artem Sevastopolsky #WACV2024

MattNiessner's profile picture

Matthias Niessner

48,430 subscribers

28,867 просмотров • 2 лет назад •via X (Twitter)

#WACV2024
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.
0:12

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 просмотров • 1 год назад

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen
0:17

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen

Matthias Niessner

37,993 просмотров • 9 месяцев назад

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢 We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_pris
0:15

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢 We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_pris

Matthias Niessner

12,499 просмотров • 1 год назад

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.
0:18

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,062 просмотров • 1 год назад

📢MeshPad: Interactive Sketch-Conditioned Artist-Designed Mesh Generation and Editing📢 Users can interactively design 3D models just from a sketch-based interface - check out the demo :) We break down the design process into addition with an autoregressive generator and deletion operations enabled by a classifier. To speed-up predictions, we propose a mesh-specific speculator such that users get immediate within a few seconds. Project: Video: Great work by Haoxuan Li Ziya Erkoç Lei Li Daniele Sirigatti V. Rosov Angela Dai
0:19

📢MeshPad: Interactive Sketch-Conditioned Artist-Designed Mesh Generation and Editing📢 Users can interactively design 3D models just from a sketch-based interface - check out the demo :) We break down the design process into addition with an autoregressive generator and deletion operations enabled by a classifier. To speed-up predictions, we propose a mesh-specific speculator such that users get immediate within a few seconds. Project: Video: Great work by Haoxuan Li Ziya Erkoç Lei Li Daniele Sirigatti V. Rosov Angela Dai

Matthias Niessner

30,133 просмотров • 1 год назад

So, today we have fast SDF sculpting + real-time AI in Unbound Loop. Old news🥱 Coming up next: -quad-view generation from sculpted geometry -image tweaks via nano🍌 -tripo HD & Low-Poly 3D generation There's more in the upcoming release, but these three deserve a closer look: Quad-View Generation Most platforms offer some version of this, but Loop has a key advantage, your sculpted model is the reference. That means less guessing from the generator. Though it’s not 100% foolproof, like any AI I guess? (I should stop stating the obvious every time). Image Tweaks via Chat Select any generated image and ask for fixes or changes in real time. Works great on unintended quad-view hallucinations, but also handy for quick iterations. Swapping colors, tweaking details, removing elements. Tripo 3D generator Especially the low-poly model, it consistently delivered fantastic game-ready topology when we tried it with our real-time AI output. And it's super fast.
0:19

So, today we have fast SDF sculpting + real-time AI in Unbound Loop. Old news🥱 Coming up next: -quad-view generation from sculpted geometry -image tweaks via nano🍌 -tripo HD & Low-Poly 3D generation There's more in the upcoming release, but these three deserve a closer look: Quad-View Generation Most platforms offer some version of this, but Loop has a key advantage, your sculpted model is the reference. That means less guessing from the generator. Though it’s not 100% foolproof, like any AI I guess? (I should stop stating the obvious every time). Image Tweaks via Chat Select any generated image and ask for fixes or changes in real time. Works great on unintended quad-view hallucinations, but also handy for quick iterations. Swapping colors, tweaking details, removing elements. Tripo 3D generator Especially the low-poly model, it consistently delivered fantastic game-ready topology when we tried it with our real-time AI output. And it's super fast.

Andrea Intg.

15,132 просмотров • 2 месяцев назад

Sneak peek from the lab. Work in progress of the real-time face tracking, via iPhone, Clone v2 cinematic rig. Simple viewport capture, no lighting or shading. Effective expression capture is key for conveying emotions accurately with your Clones. Our new cinematic rig has been constructed with flexibility in mind. We've carefully refined the 52 blend shapes that drive the facial expressions, and the face controls themselves, to achieve a more natural look. We have more iteration cycles to go through before we hit our target but you can already see a great difference from v1. This upgrade opens up a world of possibilities, from high-fidelity dynamic VRMs adaptable to various platforms to virtual production capabilities and V-Tubing with improved face filters. Our goal is simple: To help eliminate technical barriers and provide a more seamless and inclusive utility for all of our community, all while ensuring quality remains at the core of everything we do.
0:19

Sneak peek from the lab. Work in progress of the real-time face tracking, via iPhone, Clone v2 cinematic rig. Simple viewport capture, no lighting or shading. Effective expression capture is key for conveying emotions accurately with your Clones. Our new cinematic rig has been constructed with flexibility in mind. We've carefully refined the 52 blend shapes that drive the facial expressions, and the face controls themselves, to achieve a more natural look. We have more iteration cycles to go through before we hit our target but you can already see a great difference from v1. This upgrade opens up a world of possibilities, from high-fidelity dynamic VRMs adaptable to various platforms to virtual production capabilities and V-Tubing with improved face filters. Our goal is simple: To help eliminate technical barriers and provide a more seamless and inclusive utility for all of our community, all while ensuring quality remains at the core of everything we do.

Jarlan Perez

49,892 просмотров • 2 лет назад

Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!
0:14

Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!

Anshu

29,342 просмотров • 3 месяцев назад

Due to overwhelming high demand, a number of orders from our first batch have unfortunately been moved to the second batch, which is now 90% ordered! Our manufacturers are hard at work to keep up with strong demand and we estimate that the second batch will now be sent the latter part of June. As part of this second batch we will update you by email with Fedex tracking numbers so you can see in real time where the mining rig is in the system. We apologize to those affected by this production demand, but rest assured us and the manufacturers are working hard to clear the backlog. Thank you for your patience and support!
0:16

Due to overwhelming high demand, a number of orders from our first batch have unfortunately been moved to the second batch, which is now 90% ordered! Our manufacturers are hard at work to keep up with strong demand and we estimate that the second batch will now be sent the latter part of June. As part of this second batch we will update you by email with Fedex tracking numbers so you can see in real time where the mining rig is in the system. We apologize to those affected by this production demand, but rest assured us and the manufacturers are working hard to clear the backlog. Thank you for your patience and support!

SolarX

27,821 просмотров • 2 лет назад

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.
0:13

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

MrNeRF

27,428 просмотров • 1 год назад

📢📢𝐍𝐞𝐑𝐒𝐞𝐦𝐛𝐥𝐞 𝐯𝟐 𝐃𝐚𝐭𝐚𝐬𝐞𝐭 𝐑𝐞𝐥𝐞𝐚𝐬𝐞📢📢 Head captures of 7.1MP from 16 cameras at 73fps: * More recordings (425 people) * Better color calibration * Convenient download scripts The new version of our dataset adds 156 participants for a total of 425 different people. In its entirety, the dataset provides now 65 million images from over 15 hours of diverse human facial expression performances. We improved the color consistency of the recorded images with a better color calibration procedure. As a result, 3D reconstructions with images from the NeRSemble dataset should become better and look more realistic. Finally, we made it much easier to download the recordings with our new download repository. It now just takes a single command to download all frontal hair shake videos of all participants or to download all recordings of a single participant. Check it out: Awesome work by Tobias Kirschstein Simon Giebenhain !!!
0:17

📢📢𝐍𝐞𝐑𝐒𝐞𝐦𝐛𝐥𝐞 𝐯𝟐 𝐃𝐚𝐭𝐚𝐬𝐞𝐭 𝐑𝐞𝐥𝐞𝐚𝐬𝐞📢📢 Head captures of 7.1MP from 16 cameras at 73fps: * More recordings (425 people) * Better color calibration * Convenient download scripts The new version of our dataset adds 156 participants for a total of 425 different people. In its entirety, the dataset provides now 65 million images from over 15 hours of diverse human facial expression performances. We improved the color consistency of the recorded images with a better color calibration procedure. As a result, 3D reconstructions with images from the NeRSemble dataset should become better and look more realistic. Finally, we made it much easier to download the recordings with our new download repository. It now just takes a single command to download all frontal hair shake videos of all participants or to download all recordings of a single participant. Check it out: Awesome work by Tobias Kirschstein Simon Giebenhain !!!

Matthias Niessner

12,165 просмотров • 1 год назад

MARVEL JUST PLUCKED $5,000,000 OUT OF THEIR VFX BUDGET TO BUY OUT A 20-YEAR-OLD WHO REBUILT SPIDER-MAN IN 120 HOURS Disney spent $250,000,000 and forced 400 animators into crunch to render a masked, miserable hero. A broke student booted a single neural engine and built a maskless, overjoyed superhero in 5 days with zero render software. Marvel panicked, realized his pipeline beat their internal renders, and dropped a $5M buyout. Here is the exact technical stack that forced a $150B studio to write the check: > ZERO MOCAP HARDWARE - Driven by open-source spatial pose estimators. No physical tracking suits or clean plates required. > MID-AIR FACIAL LOCK - Locked consistent face features across high-speed lighting using a hyper-focused ControlNet depth harness. > LIQUIDATED CGI PIPELINES - Replaced 3D environments, manual keyframes, and render farms with real-time latent frame interpolation.
0:15

MARVEL JUST PLUCKED $5,000,000 OUT OF THEIR VFX BUDGET TO BUY OUT A 20-YEAR-OLD WHO REBUILT SPIDER-MAN IN 120 HOURS Disney spent $250,000,000 and forced 400 animators into crunch to render a masked, miserable hero. A broke student booted a single neural engine and built a maskless, overjoyed superhero in 5 days with zero render software. Marvel panicked, realized his pipeline beat their internal renders, and dropped a $5M buyout. Here is the exact technical stack that forced a $150B studio to write the check: > ZERO MOCAP HARDWARE - Driven by open-source spatial pose estimators. No physical tracking suits or clean plates required. > MID-AIR FACIAL LOCK - Locked consistent face features across high-speed lighting using a hyper-focused ControlNet depth harness. > LIQUIDATED CGI PIPELINES - Replaced 3D environments, manual keyframes, and render farms with real-time latent frame interpolation.

Shadow Nick

186,772 просмотров • 1 месяц назад

AI Is Moving Beyond “Generating Videos” — Toward “Generating Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:
0:10

AI Is Moving Beyond “Generating Videos” — Toward “Generating Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:

雪踏乌云

113,347 просмотров • 2 месяцев назад

The term "continual learning" has become overloaded if you see it as an ML problem. One classic thread is about memorization: regularization-based continual learning methods, such as EWC, MAS, and SI, estimate which parameters mattered for previous tasks and resist changing them too much. One modern thread is about adaptation: test-time training and inference-time learning methods, such as TTT, adapt part of the model on the incoming test stream before making predictions. These are sometimes discussed as separate threads. But in modern scalable architectures, I think they are better seen as complementary constraints: a model that learns quickly at test time also benefits from a mechanism for deciding what not to forget. In our #ECCV2026 paper, we study this in large-scale 4D reconstruction: how to build fast spatial memory that can adapt over long observation streams while reducing collapse and forgetting. Instead of using fully plastic test-time updates, we stabilize fast-weight adaptation with an elastic prior that balances adaptation and memory. Key ideas: - Elastic Test-Time Training: Fisher-weighted consolidation for fast-weight updates - EMA anchor weights that provide a moving reference for stability - Chunk-by-chunk inference for long 3D/4D observation streams We show that this scales across large 3D/4D pretraining settings, including both LRM-style and LVSM-style models, and improves reconstruction across benchmarks including Stereo4D, NVIDIA, and DL3DV-140. We release model checkpoints across different design choices: resolution, post-training curriculum, and whether the model uses an explicit 4DGS intermediate representation. - Homepage: - Paper: - Code: - Models: This work is co-led with Xueyang Yu, contributed by Haoyu Zhen Yuncong Yang, and advised by Michigan SLED Lab Chuang Gan.
0:14

The term "continual learning" has become overloaded if you see it as an ML problem. One classic thread is about memorization: regularization-based continual learning methods, such as EWC, MAS, and SI, estimate which parameters mattered for previous tasks and resist changing them too much. One modern thread is about adaptation: test-time training and inference-time learning methods, such as TTT, adapt part of the model on the incoming test stream before making predictions. These are sometimes discussed as separate threads. But in modern scalable architectures, I think they are better seen as complementary constraints: a model that learns quickly at test time also benefits from a mechanism for deciding what not to forget. In our #ECCV2026 paper, we study this in large-scale 4D reconstruction: how to build fast spatial memory that can adapt over long observation streams while reducing collapse and forgetting. Instead of using fully plastic test-time updates, we stabilize fast-weight adaptation with an elastic prior that balances adaptation and memory. Key ideas: - Elastic Test-Time Training: Fisher-weighted consolidation for fast-weight updates - EMA anchor weights that provide a moving reference for stability - Chunk-by-chunk inference for long 3D/4D observation streams We show that this scales across large 3D/4D pretraining settings, including both LRM-style and LVSM-style models, and improves reconstruction across benchmarks including Stereo4D, NVIDIA, and DL3DV-140. We release model checkpoints across different design choices: resolution, post-training curriculum, and whether the model uses an explicit 4DGS intermediate representation. - Homepage: - Paper: - Code: - Models: This work is co-led with Xueyang Yu, contributed by Haoyu Zhen Yuncong Yang, and advised by Michigan SLED Lab Chuang Gan.

Martin Ziqiao Ma

34,012 просмотров • 3 месяцев назад

Introducing: Control Tower. After running agencies for over a decade, three problems were clear: 1. Task management tools aren't built for client work—the constant back-and-forth, the private and public feedback loops. 2. Financial tools don't understand service businesses. We need to know what's coming in, what clients owe, when they're paying, and how that stacks up against what we're paying our team. 3. We need easy tools to track and manage our team's capacity in real-time. But here's the thing—these tools need to work together. When you work with clients, your team's capacity dictates the projects you can take on. The projects you take on dictate your revenue. And the pace of those projects restarts the entire cycle. That's what Control Tower does. Today we're announcing the foundations: Tasks, financials, and capacity. Very soon we'll be introducing Chief—the AI layer that connects everything and works proactively to keep your team moving. We built this from the ground up to be agent-native. What we have today is good. It's not great yet. There's a lot of work to do, and a lot we haven't even shipped to our beta users. I'd love for you to check it out. Sign up for the beta—we're inviting people as quickly as we can and doing personal walkthroughs.
0:18

Introducing: Control Tower. After running agencies for over a decade, three problems were clear: 1. Task management tools aren't built for client work—the constant back-and-forth, the private and public feedback loops. 2. Financial tools don't understand service businesses. We need to know what's coming in, what clients owe, when they're paying, and how that stacks up against what we're paying our team. 3. We need easy tools to track and manage our team's capacity in real-time. But here's the thing—these tools need to work together. When you work with clients, your team's capacity dictates the projects you can take on. The projects you take on dictate your revenue. And the pace of those projects restarts the entire cycle. That's what Control Tower does. Today we're announcing the foundations: Tasks, financials, and capacity. Very soon we'll be introducing Chief—the AI layer that connects everything and works proactively to keep your team moving. We built this from the ground up to be agent-native. What we have today is good. It's not great yet. There's a lot of work to do, and a lot we haven't even shipped to our beta users. I'd love for you to check it out. Sign up for the beta—we're inviting people as quickly as we can and doing personal walkthroughs.

Hunter Hammonds

53,302 просмотров • 8 месяцев назад

A 19-year-old student from China, Zhang Wei, developed an AI radar and sold it to Hong Kong for $550,000 He created it using Claude, spending just $20 and a month on development He walked into the Hong Kong administration office with a flash drive and asked for just 5 minutes of their time. 30 minutes later, he walked out with a check for $550,000 The code, connected to a camera, detects speed in real time. If the speed exceeds the limit, Claude takes a video clip and identifies the owner by the car's license plate. The video and the fine are then automatically sent to the owner's email address Unlike a conventional radar that only takes a photo and doesn't always work, this AI radar eliminates disputes because it captures video and makes the process fully autonomous by sending out the fines on its own The article includes the ready-to-use configurations.
0:15

A 19-year-old student from China, Zhang Wei, developed an AI radar and sold it to Hong Kong for $550,000 He created it using Claude, spending just $20 and a month on development He walked into the Hong Kong administration office with a flash drive and asked for just 5 minutes of their time. 30 minutes later, he walked out with a check for $550,000 The code, connected to a camera, detects speed in real time. If the speed exceeds the limit, Claude takes a video clip and identifies the owner by the car's license plate. The video and the fine are then automatically sent to the owner's email address Unlike a conventional radar that only takes a photo and doesn't always work, this AI radar eliminates disputes because it captures video and makes the process fully autonomous by sending out the fines on its own The article includes the ready-to-use configurations.

Bober_smart

4,329,225 просмотров • 3 месяцев назад

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch
0:22

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,710 просмотров • 2 лет назад

A Closer Look at Alchemist AI v3.5🔎 In v1, one agent handled everything. You’d enter a prompt like “make a snake game” and get a single block of code—one input, one output. v2 introduced multiple agents with specialized roles: prompt refinement, frontend, backend, and review. It brought more structure, but the process was still linear. Each step produced a single output. With v3, a single agent could handle multiple tasks. It generated HTML, CSS, JavaScript, searched for assets, and more—all in one go, while maintaining full context. Now in v3.5, we combine both models. Multiple agents work in parallel, each capable of producing multiple outputs at once. The prompt compiler adapts based on what you're building. A 3D game? It prioritizes game engines, rendering, and mechanics. A website or 2D app? It shifts focus to relevant frameworks and tools. The tool handler can call several services simultaneously, feeding context-aware data into the code generator for real-time execution. From single-output generation to parallel, adaptive workflows. In the next breakdown, we’ll dive into the architecture behind our proprietary engine in v3.5.
0:11

A Closer Look at Alchemist AI v3.5🔎 In v1, one agent handled everything. You’d enter a prompt like “make a snake game” and get a single block of code—one input, one output. v2 introduced multiple agents with specialized roles: prompt refinement, frontend, backend, and review. It brought more structure, but the process was still linear. Each step produced a single output. With v3, a single agent could handle multiple tasks. It generated HTML, CSS, JavaScript, searched for assets, and more—all in one go, while maintaining full context. Now in v3.5, we combine both models. Multiple agents work in parallel, each capable of producing multiple outputs at once. The prompt compiler adapts based on what you're building. A 3D game? It prioritizes game engines, rendering, and mechanics. A website or 2D app? It shifts focus to relevant frameworks and tools. The tool handler can call several services simultaneously, feeding context-aware data into the code generator for real-time execution. From single-output generation to parallel, adaptive workflows. In the next breakdown, we’ll dive into the architecture behind our proprietary engine in v3.5.

ALCHEMIST AI 🔮

12,689 просмотров • 1 год назад

Art on Tezos Friday Recap 🔔 → Opening tonight: STILL WET at Galerie Met in Berlin (Mariannenstrasse 33), a group show of physical and digital works by 18 artists across several continents, treating painting as an expanded medium spanning canvas, screen, 3D, GIFs, and machine learning traces. Curated by Kika Nicolela and Ruizhe Liang in collaboration with objkt. Reception 18:00 to 21:00, running August 15 to September 5. 🇩🇪🖌️ → Kento announced the two Open Call winners whose works will be turned into physical cards and included in Season 01 packs: Adiw & 𝖒𝖆𝖈𝖇𝖊𝖙𝖍 👏 → ACTZ “Vestige” auctions (Art Commission Tezos): For July with ACTZ - Art Commissions, Jesse 🏴‍☠️, Scott, muhju & Tipi ᑎᐱ selected five artists each to create new unique works around the theme "Vestige". Highlights and some live/recent auctions were posted Aug 13 by Art on Tezos. → #TezosTuesday spotlighted Solarchi, a collector and artist with an architecture background, who committed to collecting 1,000 tez from community replies and their own digging. 🤝 → 𝘼𝘾𝘾𝙊𝙈𝙋𝘼𝙍𝙏'𝙎 THE OBSOLETE MATTERS drops August 25 on objkt, thirty artists turning the question of obsolescence back on digital art itself, across generative systems, AI, 3D, digital painting, code, and collage. Curated by グレベンシ and Ilya Bliznets. 🕰️💾 → Artists’ Week: August (TheTezosCommunity / TheTezosCommunity ) is running. It includes Discord games (Gartic, Name That Artist, Quiz, Poker), Support Artists Days (community buys), and a “Let’s Talk Art” space that happened on Aug 13. More activities continue through the weekend. Don't Miss: → Ephemera Sunset (community auction event) via Antrum - Live today. Artists are listing 24-hour auctions themed around sunsets / fleeting light. Search #EphemeraSunset or #Ephemera2026 on X or check recent auctions on objkt. → PRECEDURE by goldcat curated by anika meier via objkt labs, starting from a single hand-made charcoal drawing and running it through procedural constraints so authorship shifts from gesture to rules, with a nod to early computer artists like Vera Molnár. ✏️⚙️ → Post_Histories by Space Case 😵‍💫 curated by anika meier via objkt labs, a series of interactive media carousels by ADHD that borrows the grammar of the multi-image social post and redirects it toward the contextual window of art. → Infinito Mesh Painting by Chris Follows via bootloader: → 1.p.3.r.d.3.C.h.1.r.1.c.0. by p.i.x.e.l.l.a.r.e. via bootloader: → Fugue by TOCA ME via drop.art Ი𐑼 Gurjos by Bárbara Bezina via drop.art Ი𐑼 📽️ Frog battle zero by daïchi mori
0:16

Art on Tezos Friday Recap 🔔 → Opening tonight: STILL WET at Galerie Met in Berlin (Mariannenstrasse 33), a group show of physical and digital works by 18 artists across several continents, treating painting as an expanded medium spanning canvas, screen, 3D, GIFs, and machine learning traces. Curated by Kika Nicolela and Ruizhe Liang in collaboration with objkt. Reception 18:00 to 21:00, running August 15 to September 5. 🇩🇪🖌️ → Kento announced the two Open Call winners whose works will be turned into physical cards and included in Season 01 packs: Adiw & 𝖒𝖆𝖈𝖇𝖊𝖙𝖍 👏 → ACTZ “Vestige” auctions (Art Commission Tezos): For July with ACTZ - Art Commissions, Jesse 🏴‍☠️, Scott, muhju & Tipi ᑎᐱ selected five artists each to create new unique works around the theme "Vestige". Highlights and some live/recent auctions were posted Aug 13 by Art on Tezos. → #TezosTuesday spotlighted Solarchi, a collector and artist with an architecture background, who committed to collecting 1,000 tez from community replies and their own digging. 🤝 → 𝘼𝘾𝘾𝙊𝙈𝙋𝘼𝙍𝙏'𝙎 THE OBSOLETE MATTERS drops August 25 on objkt, thirty artists turning the question of obsolescence back on digital art itself, across generative systems, AI, 3D, digital painting, code, and collage. Curated by グレベンシ and Ilya Bliznets. 🕰️💾 → Artists’ Week: August (TheTezosCommunity / TheTezosCommunity ) is running. It includes Discord games (Gartic, Name That Artist, Quiz, Poker), Support Artists Days (community buys), and a “Let’s Talk Art” space that happened on Aug 13. More activities continue through the weekend. Don't Miss: → Ephemera Sunset (community auction event) via Antrum - Live today. Artists are listing 24-hour auctions themed around sunsets / fleeting light. Search #EphemeraSunset or #Ephemera2026 on X or check recent auctions on objkt. → PRECEDURE by goldcat curated by anika meier via objkt labs, starting from a single hand-made charcoal drawing and running it through procedural constraints so authorship shifts from gesture to rules, with a nod to early computer artists like Vera Molnár. ✏️⚙️ → Post_Histories by Space Case 😵‍💫 curated by anika meier via objkt labs, a series of interactive media carousels by ADHD that borrows the grammar of the multi-image social post and redirects it toward the contextual window of art. → Infinito Mesh Painting by Chris Follows via bootloader: → 1.p.3.r.d.3.C.h.1.r.1.c.0. by p.i.x.e.l.l.a.r.e. via bootloader: → Fugue by TOCA ME via drop.art Ი𐑼 Gurjos by Bárbara Bezina via drop.art Ი𐑼 📽️ Frog battle zero by daïchi mori

Tezos

13,195 просмотров • 1 месяц назад