Загрузка видео...

Не удалось загрузить видео

На главную

Check out PhysHead: Simulation-Ready Gaussian Head Avatars (CVPR 2026)! We introduce a layered head & hair model using strands + Gaussian splats, enabling physics-based animation from multi-view video. PS: We didn’t dare compute our bald versions… maybe you’re braver 🙂

GerardPonsMoll1's profile picture

Gerard Pons-Moll

6,235 subscribers

10,232 просмотров • 5 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 1

Фото профиля Gerard Pons-Moll
Gerard Pons-Moll5 месяцев назад

Kudos to @bernakabadayi,@ness_pirs,@w_zielonka,@JustusThies !

Похожие видео

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.
0:20

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,354 просмотров • 2 лет назад

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.
0:32

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.

MrNeRF

50,088 просмотров • 1 год назад

GaussianSpeech: Audio-Driven Gaussian Avatars Contributions: • The first transformer-based sequence model for audio-driven head animation synthesis of a lightweight 3DGS based avatar. By animating our optimized 3DGS avatar directly with our transformer model, we achieve temporally coherent animation sequences while characterizing fine-scale face details and speaker-specific style. • A new high-quality audio-video dataset, comprising high-resolution 16-view dataset of 6 native English speakers (Standard American & British). The dataset has a total of 2500 sequences, with overall recordings of ∼3.5 hours.
1:06

GaussianSpeech: Audio-Driven Gaussian Avatars Contributions: • The first transformer-based sequence model for audio-driven head animation synthesis of a lightweight 3DGS based avatar. By animating our optimized 3DGS avatar directly with our transformer model, we achieve temporally coherent animation sequences while characterizing fine-scale face details and speaker-specific style. • A new high-quality audio-video dataset, comprising high-resolution 16-view dataset of 6 native English speakers (Standard American & British). The dataset has a total of 2500 sequences, with overall recordings of ∼3.5 hours.

MrNeRF

11,472 просмотров • 1 год назад

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,324 просмотров • 2 лет назад

📢📢 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧𝐒𝐩𝐞𝐞𝐜𝐡: Audio-Driven Gaussian Avatars 📢📢 We synthesize photorealistic and 3D-consistent talking human head avatars driven directly from spoken audio. More specifically, we introduce an efficient 3DGS-based representation, combined with an audio-conditioned transformer, to generate realistic and high-quality animations. Our method can easily generalize to a variety of in the wild scenarios like songs, foreign languages etc. We further introduce a new multiview dataset of native English speakers with overall recording time of ∼3.5 hours. Project: Video: Great work by Shivangi, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai
0:33

📢📢 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧𝐒𝐩𝐞𝐞𝐜𝐡: Audio-Driven Gaussian Avatars 📢📢 We synthesize photorealistic and 3D-consistent talking human head avatars driven directly from spoken audio. More specifically, we introduce an efficient 3DGS-based representation, combined with an audio-conditioned transformer, to generate realistic and high-quality animations. Our method can easily generalize to a variety of in the wild scenarios like songs, foreign languages etc. We further introduce a new multiview dataset of native English speakers with overall recording time of ∼3.5 hours. Project: Video: Great work by Shivangi, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai

Matthias Niessner

22,693 просмотров • 1 год назад

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.
5:19

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

MrNeRF

34,782 просмотров • 1 год назад

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.
0:40

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

MrNeRF

17,811 просмотров • 2 лет назад

Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes Contributions: • We propose STORM, the first feed-forward, self-supervised method for fast and accurate reconstruction of dynamic 3D scenes from sparse, multi-timestep, posed camera images. • Our bottom-up framework aggregates and transforms per-frame 3D Gaussian Splats into a cohesive scene representation, enabling self-supervised motion estimation. Furthermore, we introduce motion tokens that capture common motion primitives and regularize motion predictions, facilitating dynamic motion group segmentation without explicit motion or correspondence supervision. • We present several enhancements for in-the-wild scenarios, including sky modeling, camera exposure inconsistency handling, large novel-view extrapolation, and fine-grained human motions reconstruction, making STORM well-suited for real-world applications.
0:31

Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes Contributions: • We propose STORM, the first feed-forward, self-supervised method for fast and accurate reconstruction of dynamic 3D scenes from sparse, multi-timestep, posed camera images. • Our bottom-up framework aggregates and transforms per-frame 3D Gaussian Splats into a cohesive scene representation, enabling self-supervised motion estimation. Furthermore, we introduce motion tokens that capture common motion primitives and regularize motion predictions, facilitating dynamic motion group segmentation without explicit motion or correspondence supervision. • We present several enhancements for in-the-wild scenarios, including sky modeling, camera exposure inconsistency handling, large novel-view extrapolation, and fine-grained human motions reconstruction, making STORM well-suited for real-world applications.

MrNeRF

53,292 просмотров • 1 год назад

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors, no HexPlane); 🔹 Uses a clean, minimal Transformer backbone; 🔹 Generalizes with fast, high-quality feedforward rendering at any view and infinite frame rate. Check out more interactive demos and scaling behaviors on our homepage/paper. 👉Website: 👉Paper:
2:30

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors, no HexPlane); 🔹 Uses a clean, minimal Transformer backbone; 🔹 Generalizes with fast, high-quality feedforward rendering at any view and infinite frame rate. Check out more interactive demos and scaling behaviors on our homepage/paper. 👉Website: 👉Paper:

Martin Ziqiao Ma

21,851 просмотров • 1 год назад

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!
0:30

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

96,334 просмотров • 9 месяцев назад

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito
0:38

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,818 просмотров • 1 год назад

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar
0:23

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,682 просмотров • 2 лет назад

Finally got our hands on Seedance 2.0 (through YouArt) and ran it head-to-head with Kling AI 3.0. Here’s what stood out: Seedance 2.0’s physics feel stronger right away. The movement just looks more natural and realistic, especially in bigger physical actions. It also handles continuity across multiple shots from a single prompt better than I expected, which makes scene building a lot easier. And on the audio side, the lip sync generally holds up better than Kling. But here’s the tradeoff: Both models struggle with fine details. Background faces and smaller elements still get a bit messy. Seedance currently outputs at 720p, while Kling 3.0 supports 1080p, so Kling has the edge in overall sharpness. Even with upscaling, Seedance doesn’t fully recover those tiny details. Verdict🏆: Seedance 2.0 is a real step forward, especially for motion realism, but it’s definitely not the “end of Hollywood.” Check out our latest tutorial where we show you how to get access to this new model and take a deep look at the quality of this new AI Video Generator:
2:18

Finally got our hands on Seedance 2.0 (through YouArt) and ran it head-to-head with Kling AI 3.0. Here’s what stood out: Seedance 2.0’s physics feel stronger right away. The movement just looks more natural and realistic, especially in bigger physical actions. It also handles continuity across multiple shots from a single prompt better than I expected, which makes scene building a lot easier. And on the audio side, the lip sync generally holds up better than Kling. But here’s the tradeoff: Both models struggle with fine details. Background faces and smaller elements still get a bit messy. Seedance currently outputs at 720p, while Kling 3.0 supports 1080p, so Kling has the edge in overall sharpness. Even with upscaling, Seedance doesn’t fully recover those tiny details. Verdict🏆: Seedance 2.0 is a real step forward, especially for motion realism, but it’s definitely not the “end of Hollywood.” Check out our latest tutorial where we show you how to get access to this new model and take a deep look at the quality of this new AI Video Generator:

Curious Refuge

58,989 просмотров • 6 месяцев назад

At Avalon we are building "Real-time creating" - the ability to generate gameplay ready persistent worlds prompted from text. While others are building real-time video world models, Avalon is building real-time world generation inside a fully playable, persistent multiplayer engine. Internally running at 3840×2180 at 60 FPS. Built on Unreal Engine. Multiplayer by default. Persistent by default. Gameplay-ready by default. This is not a video latent replay. Not a simulation of interaction. It is a real 3D world with physics, logic, and authoritative multiplayer state. Avalon is trained on proprietary Avalon interaction data and powered by a hybrid system that combines language understanding, 3D model generation, procedural systems, and structured gameplay logic synthesis. Players can walk through a live world and generate environments, assets, mechanics, and entirely new gameplay modes using natural language. We accomplish this through a combination of 3D model generation, game logic generation based on our proprietary systems, and AI driven world creation. While other players are inside it. Changes persist instantly. State is synchronized in real time. Creation happens inside the world, not outside of it. Describe a biome. Spawn a civilization. Create a survival mode. Build a dungeon crawler. Launch a new game inside the world. Avalon interprets intent and integrates it directly into the live multiplayer environment. This is not a world model predicting video. This is a gameplay engine that understands language. If you can describe it, you can build it. And others can walk into it instantly.
1:09

At Avalon we are building "Real-time creating" - the ability to generate gameplay ready persistent worlds prompted from text. While others are building real-time video world models, Avalon is building real-time world generation inside a fully playable, persistent multiplayer engine. Internally running at 3840×2180 at 60 FPS. Built on Unreal Engine. Multiplayer by default. Persistent by default. Gameplay-ready by default. This is not a video latent replay. Not a simulation of interaction. It is a real 3D world with physics, logic, and authoritative multiplayer state. Avalon is trained on proprietary Avalon interaction data and powered by a hybrid system that combines language understanding, 3D model generation, procedural systems, and structured gameplay logic synthesis. Players can walk through a live world and generate environments, assets, mechanics, and entirely new gameplay modes using natural language. We accomplish this through a combination of 3D model generation, game logic generation based on our proprietary systems, and AI driven world creation. While other players are inside it. Changes persist instantly. State is synchronized in real time. Creation happens inside the world, not outside of it. Describe a biome. Spawn a civilization. Create a survival mode. Build a dungeon crawler. Launch a new game inside the world. Avalon interprets intent and integrates it directly into the live multiplayer environment. This is not a world model predicting video. This is a gameplay engine that understands language. If you can describe it, you can build it. And others can walk into it instantly.

AVALON

65,084 просмотров • 7 месяцев назад

Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where we make linear attention sparse in space, recurrent in time, and parallel in training, enabling the first purely-linear-attention-based neural network for asynchronous object detection with #EventCameras, outperforming the previous best asynchronous method with 20x less computation with truly event-by-event inference on CPU! Code released! Paper: Code: Video: Event cameras promise extremely low-latency vision, but to fully exploit them, the neural network must be low-latency too. We introduce #SpatiallySparseLinearAttention (#SSLA) for asynchronous object detection directly from raw events. Linear attention is particularly appealing for event cameras: it can be trained efficiently in parallel on long event sequences, while at inference it operates recurrently, updating its prediction every time a new event arrives. The problem is that conventional linear attention updates its entire state for every event. For object detection, where fine spatial resolution matters, this quickly becomes expensive. Our key idea is simple: an event only carries information about a small spatial region, so why update the entire spatial state? SSLA updates only the relevant parts of the state, enabling fine-grained spatial representations while keeping per-event computation low. We achieve: - >20× lower per-event computation than the strongest prior asynchronous baseline - State-of-the-art accuracy among asynchronous object detection methods - Truly event-by-event inference on CPU, designed to preserve the latency advantage of event cameras Come to our poster on Friday September 11, 2026 from 4-6pm at ExHall #389 Reference: Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention ECCV, 2026 Prophesee SynSense University of Zurich UZH Science European Research Council (ERC) UZHai UZH IfI Tesla BYD #EventCameras #ComputerVision #Robotics #DeepLearning #NeuromorphicVision #AI
2:40

Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where we make linear attention sparse in space, recurrent in time, and parallel in training, enabling the first purely-linear-attention-based neural network for asynchronous object detection with #EventCameras, outperforming the previous best asynchronous method with 20x less computation with truly event-by-event inference on CPU! Code released! Paper: Code: Video: Event cameras promise extremely low-latency vision, but to fully exploit them, the neural network must be low-latency too. We introduce #SpatiallySparseLinearAttention (#SSLA) for asynchronous object detection directly from raw events. Linear attention is particularly appealing for event cameras: it can be trained efficiently in parallel on long event sequences, while at inference it operates recurrently, updating its prediction every time a new event arrives. The problem is that conventional linear attention updates its entire state for every event. For object detection, where fine spatial resolution matters, this quickly becomes expensive. Our key idea is simple: an event only carries information about a small spatial region, so why update the entire spatial state? SSLA updates only the relevant parts of the state, enabling fine-grained spatial representations while keeping per-event computation low. We achieve: - >20× lower per-event computation than the strongest prior asynchronous baseline - State-of-the-art accuracy among asynchronous object detection methods - Truly event-by-event inference on CPU, designed to preserve the latency advantage of event cameras Come to our poster on Friday September 11, 2026 from 4-6pm at ExHall #389 Reference: Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention ECCV, 2026 Prophesee SynSense University of Zurich UZH Science European Research Council (ERC) UZHai UZH IfI Tesla BYD #EventCameras #ComputerVision #Robotics #DeepLearning #NeuromorphicVision #AI

Davide Scaramuzza

52,465 просмотров • 22 дней назад

Super proud and excited to finally present LiteMol-1. This is our first foundation model from LiteFold, pre-trained from scratch. Today, LiteMol-1 can generate small molecules, peptides, cyclic peptides, depsipeptides, peptides with ncAAs, macrocycles, and PROTACs. Across our peptide and small-molecule evaluations, the model shows competitive results. In several settings, we are on-par with or better than frontier structure-based models, at a fraction of the generation cost. But the part I find most interesting is that this is a model for agents. We have seen ourselves how much compute, and how many tokens it can take to generate good binders using frontier structure-based models. Sometimes you need to generate tens of thousands of designs just to get a handful worth taking forward. Now put this inside an AutoResearch loop. The agent has to continuously parse structures, inspect PDB/CIF files, compare candidates, run evaluations, modify the design, and repeat the whole thing again. It becomes extremely expensive very quickly. Sequence space gives us a very different interface. LLMs are much more efficient at inspecting and manipulating compact molecular representations like SMILES than repeatedly operating over full structural files. So LiteMol-1 becomes something like an infinite molecular canvas for the agent. For a given target and objective, the model can continuously propose what a biomolecule could look like. The agent can inspect those generations, take inspiration from them, preserve certain regions, edit others, optimize them, score them, and generate again. Generate → inspect → evaluate → edit → generate again. There is another problem I care a lot about. Most molecule design models today are heavily optimized around binding. But binding is only one part of whether something eventually becomes a therapeutic. What about ADME? Toxicity? Selectivity? Solubility? Membrane permeability? Synthesizability? For this, we also built a Monte Carlo Tree Search-based multi-objective generation framework around LiteMol-1. Instead of combining everything into one score, the search keeps multiple strong candidates, each balancing the desired properties in a different way. As our scoring functions and verifiers get better, the generation system gets better too. We can start steering molecules not just toward “binds well”, but toward a broader therapeutic design specification. The bottleneck slowly moves from simply generating molecules to having sufficiently good verifiers and scoring functions to tell us what is actually worth generating. Check out our technical research blog post for all the details. At LiteFold, our research is focused on engineering biomolecules and building systems that can carefully forecast their pre-clinical success. To stay updated on our research, follow LiteFold. Cheers!
0:41

Super proud and excited to finally present LiteMol-1. This is our first foundation model from LiteFold, pre-trained from scratch. Today, LiteMol-1 can generate small molecules, peptides, cyclic peptides, depsipeptides, peptides with ncAAs, macrocycles, and PROTACs. Across our peptide and small-molecule evaluations, the model shows competitive results. In several settings, we are on-par with or better than frontier structure-based models, at a fraction of the generation cost. But the part I find most interesting is that this is a model for agents. We have seen ourselves how much compute, and how many tokens it can take to generate good binders using frontier structure-based models. Sometimes you need to generate tens of thousands of designs just to get a handful worth taking forward. Now put this inside an AutoResearch loop. The agent has to continuously parse structures, inspect PDB/CIF files, compare candidates, run evaluations, modify the design, and repeat the whole thing again. It becomes extremely expensive very quickly. Sequence space gives us a very different interface. LLMs are much more efficient at inspecting and manipulating compact molecular representations like SMILES than repeatedly operating over full structural files. So LiteMol-1 becomes something like an infinite molecular canvas for the agent. For a given target and objective, the model can continuously propose what a biomolecule could look like. The agent can inspect those generations, take inspiration from them, preserve certain regions, edit others, optimize them, score them, and generate again. Generate → inspect → evaluate → edit → generate again. There is another problem I care a lot about. Most molecule design models today are heavily optimized around binding. But binding is only one part of whether something eventually becomes a therapeutic. What about ADME? Toxicity? Selectivity? Solubility? Membrane permeability? Synthesizability? For this, we also built a Monte Carlo Tree Search-based multi-objective generation framework around LiteMol-1. Instead of combining everything into one score, the search keeps multiple strong candidates, each balancing the desired properties in a different way. As our scoring functions and verifiers get better, the generation system gets better too. We can start steering molecules not just toward “binds well”, but toward a broader therapeutic design specification. The bottleneck slowly moves from simply generating molecules to having sufficiently good verifiers and scoring functions to tell us what is actually worth generating. Check out our technical research blog post for all the details. At LiteFold, our research is focused on engineering biomolecules and building systems that can carefully forecast their pre-clinical success. To stay updated on our research, follow LiteFold. Cheers!

Anindyadeep

53,916 просмотров • 27 дней назад

#WATCH | Mumbai, Maharashtra: When asked about how 3:15 PM cutoffs and closing auction price spreads are disrupting his algorithmic hedging and intraday risk management, with Securities and Exchange Board of India (SEBI) confirming Closing Auction Session (CAS) is permanent, Shrikant Chouhan, Head of Equity Research, Kotak Securities, says, "...If we go through with the CAS-related issues which are giving some kind of uneasiness for option traders... this time option traders, they are mainly affected. The reason is that between this time gap, the prices change a lot, and because of that, most of the time on the day of expiry, especially on the day of expiry, it actually impacts option traders, those who are like writers. Broadly, we are of the view that at present SEBI is holding a lot of meetings with brokers, traders, algo traders, and they are designing their framework... they are ready to make changes as per the difficulties which traders are facing... broadly, I am of the view that it may take some time, but in the next maybe 1-2 months, definitely something will come out concrete, and based on that, again, we can see normalisation in the market... till then, option traders, especially writers or those who are using algorithms, they are trying to stay away from the market because of which volatility has also come down sharply in the market. You can see the market is falling gradually, but at the same time, intraday volatility is very low. Even if we consider Volatility Index (VIX), India's VIX index, that is also close to 11 or maybe 11.5 sort of levels, which is the lowest... it means that option traders are not active in this market. That is why I am of the view that it will take some time to come to normalcy. But eventually, definitely, we will come out with something that will help traders, option traders, I mean to say, in terms of managing their risk. It's not that they can increase their volume and all because our government is primarily focusing on controlling this particular speculative activity... they are trying to control that, and maybe because of that, I think they will come out with something which is very balanced."
3:56

#WATCH | Mumbai, Maharashtra: When asked about how 3:15 PM cutoffs and closing auction price spreads are disrupting his algorithmic hedging and intraday risk management, with Securities and Exchange Board of India (SEBI) confirming Closing Auction Session (CAS) is permanent, Shrikant Chouhan, Head of Equity Research, Kotak Securities, says, "...If we go through with the CAS-related issues which are giving some kind of uneasiness for option traders... this time option traders, they are mainly affected. The reason is that between this time gap, the prices change a lot, and because of that, most of the time on the day of expiry, especially on the day of expiry, it actually impacts option traders, those who are like writers. Broadly, we are of the view that at present SEBI is holding a lot of meetings with brokers, traders, algo traders, and they are designing their framework... they are ready to make changes as per the difficulties which traders are facing... broadly, I am of the view that it may take some time, but in the next maybe 1-2 months, definitely something will come out concrete, and based on that, again, we can see normalisation in the market... till then, option traders, especially writers or those who are using algorithms, they are trying to stay away from the market because of which volatility has also come down sharply in the market. You can see the market is falling gradually, but at the same time, intraday volatility is very low. Even if we consider Volatility Index (VIX), India's VIX index, that is also close to 11 or maybe 11.5 sort of levels, which is the lowest... it means that option traders are not active in this market. That is why I am of the view that it will take some time to come to normalcy. But eventually, definitely, we will come out with something that will help traders, option traders, I mean to say, in terms of managing their risk. It's not that they can increase their volume and all because our government is primarily focusing on controlling this particular speculative activity... they are trying to control that, and maybe because of that, I think they will come out with something which is very balanced."

ANI

37,157 просмотров • 1 месяц назад