Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

MeshSplatting: Differentiable Rendering with Opaque Meshes Contributions: (i) An end-to-end optimization of mesh-based scene representations retains visual quality while training 2× faster than current state-of-the-art methods. (ii) Rather than a polygon soup, we generate a connected mesh by refining the vertex locations of a restricted Delaunay triangulation. (iii) Triangles... show more

MrNeRF

15,994 subscribers

15,044 görüntüleme • 7 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

[NeurIPS '24] DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation Abstract (excerpt) We introduce DreamMesh4D, a novel framework that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D object from a monocular video. To overcome the limitation of classical texture representation, we bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh provided by a single image based 3D generation method. Sparse points are then uniformly sampled across the surface of the mesh, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the bound surface Gaussians are deformed via a geometric skinning algorithm. The skinning algorithm is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate that our method outperforms prior video-to-4D generation methods in terms of rendering quality and spatial-temporal consistency.

[NeurIPS '24] DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation Abstract (excerpt) We introduce DreamMesh4D, a novel framework that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D object from a monocular video. To overcome the limitation of classical texture representation, we bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. In particular, DreamMesh4D begins with a coarse mesh provided by a single image based 3D generation method. Sparse points are then uniformly sampled across the surface of the mesh, and are used to build a deformation graph to drive the motion of the 3D object for the sake of computational efficiency and providing additional constraint. For each step, transformations of sparse control points are predicted using a deformation network, and the mesh vertices as well as the bound surface Gaussians are deformed via a geometric skinning algorithm. The skinning algorithm is a hybrid approach combining LBS (linear blending skinning) and DQS (dual-quaternion skinning), mitigating drawbacks associated with both approaches. The static surface Gaussians and mesh vertices as well as the dynamic deformation network are learned via reference view photometric loss, score distillation loss as well as other regularization losses in a two-stage manner. Extensive experiments demonstrate that our method outperforms prior video-to-4D generation methods in terms of rendering quality and spatial-temporal consistency.

MrNeRF

12,323 görüntüleme • 1 yıl önce

[SIGGRAPH '25] TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modeling Note: On the left that's a 3DGS rendering! Contributions: 1. We propose a simple approach for rigging 3D Gaussians within the continuous tangent space of 3DMM face models, allowing Gaussians to move freely across mesh triangles. 2. We propose a novel CNN-based deformation model that is agnostic to the number of 3D Gaussians, naturally enabling adaptively densification of the representation to improve detail where most needed, with expression-dependent shading. 3. We show significant improvements over baseline SOTA methods and demonstrate the ability to render even extreme close-up images at high quality.

[SIGGRAPH '25] TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modeling Note: On the left that's a 3DGS rendering! Contributions: 1. We propose a simple approach for rigging 3D Gaussians within the continuous tangent space of 3DMM face models, allowing Gaussians to move freely across mesh triangles. 2. We propose a novel CNN-based deformation model that is agnostic to the number of 3D Gaussians, naturally enabling adaptively densification of the representation to improve detail where most needed, with expression-dependent shading. 3. We show significant improvements over baseline SOTA methods and demonstrate the ability to render even extreme close-up images at high quality.

MrNeRF

29,010 görüntüleme • 1 yıl önce

VGGT: Visual Geometry Grounded Transformer TL;DR: Is DUSt3R facing a formidable new rival? Contributions: (1) We introduce VGGT, a large feed-forward transformer that can, given one, a few, or even hundreds of images of a scene, predict all its key 3D attributes - including camera intrinsics and extrinsics, point maps, depth maps, and 3D point tracks - in seconds. (2) We demonstrate that VGGT’s predictions are directly usable, being highly competitive and usually better than those of state-of-the-art methods that use slow post-processing optimization techniques. (3) We also show that when further combined with BA post-processing, VGGT achieves state-of-the-art results across the board, even when compared to methods that specialize in a subset of 3D tasks, often improving quality substantially.

VGGT: Visual Geometry Grounded Transformer TL;DR: Is DUSt3R facing a formidable new rival? Contributions: (1) We introduce VGGT, a large feed-forward transformer that can, given one, a few, or even hundreds of images of a scene, predict all its key 3D attributes - including camera intrinsics and extrinsics, point maps, depth maps, and 3D point tracks - in seconds. (2) We demonstrate that VGGT’s predictions are directly usable, being highly competitive and usually better than those of state-of-the-art methods that use slow post-processing optimization techniques. (3) We also show that when further combined with BA post-processing, VGGT achieves state-of-the-art results across the board, even when compared to methods that specialize in a subset of 3D tasks, often improving quality substantially.

MrNeRF

29,461 görüntüleme • 1 yıl önce

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives paper page: Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations.

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives paper page: Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations.

AK

38,571 görüntüleme • 3 yıl önce

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

140,992 görüntüleme • 2 yıl önce

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

AK

20,651 görüntüleme • 2 yıl önce

Over the weekend, I added multiplayer "paint ball" dogfight to Tiny Skies, so players can fly around pelting each other with colourful paint balls, just for fun. I tried meshy and tripo to generate some 3D models for the game and the mesh vertices count are really high and lags the game a lot (probably because my game is a globe with thousands of objects - albeit mostly instanced). To my surprise, I found Gemini 3.1 Pro to be really quite good at generating good looking 3D models that are low in poly count. It generated 3D models at a much higher quality than Opus 4.6, Sonnet 4.6 and GPT 5.4. You can see the upgraded biplane models and some objects in the video below. #vibejam

Over the weekend, I added multiplayer "paint ball" dogfight to Tiny Skies, so players can fly around pelting each other with colourful paint balls, just for fun. I tried meshy and tripo to generate some 3D models for the game and the mesh vertices count are really high and lags the game a lot (probably because my game is a globe with thousands of objects - albeit mostly instanced). To my surprise, I found Gemini 3.1 Pro to be really quite good at generating good looking 3D models that are low in poly count. It generated 3D models at a much higher quality than Opus 4.6, Sonnet 4.6 and GPT 5.4. You can see the upgraded biplane models and some objects in the video below. #vibejam

Danny Limanseta

36,091 görüntüleme • 3 ay önce

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,771 görüntüleme • 1 yıl önce

$I never believed A.I. could replicate the quality of 3D channels like ZackDFilms. But that is coming to an end. It was always so far off, required so much prompting and you just faced too much bs. However, the gap between A.I and high quality 3D animations really is closing faster than I thought. A person who has barely any editing experience can now make a video like the one I linked below by using Nano Banana Pro + Kling A.I. Yes, it may not be quite there yet. Yes, it still requires a lot of prompting. But this is not slop content. And it's improving rapidly. Those who believe these A.I. videos won't reach the top standard we see in 3D Shorts are just coping imo. Paired with voiceover + sound design, you have a high quality video done for a fraction of the cost and time. The need for advanced skills is disappearing, which has big pros and cons. I'm very curious to see what will be considered as 'high barrier to entry' in the short-form space by the end of next year. The opportunities are endless with A.I. - you have to use it to your advantage.$

I never believed A.I. could replicate the quality of 3D channels like ZackDFilms. But that is coming to an end. It was always so far off, required so much prompting and you just faced too much bs. However, the gap between A.I and high quality 3D animations really is closing faster than I thought. A person who has barely any editing experience can now make a video like the one I linked below by using Nano Banana Pro + Kling A.I. Yes, it may not be quite there yet. Yes, it still requires a lot of prompting. But this is not slop content. And it's improving rapidly. Those who believe these A.I. videos won't reach the top standard we see in 3D Shorts are just coping imo. Paired with voiceover + sound design, you have a high quality video done for a fraction of the cost and time. The need for advanced skills is disappearing, which has big pros and cons. I'm very curious to see what will be considered as 'high barrier to entry' in the short-form space by the end of next year. The opportunities are endless with A.I. - you have to use it to your advantage.

igorm

202,383 görüntüleme • 7 ay önce

$FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.$

FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.

AK

45,459 görüntüleme • 2 yıl önce

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

Stan Szymanowicz

31,570 görüntüleme • 3 ay önce

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,643 görüntüleme • 3 yıl önce

My submission to the Cerebras Google Gemma hackathon, Carbide. Carbide is an ontology builder. Based on a research prompt Carbide uses Cerebras Gemma4 agents to build an ontology. The agents connects individual nodes of information based on the relationship that the agents infer. The agents also assign a confidence metric for how accurate they think the relationship is. You can see each node in a relationship table of the information -> relationship -> target -> and the confidence the relationship is true. The inspector shows the node's connected edges and as an expand button to grow the ontology in the direction of that information. You can scroll through each concept and load them in the inspector. You can search the entire graph based on the kind of relationships generated, or key words. A visual shows how each nodes connects to another one and by what relationship, and changes with new nodes or search filters. Sorry for the audio quality. $CBRS $GOOG

My submission to the Cerebras Google Gemma hackathon, Carbide. Carbide is an ontology builder. Based on a research prompt Carbide uses Cerebras Gemma4 agents to build an ontology. The agents connects individual nodes of information based on the relationship that the agents infer. The agents also assign a confidence metric for how accurate they think the relationship is. You can see each node in a relationship table of the information -> relationship -> target -> and the confidence the relationship is true. The inspector shows the node's connected edges and as an expand button to grow the ontology in the direction of that information. You can scroll through each concept and load them in the inspector. You can search the entire graph based on the kind of relationships generated, or key words. A visual shows how each nodes connects to another one and by what relationship, and changes with new nodes or search filters. Sorry for the audio quality. $CBRS $GOOG

QuantHood

16,376 görüntüleme • 18 gün önce

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

MrNeRF

11,638 görüntüleme • 1 yıl önce

The #GTA 3 port to the Sega #Dreamcast has been progressing at an incredible pace. It's been amazing to see the whole DC community come together to tag-team this "impossible" project... Here it is running on a stock DC, no longer requiring the 32MB RAM hardware modification, only a few weeks into development. Since I finally got the time to sit down, build the codebase, and look into some of what I think is the critical path for performance, let's talk about some technical shit, and some of the future steps I think can be taken to further improve performance. First of all, I want everyone to take note that this is NOT a port of the PS2 version. This is a port of the PC version, which has extra content, increased draw distance, improved textures, and other things that have actually increased the challenge here... Whether the DC version will ultimately have these additions or not will remain to be seen, but we're running into plenty of shit that the PS2 didn't have to worry about (like these big-ass PC replay saves won't fit onto a Visual Memory Unit!) Secondly, lets talk about what is and isn't currently optimized, because it's absolutely vital that the DC's hardware is fully utilized here for the sake of performance and achieving a competitive polygon count. Unlike with modern devices, where the whole graphics pipeline is handled by the GPU, both the PS2 and Dreamcast were responsible for transforming and doing lighting calculations for each vertex BEFORE they got submitted to the GPU. The PS2 had a vector coprocessor to do this, while the Dreamcast had a few extremely important SIMD and fast math assembly instructions on its CPU to do these computations. Up until literally just a few hours ago (not shown in this footage), the Dreamcast's SH4 was doing 100% of these operations in slow-ass plain C and C++ code, which is absolutely sub-optimal and is immediately bogging down its CPU with just transforming vertices, bottlenecking the entire graphics pipeline on the fist T&L stage, and also leaving less CPU time for handling other gameplay logic... this is going to absolutely have to be addressed (and already has begun to be). Another issue that is crippling performance here is the fact that the models are all using individual triangles rather than triangle strips, which the Dreamcast's PVR GPU was designed to handle better... Converting these models to use strips rather than individual triangles will result in MANY different gainz for the DC, as you're going from 3N to N+2 vertices per triangle. Converting the models to triangle strips will 1) reduce load times, since model assets will be smaller 2) reduce the amount of video memory required to hold these vertices on the GPU 3) reduce the amount of shit that must be transferred from the CPU to the GPU and 4) give us back a bunch of CPU time, since the SH4 will be less bogged down transforming redundant vertices! TL;DR: This is still EXTREMELY suboptimal in terms of fully utilizing the graphical potential of the Dreamcast. There is going to be a LOT that can be done still to both improve performance and polygon counts, so stay tuned! FINALLY: Mad respect and love to Stefanos Kornilios Mitsis Poiitidis, for doing an amazing job leading this project, and to Frogbull , Esppiral, and everyone else who is helping us stick it to the PS2 by making this happen! #gamedev #retrogaming #cplusplus

The #GTA 3 port to the Sega #Dreamcast has been progressing at an incredible pace. It's been amazing to see the whole DC community come together to tag-team this "impossible" project... Here it is running on a stock DC, no longer requiring the 32MB RAM hardware modification, only a few weeks into development. Since I finally got the time to sit down, build the codebase, and look into some of what I think is the critical path for performance, let's talk about some technical shit, and some of the future steps I think can be taken to further improve performance. First of all, I want everyone to take note that this is NOT a port of the PS2 version. This is a port of the PC version, which has extra content, increased draw distance, improved textures, and other things that have actually increased the challenge here... Whether the DC version will ultimately have these additions or not will remain to be seen, but we're running into plenty of shit that the PS2 didn't have to worry about (like these big-ass PC replay saves won't fit onto a Visual Memory Unit!) Secondly, lets talk about what is and isn't currently optimized, because it's absolutely vital that the DC's hardware is fully utilized here for the sake of performance and achieving a competitive polygon count. Unlike with modern devices, where the whole graphics pipeline is handled by the GPU, both the PS2 and Dreamcast were responsible for transforming and doing lighting calculations for each vertex BEFORE they got submitted to the GPU. The PS2 had a vector coprocessor to do this, while the Dreamcast had a few extremely important SIMD and fast math assembly instructions on its CPU to do these computations. Up until literally just a few hours ago (not shown in this footage), the Dreamcast's SH4 was doing 100% of these operations in slow-ass plain C and C++ code, which is absolutely sub-optimal and is immediately bogging down its CPU with just transforming vertices, bottlenecking the entire graphics pipeline on the fist T&L stage, and also leaving less CPU time for handling other gameplay logic... this is going to absolutely have to be addressed (and already has begun to be). Another issue that is crippling performance here is the fact that the models are all using individual triangles rather than triangle strips, which the Dreamcast's PVR GPU was designed to handle better... Converting these models to use strips rather than individual triangles will result in MANY different gainz for the DC, as you're going from 3N to N+2 vertices per triangle. Converting the models to triangle strips will 1) reduce load times, since model assets will be smaller 2) reduce the amount of video memory required to hold these vertices on the GPU 3) reduce the amount of shit that must be transferred from the CPU to the GPU and 4) give us back a bunch of CPU time, since the SH4 will be less bogged down transforming redundant vertices! TL;DR: This is still EXTREMELY suboptimal in terms of fully utilizing the graphical potential of the Dreamcast. There is going to be a LOT that can be done still to both improve performance and polygon counts, so stay tuned! FINALLY: Mad respect and love to Stefanos Kornilios Mitsis Poiitidis, for doing an amazing job leading this project, and to Frogbull , Esppiral, and everyone else who is helping us stick it to the PS2 by making this happen! #gamedev #retrogaming #cplusplus

Falco Girgis

88,356 görüntüleme • 1 yıl önce

YOKO ONO: ONOCHORD, VENICE, 2004 Yoko: The world is divided in two industries. One is the War Industry and the other is the Peace Industry. The people in the War Industry are totally together. They don't have to talk to each other, even. They know exactly what they want to do. They want to go out there, kill and make money. But the people in the Peace Industry, which are us - we are so idealistic that each one of us criticises the other Peace Person in the Peace Industry. And we are always just arguing and we are wasting our energies doing that. So let's just forgive each other and see that we are in the Peace Industry and that's all that counts. Even if you are not marching for peace, just be yourself, being a florist, being a merchant, being a talior, anything. That way you're contributing to the Peace Industry. People are just concentrating on fear, confusion and anger. And therefore just for a moment, I'd like us to think about Love. In a very magical, straight way, John and I met in London and from then on we stood for Peace and Love. And when I do this kind of event. Well it is... I was inspired to do it, but I still think that I'm still with John in spirit. John and I created the country called Nutopia. Not Utopia, because there was Utopia as a concept already. And we wanted to create a new concept, so we just added N on it - Nutopia - and as a country. Well, that is the concept of a country. And we all are citizens of that country. And in my apartment in the Dakota Building, we put a little plaque on the back door, the kitchen door. It says 'Nutopian Embassy' and even now we have that. (laughs). Nutopia exists in our minds. And because of that, some people want to rebel against it. The reason some want to rebel against it is a good proof that it exists. I think that it was a terrible thing that happened in Chechnya. But we have to still keep our hopes up. And instead of giving up, we have to keep on sending the message of Love to each other. You say that I am the Ambassador of Peace. We are all Ambassadors of Peace. You are too. Everybody in this room are Ambassadors of Peace. Just the fact that we are not participating in War. The fact that we are here, and we are what we are, means that we are in the Peace Industry. All of us. John and I used to say that our apartment in the Dakota is a conceptual monastry, just for the two of us. And when we go out of the Dakota, we get so many people communicating with us, so it's very important that we had silence and quietness. And my apartment is a very small space compared to the world. And I need that for my peace of mind. You should be kind to each other. You should come together, hug each other, love each other, express our love to each other and we should make it work. We should finally create a world that is a totally an Earth for Us. So let's do it. Yoko Ono, OpenAsia Press Conference, whilst exhibiting Onochord, 2004 by Yoko Ono (Nutopia) at the Venice Biennale: OpenAsia 2004, Lido Di Venezia, Venice, Italy, 9 September 2004.

YOKO ONO: ONOCHORD, VENICE, 2004 Yoko: The world is divided in two industries. One is the War Industry and the other is the Peace Industry. The people in the War Industry are totally together. They don't have to talk to each other, even. They know exactly what they want to do. They want to go out there, kill and make money. But the people in the Peace Industry, which are us - we are so idealistic that each one of us criticises the other Peace Person in the Peace Industry. And we are always just arguing and we are wasting our energies doing that. So let's just forgive each other and see that we are in the Peace Industry and that's all that counts. Even if you are not marching for peace, just be yourself, being a florist, being a merchant, being a talior, anything. That way you're contributing to the Peace Industry. People are just concentrating on fear, confusion and anger. And therefore just for a moment, I'd like us to think about Love. In a very magical, straight way, John and I met in London and from then on we stood for Peace and Love. And when I do this kind of event. Well it is... I was inspired to do it, but I still think that I'm still with John in spirit. John and I created the country called Nutopia. Not Utopia, because there was Utopia as a concept already. And we wanted to create a new concept, so we just added N on it - Nutopia - and as a country. Well, that is the concept of a country. And we all are citizens of that country. And in my apartment in the Dakota Building, we put a little plaque on the back door, the kitchen door. It says 'Nutopian Embassy' and even now we have that. (laughs). Nutopia exists in our minds. And because of that, some people want to rebel against it. The reason some want to rebel against it is a good proof that it exists. I think that it was a terrible thing that happened in Chechnya. But we have to still keep our hopes up. And instead of giving up, we have to keep on sending the message of Love to each other. You say that I am the Ambassador of Peace. We are all Ambassadors of Peace. You are too. Everybody in this room are Ambassadors of Peace. Just the fact that we are not participating in War. The fact that we are here, and we are what we are, means that we are in the Peace Industry. All of us. John and I used to say that our apartment in the Dakota is a conceptual monastry, just for the two of us. And when we go out of the Dakota, we get so many people communicating with us, so it's very important that we had silence and quietness. And my apartment is a very small space compared to the world. And I need that for my peace of mind. You should be kind to each other. You should come together, hug each other, love each other, express our love to each other and we should make it work. We should finally create a world that is a totally an Earth for Us. So let's do it. Yoko Ono, OpenAsia Press Conference, whilst exhibiting Onochord, 2004 by Yoko Ono (Nutopia) at the Venice Biennale: OpenAsia 2004, Lido Di Venezia, Venice, Italy, 9 September 2004.

Yoko Ono

35,208 görüntüleme • 2 yıl önce

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

125,146 görüntüleme • 1 yıl önce

Here are the graphics we made at Intel. I work at Nike innovation now so don’t work on them anymore so not sure all the details for this version but basically we used cameras and computer vision to track each athlete in each lane. Because the cameras were calibrated to specific locations and each frame of video timestamped, we could get distance and time data which you can use to get speed, naturally. Feed the data to a 3D rendered stadium and overlay graphics, and voila. We chose to show speed at the end of each “phase” of the race which I based the location on the data we got from testing and common knowledge of the 100m. So stopping at 30m (drive) and 60 (maintenance) and the end was intentional. Looks like a white label indicates a leader at the end of the phase. The green thing at the front of their label looks to indicate if their are accelerating. This is a bit of a deception because you’re always either accelerating or decelerating. So what it’s actually an indication of is if you’re within a certain percentage of your top speed. The green at the end is graded based on that which is why it’s a gradient. Basically shows that Noah not only has the highest tops speed but holds it best. The final view is everyone’s top speed and where they hit it in the race. It’s freaking wild that Tobago hit his top speed so late.

Here are the graphics we made at Intel. I work at Nike innovation now so don’t work on them anymore so not sure all the details for this version but basically we used cameras and computer vision to track each athlete in each lane. Because the cameras were calibrated to specific locations and each frame of video timestamped, we could get distance and time data which you can use to get speed, naturally. Feed the data to a 3D rendered stadium and overlay graphics, and voila. We chose to show speed at the end of each “phase” of the race which I based the location on the data we got from testing and common knowledge of the 100m. So stopping at 30m (drive) and 60 (maintenance) and the end was intentional. Looks like a white label indicates a leader at the end of the phase. The green thing at the front of their label looks to indicate if their are accelerating. This is a bit of a deception because you’re always either accelerating or decelerating. So what it’s actually an indication of is if you’re within a certain percentage of your top speed. The green at the end is graded based on that which is why it’s a gradient. Basically shows that Noah not only has the highest tops speed but holds it best. The final view is everyone’s top speed and where they hit it in the race. It’s freaking wild that Tobago hit his top speed so late.

Ashton Eaton

1,164,463 görüntüleme • 1 yıl önce

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

Presto! Distilling Steps and Layers for Accelerating Music Generation Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge.

AK

30,430 görüntüleme • 1 yıl önce