Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

MeshSplatting: Differentiable Rendering with Opaque Meshes Contributions: (i) An end-to-end optimization of mesh-based scene representations retains visual quality while training 2× faster than current state-of-the-art methods. (ii) Rather than a polygon soup, we generate a connected mesh by refining the vertex locations of a restricted Delaunay triangulation. (iii) Triangles... show more

MrNeRF

16,880 subscribers

15,349 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

14 Comments

MrNeRF9 months ago

Author thread:

MrNeRF9 months ago

Paper: Project: Code:

Lucky Iyinbor9 months ago

The idea to use triangle soup as connectivity signal for Delaunay is very hacky - I like It 😁

ǝɥǝ⅄9 months ago

over?

MrNeRF9 months ago

We are cooked :)

Jan Held9 months ago

Thanks for sharing @janusch_patas ! 😁

Adrian9 months ago

The mesh results look like voxel modeling , unclean but it works!

Zwecharki of Immense AGI fundamentalism9 months ago

At this point i don’t care about speed. Just quality. Especially subsurface scattering, reflections, glass distortions and transparency. Blur as well

S K Ʌ N9 months ago

Cool! But i don't get why you don't crosspost to bluesky, i come here only to check your stuff now lol ;)

MrNeRF9 months ago

Too may platforms out there:)

S K Ʌ N9 months ago

Yeah, but is the worst anyway;(

Luc 💭 🇺🇸 🇺🇦 🇧🇷 💭9 months ago

too bad we can't use it, no license?

Luca | ReplyGrinder.ai 🌊9 months ago

Nice to see research that directly plugs into existing game pipelines - brings theory to the playfield! Thx

Himanshu Kumar9 months ago

Interesting work, MrNeRF! But can it truly replace the polygon soup, I wonder?

Related Videos

[SIGGRAPH '25] TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modeling Note: On the left that's a 3DGS rendering! Contributions: 1. We propose a simple approach for rigging 3D Gaussians within the continuous tangent space of 3DMM face models, allowing Gaussians to move freely across mesh triangles. 2. We propose a novel CNN-based deformation model that is agnostic to the number of 3D Gaussians, naturally enabling adaptively densification of the representation to improve detail where most needed, with expression-dependent shading. 3. We show significant improvements over baseline SOTA methods and demonstrate the ability to render even extreme close-up images at high quality.

[SIGGRAPH '25] TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modeling Note: On the left that's a 3DGS rendering! Contributions: 1. We propose a simple approach for rigging 3D Gaussians within the continuous tangent space of 3DMM face models, allowing Gaussians to move freely across mesh triangles. 2. We propose a novel CNN-based deformation model that is agnostic to the number of 3D Gaussians, naturally enabling adaptively densification of the representation to improve detail where most needed, with expression-dependent shading. 3. We show significant improvements over baseline SOTA methods and demonstrate the ability to render even extreme close-up images at high quality.

MrNeRF

29,057 views • 1 year ago

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization paper page: This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly common paradigm in applications including photogrammetry, generative modeling, and inverse physics. Existing implementations adapt classic isosurface extraction algorithms like Marching Cubes or Dual Contouring; these techniques were designed to extract meshes from fixed, known fields, and in the optimization setting they lack the degrees of freedom to represent high-quality feature-preserving meshes, or suffer from numerical instabilities. We introduce FlexiCubes, an isosurface representation specifically designed for optimizing an unknown mesh with respect to geometric, visual, or even physical objectives. Our main insight is to introduce additional carefully-chosen parameters into the representation, which allow local flexible adjustments to the extracted mesh geometry and connectivity. These parameters are updated along with the underlying scalar field via automatic differentiation when optimizing for a downstream task. We base our extraction scheme on Dual Marching Cubes for improved topological properties, and present extensions to optionally generate tetrahedral and hierarchically-adaptive meshes. Extensive experiments validate FlexiCubes on both synthetic benchmarks and real-world applications, showing that it offers significant improvements in mesh quality and geometric fidelity.

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization paper page: This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly common paradigm in applications including photogrammetry, generative modeling, and inverse physics. Existing implementations adapt classic isosurface extraction algorithms like Marching Cubes or Dual Contouring; these techniques were designed to extract meshes from fixed, known fields, and in the optimization setting they lack the degrees of freedom to represent high-quality feature-preserving meshes, or suffer from numerical instabilities. We introduce FlexiCubes, an isosurface representation specifically designed for optimizing an unknown mesh with respect to geometric, visual, or even physical objectives. Our main insight is to introduce additional carefully-chosen parameters into the representation, which allow local flexible adjustments to the extracted mesh geometry and connectivity. These parameters are updated along with the underlying scalar field via automatic differentiation when optimizing for a downstream task. We base our extraction scheme on Dual Marching Cubes for improved topological properties, and present extensions to optionally generate tetrahedral and hierarchically-adaptive meshes. Extensive experiments validate FlexiCubes on both synthetic benchmarks and real-world applications, showing that it offers significant improvements in mesh quality and geometric fidelity.

AK

38,452 views • 3 years ago

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives paper page: Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations.

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives paper page: Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations.

AK

38,571 views • 3 years ago

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

141,058 views • 2 years ago

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

AK

20,668 views • 2 years ago

Over the weekend, I added multiplayer "paint ball" dogfight to Tiny Skies, so players can fly around pelting each other with colourful paint balls, just for fun. I tried meshy and tripo to generate some 3D models for the game and the mesh vertices count are really high and lags the game a lot (probably because my game is a globe with thousands of objects - albeit mostly instanced). To my surprise, I found Gemini 3.1 Pro to be really quite good at generating good looking 3D models that are low in poly count. It generated 3D models at a much higher quality than Opus 4.6, Sonnet 4.6 and GPT 5.4. You can see the upgraded biplane models and some objects in the video below. #vibejam

Over the weekend, I added multiplayer "paint ball" dogfight to Tiny Skies, so players can fly around pelting each other with colourful paint balls, just for fun. I tried meshy and tripo to generate some 3D models for the game and the mesh vertices count are really high and lags the game a lot (probably because my game is a globe with thousands of objects - albeit mostly instanced). To my surprise, I found Gemini 3.1 Pro to be really quite good at generating good looking 3D models that are low in poly count. It generated 3D models at a much higher quality than Opus 4.6, Sonnet 4.6 and GPT 5.4. You can see the upgraded biplane models and some objects in the video below. #vibejam

Danny Limanseta

36,419 views • 5 months ago

Sophie Cunningham says she is considering retiring from professional basketball and the WNBA at the end of the season following the altercation with DiJonai Carrington: “We are all women. We should support each other, protect each other and treat each other like sisters. We should take each other’s hands instead of constantly tearing each other down. Society is already divided enough, and it’s heartbreaking to see players treating one another like enemies. After what happened, I cried in my room for an entire day. It affected me more than people realize. I thought about whether this is still the life I want. For the first time, I seriously considered that retiring from professional basketball at the end of the season might be best for me.”

Sophie Cunningham says she is considering retiring from professional basketball and the WNBA at the end of the season following the altercation with DiJonai Carrington: “We are all women. We should support each other, protect each other and treat each other like sisters. We should take each other’s hands instead of constantly tearing each other down. Society is already divided enough, and it’s heartbreaking to see players treating one another like enemies. After what happened, I cried in my room for an entire day. It affected me more than people realize. I thought about whether this is still the life I want. For the first time, I seriously considered that retiring from professional basketball at the end of the season might be best for me.”

Fever Updates

801,521 views • 1 month ago

More PlayStation 5 games are now running smoothly on Kyty PS5 Emulator. Current playable titles include: • Tomb Raider IV-VI Remastered • STORY OF SEASONS: A Wonderful Life • PAC-MAN WORLD Re-PAC • Double Dragon Gaiden: Rise of the Dragons All of them are now rendering and running smoothly at 60 FPS. PS5 emulation is progressing faster than many expected, and each update continues to improve game compatibility and overall stability. Which PS5 game would you like to see become playable next?

More PlayStation 5 games are now running smoothly on Kyty PS5 Emulator. Current playable titles include: • Tomb Raider IV-VI Remastered • STORY OF SEASONS: A Wonderful Life • PAC-MAN WORLD Re-PAC • Double Dragon Gaiden: Rise of the Dragons All of them are now rendering and running smoothly at 60 FPS. PS5 emulation is progressing faster than many expected, and each update continues to improve game compatibility and overall stability. Which PS5 game would you like to see become playable next?

Techjunkie Aman

141,620 views • 1 month ago

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 views • 1 year ago

$I never believed A.I. could replicate the quality of 3D channels like ZackDFilms. But that is coming to an end. It was always so far off, required so much prompting and you just faced too much bs. However, the gap between A.I and high quality 3D animations really is closing faster than I thought. A person who has barely any editing experience can now make a video like the one I linked below by using Nano Banana Pro + Kling A.I. Yes, it may not be quite there yet. Yes, it still requires a lot of prompting. But this is not slop content. And it's improving rapidly. Those who believe these A.I. videos won't reach the top standard we see in 3D Shorts are just coping imo. Paired with voiceover + sound design, you have a high quality video done for a fraction of the cost and time. The need for advanced skills is disappearing, which has big pros and cons. I'm very curious to see what will be considered as 'high barrier to entry' in the short-form space by the end of next year. The opportunities are endless with A.I. - you have to use it to your advantage.$

I never believed A.I. could replicate the quality of 3D channels like ZackDFilms. But that is coming to an end. It was always so far off, required so much prompting and you just faced too much bs. However, the gap between A.I and high quality 3D animations really is closing faster than I thought. A person who has barely any editing experience can now make a video like the one I linked below by using Nano Banana Pro + Kling A.I. Yes, it may not be quite there yet. Yes, it still requires a lot of prompting. But this is not slop content. And it's improving rapidly. Those who believe these A.I. videos won't reach the top standard we see in 3D Shorts are just coping imo. Paired with voiceover + sound design, you have a high quality video done for a fraction of the cost and time. The need for advanced skills is disappearing, which has big pros and cons. I'm very curious to see what will be considered as 'high barrier to entry' in the short-form space by the end of next year. The opportunities are endless with A.I. - you have to use it to your advantage.

igorm

202,383 views • 9 months ago

$FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.$

FAU Erlangen-Nürnberg presents TRIPS Trilinear Point Splatting for Real-Time Radiance Field Rendering paper page: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage.

AK

45,489 views • 2 years ago

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

Stan Szymanowicz

32,815 views • 6 months ago

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,729 views • 3 years ago

My submission to the Cerebras Google Gemma hackathon, Carbide. Carbide is an ontology builder. Based on a research prompt Carbide uses Cerebras Gemma4 agents to build an ontology. The agents connects individual nodes of information based on the relationship that the agents infer. The agents also assign a confidence metric for how accurate they think the relationship is. You can see each node in a relationship table of the information -> relationship -> target -> and the confidence the relationship is true. The inspector shows the node's connected edges and as an expand button to grow the ontology in the direction of that information. You can scroll through each concept and load them in the inspector. You can search the entire graph based on the kind of relationships generated, or key words. A visual shows how each nodes connects to another one and by what relationship, and changes with new nodes or search filters. Sorry for the audio quality. $CBRS $GOOG

My submission to the Cerebras Google Gemma hackathon, Carbide. Carbide is an ontology builder. Based on a research prompt Carbide uses Cerebras Gemma4 agents to build an ontology. The agents connects individual nodes of information based on the relationship that the agents infer. The agents also assign a confidence metric for how accurate they think the relationship is. You can see each node in a relationship table of the information -> relationship -> target -> and the confidence the relationship is true. The inspector shows the node's connected edges and as an expand button to grow the ontology in the direction of that information. You can scroll through each concept and load them in the inspector. You can search the entire graph based on the kind of relationships generated, or key words. A visual shows how each nodes connects to another one and by what relationship, and changes with new nodes or search filters. Sorry for the audio quality. $CBRS $GOOG

QuantHood

16,376 views • 2 months ago

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

AK

143,469 views • 3 years ago

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

AK

117,068 views • 2 years ago